We Keep Coding

Unexpected data returned by PyMySQL when using GROUP_CONCAT

I generated SQL query for my database as following:
SELECT match_id, Group_concat(name SEPARATOR ', ') AS 'winners'
FROM players
WHERE match_id = 4
AND rank = 1
GROUP BY match_id;
(for structure and example data, see sql fiddle)
which results in MySQL:
+----------+------------+
| match_id | winners |
+----------+------------+
| 4 | P106, P107 |
+----------+------------+
So I tried to apply this SQL query into python using pymysql module:
conn = pymysql.connect(**DB_CONFIG, cursorclass=pymysql.cursors.DictCursor)
cur = conn.cursor()
sql = """SELECT match_id, Group_concat(name SEPARATOR ', ') AS 'winners'
FROM players
WHERE match_id = %s
AND rank = 1
GROUP BY match_id """
cur.execute(sql, (4, ))
result = cur.fetchall()
But strangely, the value of result was:
[{'match_id': 4, 'winners': 'P106'}]
So, why is the result of pymysql different from the result of MySQL Client? How can I fix it?

Why am I getting this Not all parameters were used in the SQL statement error using python?

I have a mysql database with one table that I'm trying to import from CSV using python.
The error I'm getting is:
mysql.connector.errors.ProgrammingError: Not all parameters were used in the SQL statement
But I have only 1 field in the table, so I am only using 1 parameter.
This is the table in MySQL:
desc billing_info;
+-----------+---------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+---------+------+-----+---------+-------+
| InvoiceId | int(11) | NO | | NULL | |
+-----------+---------+------+-----+---------+-------+
This is my code:
import mysql.connector
import csv
source_dir = 'source_files/aws_bills/'
source_file = 'test_data.csv'
source = source_dir + source_file
mydb = mysql.connector.connect(user='xxxx', password='xxxx',
host='xxxx',
database='aws_bill')
cursor = mydb.cursor()
csv_data = csv.reader(source)
sql = "INSERT INTO billing_info (InvoiceId) VALUES (%i)"
for row in csv_data:
cursor.execute(sql, row)
#close the connection to the database.
mydb.commit()
cursor.close()

Your row variable has more than one value in it, maybe you meant:
for row in csv_data:
cursor.execute(sql, (row[0],)) # a one-tuple with the first element in the row..
also, the mysql connector usually wants you to use %s for any type of parameter, i.e.:
sql = "INSERT INTO billing_info (InvoiceId) VALUES (%s)"
update: your second issue is that you haven't opened the file, i.e.:
import os
import mysql.connector
import csv
# source_dir = 'source_files/aws_bills/'
# source_file = 'test_data.csv'
# source = source_dir + source_file
source = os.path.join('source_files', 'aws_bills', 'test_data.csv')
sql = "INSERT INTO billing_info (InvoiceId) VALUES (%s)"
mydb = mysql.connector.connect(user='xxxx', password='xxxx', host='xxxx', database='aws_bill')
cursor = mydb.cursor()
try:
with open(source, 'rb') as fp:
for row in csv.reader(fp):
cursor.execute(sql, (row[0],))
cursor.close()
mydb.commit()
except:
mydb.rollback()
finally:
mydb.close()

MySQL connector Python variables not registering

I'm currently using the mysql-connector-python package to execute database actions on Flask. It's been working so well until suddenly the variables don't seem to working correctly anymore. My code is here:
#bp.route('/addcart', methods=('OPTIONS', 'POST'))
def addcart():
...
userID = session.get("user_id")
reqDict = request.get_json()
itemCode = str(reqDict['itemCode'])
itemAmt = reqDict['itemAmt']
if userID is not None:
db = get_db()
cursor = db.cursor()
query = ('SELECT %s FROM cartdata WHERE id = %s')
cursor.execute(query, (itemCode, userID))
currentNum = cursor.fetchone()[0]
if currentNum is None:
stmt = ('UPDATE cartdata SET %s = 1 WHERE id = %s')
cursor.execute(stmt, (itemCode, userID))
else:
currentNum = int(currentNum) + int(itemAmt)
stmt = ('UPDATE cartdata SET %s = %s WHERE id = %s')
cursor.execute(stmt, (itemCode, currentNum, userID))
....
For some reason, I seem to having trouble with the itemCode variable. When I use it properly, like in the execution of 'query' or 'stmt', it doesn't work. Typically I will get an error saying
" You have an error in your SQL syntax; check the manual that
corresponds to your MySQL server version for the right syntax to use
near ''p1' = 1 WHERE id = 21'".
However, if I do this:
query = ('SELECT ' + itemCode + ' FROM cartdata WHERE id = %s')
...
stmt = ('UPDATE cartdata SET '+ itemCode +' = 1 WHERE id = %s')
...
It works properly as intended.
EDIT: I've checked my backend, and apparently the UPDATE statement does not actually update anything. So now I'm at a complete loss.
I don't understand why the connector suddenly breaks now for variables. I've checked this variables and its types, but they were the expected types. Any insight would be helpful.
My table schema for 'cartdata' looks something like this:
+-------+---------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------+---------+------+-----+---------+-------+
| id | int(11) | NO | PRI | NULL | |
| p1 | int(8) | YES | | NULL | |
| p2 | int(8) | YES | | NULL | |
| p3 | int(8) | YES | | NULL | |
| p4 | int(8) | YES | | NULL | |
| p5 | int(8) | YES | | NULL | |
+-------+---------+------+-----+---------+-------+

That's because when MySQL connector injects your variables into the SQL statement, it formats them according to their type.
You can actually see it in the error message that you get:
"p1' = 1 WHERE id = 21'"
^
So probably, your SQL query looks like this:
SELECT 'p1' FROM cartdata WHERE id = someId
Which is syntactically invalid SQL...
Your second option however seems okay. Btw, it seems weird to adapt the column you want to select depending on the user's input... I'd highly recommend to validate this value with something efficient...
Details
You cannot use %s for column names since this injects a string value in your SQL query and this results in a non valid SQL syntax (column names are not string values).
As above:
SET %s = ...
Generates:
SET 'colName' = ...
which is not valid because you are attempting to affect a value to another value...
That would be the same as trying to do the following in python:
'foo' = 'bar'
or
'foo' = 4
You can use %s when setting values (using SET colName = %s) or filtering values (using WHERE colName = %s) because the type of the values in the column colName is actually a string.
As above:
WHERE colName = %s
Generates:
WHERE colName = 'fooBar'
which is valid because you filter on the values that are equal to the string fooBar.
By the way, you might want to check what
SELECT %s FROM cartdata WHERE id = %s
gives you as a result. That could result problems... Actually MySQL won't tell you anything, but you result will probably be exactly the value of itemCode. (it is valid SQL SELECT 'hello', it just returns 'hello').

MySQLdb placeholder implementation does not work

Python’s MySQLdb module should implement placeholders using format specifiers in the SQL statement string. I am following an exemple from the MYSQL CookBook
import sys
import MySQLdb
import Cookbook
try:
conn = Cookbook.connect ()
print("Connected")
except MySQLdb.Error as e:
print("Cannot connect to server")
print("Error code:", e.args[0])
print("Error message:", e.args[1])
sys.exit (1)
cursor = conn.cursor ()
cursor.execute ("""
INSERT INTO profile (name,birth,color,foods,cats)
VALUES(%s,%s,%s,%s,%s)
""",("Josef", "1971-01-01", None, "eggroll", 4))
But when I check from the shell
mysql> SELECT * FROM profile WHERE name LIKE 'J%';
+----+--------+------------+-------+----------------+------+
| id | name | birth | color | foods | cats |
+----+--------+------------+-------+----------------+------+
| 7 | Joanna | 1952-08-20 | green | lutefisk,fadge | 0 |
+----+--------+------------+-------+----------------+------+
It is obvious that nothing is inserted.Why?
If I add cursor.commit as suggested
cursor.commit()
AttributeError: 'Cursor' object has no attribute 'commit'

You are not committing the transaction.
Add conn.commit() in the end after executing the query.
cursor = conn.cursor()
cursor.execute ("""
INSERT INTO profile (name,birth,color,foods,cats)
VALUES(%s,%s,%s,%s,%s)
""",("Josef", "1971-01-01", None, "eggroll", 4))
conn.commit()

python: psycopg2: psql: Delete row where condition

I'm trying to delete rows from a psql table on a condition.
I want all rows to be deleted if column "TagNaam" equals a variable var_tagnaam.
I've tried the following code and some variants but I can't get it to work.
There aren't any errors though.
cur.execute("DELETE FROM opc_actuelewaardentags WHERE 'TagNaam' = %s", (var_tagnaam,))
Is there something wrong with the syntax?
Edit:
Maybe it is more clear with additional code, the error might be in the other code?
for i in range(len(taginhoud)):
(var_tagnaam, var_tagwaarde, var_tagkwaliteit, var_tagtime) = taginhoud[i]
print (var_tagnaam)
cur.execute("DELETE FROM opc_actuelewaardentags WHERE 'TagNaam' = %s", (var_tagnaam,))
conn.commit()
cur.execute('INSERT INTO opc_actuelewaardentags ("TagNaam", "TagWaarde", "TagKwaliteit", create_date, write_date) VALUES (%s,%s,%s,now(),now())',
(var_tagnaam, var_tagwaarde, var_tagkwaliteit))
conn.commit()
So what I try to do here is:
Retrieve "var_tagnaam" from list "taginhoud".
Then in table opc_actuelewaardentags find all rows where column "Tagnaam" equals the value in "var_tagnaam". (Should be a string)
Then delete those rows where "Tagnaam" = "var_tagnaam". This part doesn't work.
Then insert new rows with data. This part works.
Could this code be wrong to do what I want?
I have tried many things already to solve the upper/lower case problem.
Edit 2:Query in pgadmin worked, trying to do the same thing in python:
I ran this query in pgadmin and it deleted the rows:
delete FROM opc_actuelewaardentags where "TagNaam" = 'Bakkerij.Device1.DB100INT8';
My attempt to make it as similar as possible in python:
var_tagnaam2 = "'"+var_tagnaam+"'"
cur.execute("DELETE FROM opc_actuelewaardentags WHERE \"TagNaam\" = %s", (var_tagnaam2,))
conn.commit()
Tried to escape the double quotes in attempt to make it the same as in pgadmin.

'TagNaam' is not a valid column_name identifier in sql language. You must not use single or double quotes in writing database name, table name or colunm name, but you can use apostrophe (`) .
Invalid:
DELETE FROM opc_actuelewaardentags WHERE 'TagNaam' = 'test';
DELETE FROM opc_actuelewaardentags WHERE "TagNaam" = 'test';
Valid:
DELETE FROM opc_actuelewaardentags WHERE TagNaam = 'test';
DELETE FROM opc_actuelewaardentags WHERE `TagNaam` = 'test';
DELETE FROM opc_actuelewaardentags WHERE "TagNaam" = 'test';
Update: According to PSQL dosc, double quote is a valid character in table and column names. It is especially used for key words while usinga as a table or column name. So following is valid:
DELETE FROM opc_actuelewaardentags WHERE "TagNaam" = 'test';
More is here...

I don't have a psql server, but a mysql server.
For MySQL:
mysql> select * from user where '1' = '1';
+------+
| id |
+------+
| 2 |
| 1 |
+------+
2 rows in set (0.05 sec)
mysql> select * from user;
+------+
| id |
+------+
| 2 |
| 1 |
+------+
2 rows in set (0.00 sec)
mysql> select * from user where '1' = "1";
+------+
| id |
+------+
| 2 |
| 1 |
+------+
2 rows in set (0.00 sec)
mysql> select * from user where 'id' = "1";
Empty set (0.00 sec)
mysql> select * from user where 'id' = 1;
Empty set, 1 warning (0.02 sec)
mysql> select * from user where id = 1;
+------+
| id |
+------+
| 1 |
+------+
1 row in set (0.02 sec)
mysql> select * from user where 'id' = "id";
+------+
| id |
+------+
| 2 |
| 1 |
+------+
2 rows in set (0.00 sec)
The SQL grammar should be similar. Therefore,
cur.execute("DELETE FROM opc_actuelewaardentags WHERE 'TagNaam' = %s", (var_tagnaam,))
should be
cur.execute("DELETE FROM opc_actuelewaardentags WHERE TagNaam = %s", (var_tagnaam,))
or
cur.execute("DELETE FROM opc_actuelewaardentags WHERE `TagNaam` = %s", (var_tagnaam,))
Above analyusis is error.
Simple Postgresql Statement - column name does not exists gives the answer.

RobbeM wrote: Edit 2:Query in pgadmin worked, trying to do the same thing in python
I've had the same symptoms - I could delete table rows using pgadmin or in SQL console, but Python code wouldn't work. The thing was I was accidentally creating cursor before establishing connection with postgreSQL server:
c = db_conn.cursor()
db_conn = psycopg2.connect(conn_string)
So, the solution for me was to create cursor after establishing connection with database:
db_conn = psycopg2.connect(conn_string)
c = db_conn.cursor()