Following the advice on this thread, I am storing my list as string type in MySQL database, but, I'm facing this error:
_mysql_exceptions.ProgrammingError: (1064, 'You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near \'foo_bar" but I can\\\'t bar.\', u\'high\', 0]]")\' at line 1')
on some list entries like this one:
var1 = 'Name'
var2 = 'Surname'
var3 = 15
var4 = [u'The Meter', [[u'Black', u'foo foo bar bar "foo_bar" but I can\'t bar', u'high', 0]]]
I figured that's because there's a double quote at the beginning of foo_bar and I am using the following code to make entries in the database:
SQL = 'INSERT INTO test_table (col1, col2, col3, col4) VALUES ("{}", "{}", "{}", "{}")'.format(var1, var2, var3, var4)
cursor.execute(SQL)
And the double quotes are not being escaped.
If i remove the double quotes from: ("{}", "{}", "{}", "{}"), I get the same error,
I tried using a different string formatting, (using %s) but that didn't work either.
Any ideas?
Do not use string formatting to interpolate SQL values. Use SQL parameters:
SQL = 'INSERT INTO test_table (col1, col2, col3, col4) VALUES (%s, %s, %s, %s)'
cursor.execute(SQL, (var1, var2, var3, var4))
Here the %s are SQL parameters; the database then takes care of escaping the values (passed in as the 2nd argument to `cursor.execute) for you.
Exactly what syntax you need to use depends on your database adapter; some use %s, others use ? for the placeholders.
You can't otherwise use Python containers, like a list, for these parameters. You'd have to serialise that to a string format first; you could use JSON for that, but then you'd also have to remember to decode the JSON into a Python string again when you query the database. That's what the answers to the other question tried to convey.
For example, if var4 is the list, you could use:
cursor.execute(SQL, (var1, var2, var3, json.dumps(var4)))
SQL = 'INSERT INTO test_table (col1, col2, col3, col4) VALUES ("{!a}", "{!a}", "{!a}", "{!a}")'.format(var1, var2, var3, str(var4))
cursor.execute(SQL)
{!a} applies ascii() and hence escapes non-ASCII characters like quotes and even emoticons.
Check out Python3 docs
Related
I have the following Python code:
cursor.execute("INSERT INTO table VALUES var1, var2, var3,")
where var1 is an integer, var2 and var3 are strings.
How can I write the variable names without Python including them as part of the query text?
cursor.execute("INSERT INTO table VALUES (%s, %s, %s)", (var1, var2, var3))
Note that the parameters are passed as a tuple.
The database API does proper escaping and quoting of variables. Be careful not to use the string formatting operator (%), because
It does not do any escaping or quoting.
It is prone to uncontrolled string format attacks e.g. SQL injection.
Different implementations of the Python DB-API are allowed to use different placeholders, so you'll need to find out which one you're using -- it could be (e.g. with MySQLdb):
cursor.execute("INSERT INTO table VALUES (%s, %s, %s)", (var1, var2, var3))
or (e.g. with sqlite3 from the Python standard library):
cursor.execute("INSERT INTO table VALUES (?, ?, ?)", (var1, var2, var3))
or others yet (after VALUES you could have (:1, :2, :3) , or "named styles" (:fee, :fie, :fo) or (%(fee)s, %(fie)s, %(fo)s) where you pass a dict instead of a map as the second argument to execute). Check the paramstyle string constant in the DB API module you're using, and look for paramstyle at http://www.python.org/dev/peps/pep-0249/ to see what all the parameter-passing styles are!
Many ways. DON'T use the most obvious one (%s with %) in real code, it's open to attacks.
Here copy-paste'd from pydoc of sqlite3:
# Never do this -- insecure!
symbol = 'RHAT'
cur.execute("SELECT * FROM stocks WHERE symbol = '%s'" % symbol)
# Do this instead
t = ('RHAT',)
cur.execute('SELECT * FROM stocks WHERE symbol=?', t)
print(cur.fetchone())
# Larger example that inserts many records at a time
purchases = [('2006-03-28', 'BUY', 'IBM', 1000, 45.00),
('2006-04-05', 'BUY', 'MSFT', 1000, 72.00),
('2006-04-06', 'SELL', 'IBM', 500, 53.00),
]
cur.executemany('INSERT INTO stocks VALUES (?,?,?,?,?)', purchases)
More examples if you need:
# Multiple values single statement/execution
c.execute('SELECT * FROM stocks WHERE symbol=? OR symbol=?', ('RHAT', 'MSO'))
print c.fetchall()
c.execute('SELECT * FROM stocks WHERE symbol IN (?, ?)', ('RHAT', 'MSO'))
print c.fetchall()
# This also works, though ones above are better as a habit as it's inline with syntax of executemany().. but your choice.
c.execute('SELECT * FROM stocks WHERE symbol=? OR symbol=?', 'RHAT', 'MSO')
print c.fetchall()
# Insert a single item
c.execute('INSERT INTO stocks VALUES (?,?,?,?,?)', ('2006-03-28', 'BUY', 'IBM', 1000, 45.00))
http://www.amk.ca/python/writing/DB-API.html
Be careful when you simply append values of variables to your statements:
Imagine a user naming himself ';DROP TABLE Users;' --
That's why you need to use SQL escaping, which Python provides for you when you use cursor.execute in a decent manner. Example in the URL is:
cursor.execute("insert into Attendees values (?, ?, ?)", (name, seminar, paid))
The syntax for providing a single value can be confusing for inexperienced Python users.
Given the query
INSERT INTO mytable (fruit) VALUES (%s)
Generally*, the value passed to cursor.execute must wrapped in an ordered sequence such as a tuple or list even though the value itself is a singleton, so we must provide a single element tuple, like this: (value,).
cursor.execute("""INSERT INTO mytable (fruit) VALUES (%s)""", ('apple',))
Passing a single string
cursor.execute("""INSERT INTO mytable (fruit) VALUES (%s)""", ('apple'))
will result in an error which varies by the DB-API connector, for example
psycopg2:
TypeError: not all arguments converted during string formatting
sqlite3
sqlite3.ProgrammingError: Incorrect number of bindings supplied. The current statement uses 1, and there are 5 supplied
mysql.connector
mysql.connector.errors.ProgrammingError: 1064 (42000): You have an error in your SQL syntax;
* The pymysql connector handles a single string parameter without erroring. However it's better to wrap the string in a tuple even if it's a single because
you won't need to change the code if you switch connector package
you keep a consistent mental model of the query parameters being a sequence of objects rather than a single object.
I have the following Python code:
cursor.execute("INSERT INTO table VALUES var1, var2, var3,")
where var1 is an integer, var2 and var3 are strings.
How can I write the variable names without Python including them as part of the query text?
cursor.execute("INSERT INTO table VALUES (%s, %s, %s)", (var1, var2, var3))
Note that the parameters are passed as a tuple.
The database API does proper escaping and quoting of variables. Be careful not to use the string formatting operator (%), because
It does not do any escaping or quoting.
It is prone to uncontrolled string format attacks e.g. SQL injection.
Different implementations of the Python DB-API are allowed to use different placeholders, so you'll need to find out which one you're using -- it could be (e.g. with MySQLdb):
cursor.execute("INSERT INTO table VALUES (%s, %s, %s)", (var1, var2, var3))
or (e.g. with sqlite3 from the Python standard library):
cursor.execute("INSERT INTO table VALUES (?, ?, ?)", (var1, var2, var3))
or others yet (after VALUES you could have (:1, :2, :3) , or "named styles" (:fee, :fie, :fo) or (%(fee)s, %(fie)s, %(fo)s) where you pass a dict instead of a map as the second argument to execute). Check the paramstyle string constant in the DB API module you're using, and look for paramstyle at http://www.python.org/dev/peps/pep-0249/ to see what all the parameter-passing styles are!
Many ways. DON'T use the most obvious one (%s with %) in real code, it's open to attacks.
Here copy-paste'd from pydoc of sqlite3:
# Never do this -- insecure!
symbol = 'RHAT'
cur.execute("SELECT * FROM stocks WHERE symbol = '%s'" % symbol)
# Do this instead
t = ('RHAT',)
cur.execute('SELECT * FROM stocks WHERE symbol=?', t)
print(cur.fetchone())
# Larger example that inserts many records at a time
purchases = [('2006-03-28', 'BUY', 'IBM', 1000, 45.00),
('2006-04-05', 'BUY', 'MSFT', 1000, 72.00),
('2006-04-06', 'SELL', 'IBM', 500, 53.00),
]
cur.executemany('INSERT INTO stocks VALUES (?,?,?,?,?)', purchases)
More examples if you need:
# Multiple values single statement/execution
c.execute('SELECT * FROM stocks WHERE symbol=? OR symbol=?', ('RHAT', 'MSO'))
print c.fetchall()
c.execute('SELECT * FROM stocks WHERE symbol IN (?, ?)', ('RHAT', 'MSO'))
print c.fetchall()
# This also works, though ones above are better as a habit as it's inline with syntax of executemany().. but your choice.
c.execute('SELECT * FROM stocks WHERE symbol=? OR symbol=?', 'RHAT', 'MSO')
print c.fetchall()
# Insert a single item
c.execute('INSERT INTO stocks VALUES (?,?,?,?,?)', ('2006-03-28', 'BUY', 'IBM', 1000, 45.00))
http://www.amk.ca/python/writing/DB-API.html
Be careful when you simply append values of variables to your statements:
Imagine a user naming himself ';DROP TABLE Users;' --
That's why you need to use SQL escaping, which Python provides for you when you use cursor.execute in a decent manner. Example in the URL is:
cursor.execute("insert into Attendees values (?, ?, ?)", (name, seminar, paid))
The syntax for providing a single value can be confusing for inexperienced Python users.
Given the query
INSERT INTO mytable (fruit) VALUES (%s)
Generally*, the value passed to cursor.execute must wrapped in an ordered sequence such as a tuple or list even though the value itself is a singleton, so we must provide a single element tuple, like this: (value,).
cursor.execute("""INSERT INTO mytable (fruit) VALUES (%s)""", ('apple',))
Passing a single string
cursor.execute("""INSERT INTO mytable (fruit) VALUES (%s)""", ('apple'))
will result in an error which varies by the DB-API connector, for example
psycopg2:
TypeError: not all arguments converted during string formatting
sqlite3
sqlite3.ProgrammingError: Incorrect number of bindings supplied. The current statement uses 1, and there are 5 supplied
mysql.connector
mysql.connector.errors.ProgrammingError: 1064 (42000): You have an error in your SQL syntax;
* The pymysql connector handles a single string parameter without erroring. However it's better to wrap the string in a tuple even if it's a single because
you won't need to change the code if you switch connector package
you keep a consistent mental model of the query parameters being a sequence of objects rather than a single object.
I have the following Python code:
cursor.execute("INSERT INTO table VALUES var1, var2, var3,")
where var1 is an integer, var2 and var3 are strings.
How can I write the variable names without Python including them as part of the query text?
cursor.execute("INSERT INTO table VALUES (%s, %s, %s)", (var1, var2, var3))
Note that the parameters are passed as a tuple.
The database API does proper escaping and quoting of variables. Be careful not to use the string formatting operator (%), because
It does not do any escaping or quoting.
It is prone to uncontrolled string format attacks e.g. SQL injection.
Different implementations of the Python DB-API are allowed to use different placeholders, so you'll need to find out which one you're using -- it could be (e.g. with MySQLdb):
cursor.execute("INSERT INTO table VALUES (%s, %s, %s)", (var1, var2, var3))
or (e.g. with sqlite3 from the Python standard library):
cursor.execute("INSERT INTO table VALUES (?, ?, ?)", (var1, var2, var3))
or others yet (after VALUES you could have (:1, :2, :3) , or "named styles" (:fee, :fie, :fo) or (%(fee)s, %(fie)s, %(fo)s) where you pass a dict instead of a map as the second argument to execute). Check the paramstyle string constant in the DB API module you're using, and look for paramstyle at http://www.python.org/dev/peps/pep-0249/ to see what all the parameter-passing styles are!
Many ways. DON'T use the most obvious one (%s with %) in real code, it's open to attacks.
Here copy-paste'd from pydoc of sqlite3:
# Never do this -- insecure!
symbol = 'RHAT'
cur.execute("SELECT * FROM stocks WHERE symbol = '%s'" % symbol)
# Do this instead
t = ('RHAT',)
cur.execute('SELECT * FROM stocks WHERE symbol=?', t)
print(cur.fetchone())
# Larger example that inserts many records at a time
purchases = [('2006-03-28', 'BUY', 'IBM', 1000, 45.00),
('2006-04-05', 'BUY', 'MSFT', 1000, 72.00),
('2006-04-06', 'SELL', 'IBM', 500, 53.00),
]
cur.executemany('INSERT INTO stocks VALUES (?,?,?,?,?)', purchases)
More examples if you need:
# Multiple values single statement/execution
c.execute('SELECT * FROM stocks WHERE symbol=? OR symbol=?', ('RHAT', 'MSO'))
print c.fetchall()
c.execute('SELECT * FROM stocks WHERE symbol IN (?, ?)', ('RHAT', 'MSO'))
print c.fetchall()
# This also works, though ones above are better as a habit as it's inline with syntax of executemany().. but your choice.
c.execute('SELECT * FROM stocks WHERE symbol=? OR symbol=?', 'RHAT', 'MSO')
print c.fetchall()
# Insert a single item
c.execute('INSERT INTO stocks VALUES (?,?,?,?,?)', ('2006-03-28', 'BUY', 'IBM', 1000, 45.00))
http://www.amk.ca/python/writing/DB-API.html
Be careful when you simply append values of variables to your statements:
Imagine a user naming himself ';DROP TABLE Users;' --
That's why you need to use SQL escaping, which Python provides for you when you use cursor.execute in a decent manner. Example in the URL is:
cursor.execute("insert into Attendees values (?, ?, ?)", (name, seminar, paid))
The syntax for providing a single value can be confusing for inexperienced Python users.
Given the query
INSERT INTO mytable (fruit) VALUES (%s)
Generally*, the value passed to cursor.execute must wrapped in an ordered sequence such as a tuple or list even though the value itself is a singleton, so we must provide a single element tuple, like this: (value,).
cursor.execute("""INSERT INTO mytable (fruit) VALUES (%s)""", ('apple',))
Passing a single string
cursor.execute("""INSERT INTO mytable (fruit) VALUES (%s)""", ('apple'))
will result in an error which varies by the DB-API connector, for example
psycopg2:
TypeError: not all arguments converted during string formatting
sqlite3
sqlite3.ProgrammingError: Incorrect number of bindings supplied. The current statement uses 1, and there are 5 supplied
mysql.connector
mysql.connector.errors.ProgrammingError: 1064 (42000): You have an error in your SQL syntax;
* The pymysql connector handles a single string parameter without erroring. However it's better to wrap the string in a tuple even if it's a single because
you won't need to change the code if you switch connector package
you keep a consistent mental model of the query parameters being a sequence of objects rather than a single object.
At the moment, I'm trying to use query a mySQL database with Python variables. I understand the convention for using variables in a query are as follows:
curr.execute("INSERT INTO table VALUES (%s, %s, %s)", (var1, var2, var3))
However, the query I want to execute is as follows:
SELECT * FROM table1
WHERE City = (cityName)
AND AdmissionDate BETWEEN (startDate) AND (endDate)
I'm not sure how the variables would need to be formatted, whether they all need to be in the 1 tuple or I can format it as I would in a printf statement so I'm not sure which of the below would be correct:
curr.execute("SELECT * FROM table1
WHERE City = (%s)
AND AdmissionDate BETWEEN (%s) AND (%s)",(cityName, startDate, endDate))
curr.execute("SELECT * FROM table1
WHERE City = (%s)
AND AdmissionDate BETWEEN (%s) AND (%s)", (cityName), (startDate), (endDate))
Any tips would be greatly appreciated. Thanks in advance.
I have a problem with MySQL and Python's MySQLdb when I try to INSERT more than one variable.
I have a table wordurl with three fields. The first one is an auto_increment ID, second and third should hold the values. Second and third fields are named word_id and url_id.
This is the code.
cursor.execute("INSERT INTO wordurl (word_id, url_id) VALUES (%s, %s)", (word_temp_id, url_temp_id))
When I try to INSERT only one value the code works, two not.
Error message:
(1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '),), (('2',),))' at line 1")
I also tried it with tripple ticks around the statement, the variables with and without bracket, without the field names and with the first id field included. I also tried with C-style-printf-stuff % (which isn't clever!). Nothing worked.
And you, glorious people on stackoverflow, you are my last hope :)
MySQL-Server is 5.5.9 on FreeBSD 8.2. Python version is 2.7:82508 on OSX Lion
Thanks in advance!
Steffen
UPDATE: I use cursor.fetchall() and cursor.fetchone() to get the IDs. Maybe this information is important.
Regarding your update:
AFAIK:
fetchall() returns a tuple with tuples inside. Those tuples inside are what you would get from fetchone(). You can image it like
fetchall() = (fetchone(), fetchtwo(), ...)
And fetchone also returns a tuple which has the actual variable values. This is why you can't simply insert the return value of fetchall().
Just to clarify: the insert statement you pasted, look like this when you fill in the value for max_price:
c.execute("""SELECT spam, eggs, sausage FROM breakfast WHERE price < %s""", (5,))
so the first %s is replaced with the first element of the tuple (5). Your insert statement looks like this:
cursor.execute("INSERT INTO wordurl (word_id, url_id) VALUES (%s, %s)", (((233L,),), ((3L,),)))
I can't make it more clear.
Did you try this:
cursor.execute("INSERT INTO wordurl (word_id, url_id) VALUES ('%s', '%s')" %(word_temp_id, url_temp_id))
It depends on what variables you're trying to insert, but on strings this is needed.
Notice the difference:
>>> word_temp_id = "bla1"
>>> url_temp_id = "bla2"
>>> query = "INSERT INTO wordurl (word_id, url_id) VALUES (%s, %s)" %(word_temp_id, url_temp_id)
>>> print query
INSERT INTO wordurl (word_id, url_id) VALUES (bla1, bla2)
>>> query = "INSERT INTO wordurl (word_id, url_id) VALUES ('%s', '%s')" %(word_temp_id, url_temp_id)
>>> print query
INSERT INTO wordurl (word_id, url_id) VALUES ('bla1', 'bla2')