I'm quite new to python and mysql.
When I try to insert some records with fields of string, I failed to insert them into mysql because it always report error like: ProgrammingError: (1064, 'You have an error in your SQL syntax.
I am quite confused.
Occasionlly, I find it works by add additional "" to the string field, like the following:
history_market_table_name = 'my_test'
f = ['field_a', 'field_b', 'field_c', 'field_d']
r = ['a', 'b', 'c', 'd']
my_time = "'" + str(datetime.now()) + "'"
target_sql = "insert into %s (time, %s, %s, %s, %s) values(%s, %s, %s, %s, %s)" % (
history_market_table_name, f[0], f[1], f[2], f[3], my_time, repr(r[0]), repr(r[1]), repr(r[2]), repr(r[3]))
Sorry the codes are quite tedious.
Moreover, there must be something wrong with my method -- it couldn't be so stupid to insert fields with string type!
Look at the my_time variable, how ridiculous.
Any one please explain how to correctly insert string into mysql with python.
I am quit new.
So please give a correct answer and detailed explaination is highly appreciated.
Change your format string to
"insert into %s (time, %s, %s, %s, %s) values(%s, '%s', '%s', '%s', '%s')"
I.e, single quotes around the otherwise-unquoted string values you're inserting -- and lose the repr calls after the %. This will still fail if you're inserting a string value which contains a ' character... there are better ways but they don't let you parameterize the table name and field names (which is a truly peculiar requirement...), only the values.
Added: as the OP requires "a direct expression for the 'target_sql' variable", here it is:
fmt_str = "insert into %s (time, %s, %s, %s, %s) values(%s, '%s', '%s', '%s', '%s')"
target_sql = fmt_str % (history_market_table_name, f[0], f[1], f[2], f[3],
my_time, r[0], r[1], r[2], r[3])
I think it's more readable split up this way. As for "some documents for such things" which the OP also requires, MySQL has such documents with every version (and the OP didn't mention which version he or she uses) and so does Python (ditto, ditto).
The "better way" (which however don't allow parameterizing table and field names) involve using "placeholders" in lieu of value in the SQL string, rather than string substitution.
Here,
sql = "insert into my_test (time, field_a, field_b, field_c, field_d) values(%s, %s, %s, %s, %s)'
(with hard-coded, not parameterized, table and field names), and later, for a given cursor opened on the MySql DB,
cursor.execute(sql, (
str(datetime.now()), r[0], r[1], r[2], r[3]))
with no worries about quoting nor about any potential for a "sql injection" attack.
Related
Suppose I have a list:
row = [u'28d07ef48e40', u'373ac79f615f', u'a3ec4faddbec', u'c0195f9568c6', u'cc4ebc7b826c', u'ccdfdb826c', u'cc4fa826c', u'cc4eeesb826c', u'ccfesb826c']
my print is
fw.write("%s %s <%s> [%s] %s %s (%s) %s: %s" %(str(row[0]),str(row[1]),str(row[2]),str(row[3]),str(row[4]),str(row[5]),str(row[6]),str(row[7]),str(row[8])))
How can I simplify this python print?
you can try using the join function
row = [u'28d07ef48e40', u'373ac79f615f', u'a3ec4faddbec', u'c0195f9568c6', u'cc4ebc7b826c', u'ccdfdb826c', u'cc4fa826c', u'cc4eeesb826c', u'ccfesb826c']
fw.write(' '.join(row))
this works beacuse the join function joins everything in the list into the string
Depending on your Python version, you can:
For Python 3.6 and later use new format-strings, which are the most-recommended approach:
row = [u'28d07ef48e40', u'373ac79f615f', u'a3ec4faddbec', u'c0195f9568c6', u'cc4ebc7b826c', u'ccdfdb826c', u'cc4fa826c', u'cc4eeesb826c', u'ccfesb826c']
fw.write(f'{row[0]} {row[1]} [{row[2]}] ...')
For lower versions of Python 3, you can use str.format(), which is recommended over %-formatting:
fw.write('{} {} <{}> [{}] {} {} ({}) {}: {}'.format(row[0], row[1], row[2], ...)
For Python 2 you continue with %-formatting, but you dont need to call str() on arguments - it is done automatically:
fw.write("%s %s <%s> [%s] %s %s (%s) %s: %s" % tuple(row))
The format string is in it's simplest form. What you can "simplify" is the conversion to string:
"%s %s <%s> [%s] %s %s (%s) %s: %s".format(map(str, row))
map(str, row) will call str() on all elements n row, and returns a list in 2.7, and iterator in 3.7.
The *row operator will pack the list
I've been playing around with sqlite3, and I get an sqlite3.OperationalError: near "sweet": syntax error for this line of my code query_cursor.execute("INSERT INTO mcdonalds_menu VALUES(%d, %s, %f, %s, %d)" % (ids[num],names[num], price[num], descriptions[num], calories[num])) When I put in the values in 3 separate queries the code seems to work, but I'm trying to keep my code more DRY by using a for loop. The code so far:
import sqlite3
filename = sqlite3.connect("McDonalds_Menu.db")
query_cursor = filename.cursor()
def create_table():
query_cursor.execute( "CREATE TABLE mcdonalds_menu (id INTEGER, name VARCHAR(20), price DECIMAL(3, 2), description TEXT, calories INTEGER)")
ids = range(1,4)
names = ["McFlurry", "Fillet-o-Fish", "McCafe"]
price = 1.50, 2.25, 0.99
descriptions = ["Delicious sweet icecream", "Best fish in the sea", "Freshly brewed Colombian coffee"]
calories = 220, 450, 75
def data_entry():
for num in xrange(3):
query_cursor.execute("INSERT INTO mcdonalds_menu VALUES(%d, %s, %f, %s, %d)" % (ids[num], names[num], price[num], descriptions[num], calories[num]))
filename.commit()
if __name__ == "__main__":
create_table()
data_entry()
Is it possible to string format a sql query using a loop?
All the other answers relying on python's string manipulation are insecure and might not correctly escape quotes in your strings.
The best way to do it, as suggested in sqlite3 documentation, is to use the DB-API’s parameter substitution. In your example, it would look like this:
menu_items = [(1, 'McFlurry', 1.5, 'Delicious sweet icecream', 220),
(2, 'Fillet-o-Fish', 2.25, 'Best fish in the sea', 450),
(3, 'McCafe', 0.99, 'Freshly brewed Colombian coffee', 75)
]
c.executemany('INSERT INTO mcdonalds_menu VALUES (?,?,?,?,?)', menu_items)
SQL needs strings in VALUES to be quoted. Integers and floats do not need to be quoted.
In the commented output below, notice that the SQL VALUES contains unquoted strings for "Fillet-o-Fish" and "Best fish in the sea":
sql = "INSERT INTO mcdonalds_menu VALUES(%d, %s, %f, %s, %d)".format(ids[num], names[num], price[num], descriptions[num], calories[num])
# INSERT INTO mcdonalds_menu VALUES(2, Fillet-o-Fish, 2.250000, Best fish in the sea, 450)
Adding some escaped quotes around your string values produces valid SQL:
sql = "INSERT INTO mcdonalds_menu VALUES(%d, \"%s\", %f, \"%s\", %d)" % (ids[num],names[num], price[num], descriptions[num], calories[num])
# INSERT INTO mcdonalds_menu VALUES(2, "Fillet-o-Fish", 2.250000, "Best fish in the sea", 450)
With Python 3.6+ you can simplify this quoting mess with f strings. For example:
c.execute(f"select sql from sqlite_master where type='table' and name='{table_name}';")
for r in c.fetchall():
print(r)
In this snippet, the important thing to note is f preceding the sql string. This allows one to pass in variables surrounded by curly braces, in my example: '{table_name}'
psycopg2 complains when inserting multiple words, empty strings, and empty arrays:
name = "Meal Rounds"
description = ""
sizes = []
cur.execute(""" INSERT INTO items (name, description, sizes) VALUES (%s, %s, %s)""" % (name, description, sizes))
Errors:
# Multi word error
psycopg2.ProgrammingError: syntax error at or near "Rounds"
LINE 1: ... (name, description, sizes) VALUES (Meal Rounds, , ...
^
# Empty string error
psycopg2.ProgrammingError: syntax error at or near ","
LINE 1: ...scription, sizes) VALUES ("Meal Rounds", , [], Fals...
^
# Empty array error
psycopg2.ProgrammingError: syntax error at or near "["
LINE 1: ...n, sizes) VALUES ("Meal Rounds", "None", [], False)...
^
I can get around the multi word error by escaping:
""" INSERT INTO items (name, description, sizes) VALUES (\"%s\", \"%s\", %s)"""
But for tables with 15+ columns, escaping each one is a pain. Does psycopg2 not handle this in an easier fashion? It will still throw errors for empty strings though.
How do I insert multiple words more efficiently, and how to insert empty strings and arrays?
Here is what psql prints out on my columns:
name | character varying(255) |
description | character varying(255) |
sizes | integer[] |
Your call to execute is creating a string with Python string substitution, which is turning out to be invalid SQL. You should be using the parameter substitution provided by the Python DB API:
https://www.python.org/dev/peps/pep-0249/#id15
To call execute using parameter substitution, you pass it two arguments. The first is the query with parameter strings which are database dependent. Psycopg2 uses "pyformat" paramstyle so your query will work as written. The second argument should be the variables you want to substitute into the query. The database driver will handle all the quoting/escaping you need. So your call to execute should be
cur.execute("""INSERT INTO items (name, description, sizes) VALUES (%s, %s, %s)""", (name, description, sizes))
I'm wondering if you can help me. I'm trying to change the value in each column if the text matches a corresponding keyword. This is the loop:
for i in range(0, 20, 1):
cur.execute("UPDATE table SET %s = 1 WHERE text rlike %s") %(column_names[i], search_terms[i])
The MySQL command works fine on its own, but not when I put it in the loop. It's giving an error at the first %s
Does anyone have any insights?
This is the error:
_mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '%s = 1 WHERE text rlike %s' at line 1")
Column names looks like
column_names = ["col1","col2","col3"...]
Search terms look like
search_terms = ["'(^| |.|-)word1[;:,. ?-]'","'(^| |.|-)word2[;:,. ?-]'",...]
The right way to do this is to give values to Python, which will quote things correctly.
adapted from voyager's post:
for i in range(0, 20, 1):
cur.execute("UPDATE table SET {} = 1 WHERE text rlike %s".format(column_names[i]),
(search_terms[i],),
)
In this case it's confusing because the column_name isn't a value, it's part of the table structure, so it's inserted using good old string formatting. The search_term is a value, so is passed to cursor.execute() for correct, safe quoting.
(Don't use string manipulation to add the quotes -- you're exposing yourself to SQL injection.)
Missing quotes and wrong parenthesis placement...
for i in range(0, 20, 1):
cur.execute("UPDATE table SET %s = 1 WHERE text rlike '%s'" %(column_names[i], search_terms[i]))
# ^ ^
# (-----------------------------------------------------------------------------------)
Please note, this is not the right way of doing this, if your string may contain quotes by itself...
What about that instead:
for i in range(0, 20, 1):
cur.execute("UPDATE table SET %s = 1 WHERE text rlike ?" % (column_names[i],),
(search_terms[i],))
This uses the % operator to set the column name, but uses an executes parameter to bind the data, letting the DB driver escape all characters that need so.
I've been trying to parse a text file (opened with parameter encoding='utf8') and insert extracted values into an mdb database using pyodbc module.
I have tried the code below:
for line in fp:
tokens = line.split('\t')
tokens[4] = tokens[4][:len(tokens[4])-1] #to avoid the \n
tokens[1] = tokens[1][1:] #to remove the 'u' from the beginning of utf8 characters like u'\u0622'
content = conn.execute("INSERT INTO Entries (PForm, WForm, Code, Freq, Pattern) VALUES ("+tokens[0]+","+tokens[1]+","+tokens[2]+","+tokens[3]+","+tokens[4]+")")
conn.commit()
and received the following error:
Error: ('07002', '[07002] [Microsoft][ODBC Microsoft Access Driver] Too few parameters. Expected 4. (-3010) (SQLExecDirectW)')
P.S. the first line of my file is: آ 'A Ab 1 S
And the other lines are of the same format.
Your comments will be appreciated :)
You don't put quotes around the strings which you want to insert. Assuming the "Freq" row is of type INTEGER:
stmt = """
INSERT INTO Entries (PForm, WForm, Code, Freq, Pattern)
VALUES ('%s', '%s', '%s', %s, '%s')
"""
params = tuple(t for t in tokens)
conn.execute(stmt % params)
But anyway, you shouldn't be formatting an INSERT statement like this. Doesn't the library you're using provide a facility to parameterize statements ? Something like this:
conn.execute("INSERT INTO Foo VALUES (?, ?, ?)", (foo, bar, baz))