I've been trying to parse a text file (opened with parameter encoding='utf8') and insert extracted values into an mdb database using pyodbc module.
I have tried the code below:
for line in fp:
tokens = line.split('\t')
tokens[4] = tokens[4][:len(tokens[4])-1] #to avoid the \n
tokens[1] = tokens[1][1:] #to remove the 'u' from the beginning of utf8 characters like u'\u0622'
content = conn.execute("INSERT INTO Entries (PForm, WForm, Code, Freq, Pattern) VALUES ("+tokens[0]+","+tokens[1]+","+tokens[2]+","+tokens[3]+","+tokens[4]+")")
conn.commit()
and received the following error:
Error: ('07002', '[07002] [Microsoft][ODBC Microsoft Access Driver] Too few parameters. Expected 4. (-3010) (SQLExecDirectW)')
P.S. the first line of my file is: آ 'A Ab 1 S
And the other lines are of the same format.
Your comments will be appreciated :)
You don't put quotes around the strings which you want to insert. Assuming the "Freq" row is of type INTEGER:
stmt = """
INSERT INTO Entries (PForm, WForm, Code, Freq, Pattern)
VALUES ('%s', '%s', '%s', %s, '%s')
"""
params = tuple(t for t in tokens)
conn.execute(stmt % params)
But anyway, you shouldn't be formatting an INSERT statement like this. Doesn't the library you're using provide a facility to parameterize statements ? Something like this:
conn.execute("INSERT INTO Foo VALUES (?, ?, ?)", (foo, bar, baz))
Related
def quantity():
i = 0
x = 1
file = open("john.txt", "r")
while i < 5000:
for line in file:
c.execute("INSERT INTO test (playerNAME, playerID) VALUES ("+line+", "+str(x)+")")
conn.commit()
x = random.randint(100,10000000000000000)
i += 1
I try to iterate through the John.txt file and insert each value into a table. The first word in the txt file is "abc123". When I run this code there is an error: sqlite3.OperationalError: no such column: abc123
I can get the code to enter the random numbers into playerID but I can't get the txt file query to work...
You need single quotes around the string.
c.execute("INSERT INTO test (playerNAME, playerID) VALUES ('"+line+"', "+str(x)+")")
Otherwise it tries to interpret it as a sql expression and looks for the named column.
More generally you should use parameters or sanitize the incoming data from the file for safety against sql insertion. Even if you trust this particular file. It's a good habit.
c.execute("INSERT INTO test (playerName, playerID) VALUES (?, ?)", (line, x))
Details are here and here is why it's important.
Formatting sql queries via string concatenation is very bad practice.
Variable bindging should always be used:
c.execute("INSERT INTO test (playerNAME, playerID) VALUES (?, ?)", [line, x])
In your case the line probably contains spaces or any punctuation mark.
The sqlite's error string is misleading, though.
psycopg2 complains when inserting multiple words, empty strings, and empty arrays:
name = "Meal Rounds"
description = ""
sizes = []
cur.execute(""" INSERT INTO items (name, description, sizes) VALUES (%s, %s, %s)""" % (name, description, sizes))
Errors:
# Multi word error
psycopg2.ProgrammingError: syntax error at or near "Rounds"
LINE 1: ... (name, description, sizes) VALUES (Meal Rounds, , ...
^
# Empty string error
psycopg2.ProgrammingError: syntax error at or near ","
LINE 1: ...scription, sizes) VALUES ("Meal Rounds", , [], Fals...
^
# Empty array error
psycopg2.ProgrammingError: syntax error at or near "["
LINE 1: ...n, sizes) VALUES ("Meal Rounds", "None", [], False)...
^
I can get around the multi word error by escaping:
""" INSERT INTO items (name, description, sizes) VALUES (\"%s\", \"%s\", %s)"""
But for tables with 15+ columns, escaping each one is a pain. Does psycopg2 not handle this in an easier fashion? It will still throw errors for empty strings though.
How do I insert multiple words more efficiently, and how to insert empty strings and arrays?
Here is what psql prints out on my columns:
name | character varying(255) |
description | character varying(255) |
sizes | integer[] |
Your call to execute is creating a string with Python string substitution, which is turning out to be invalid SQL. You should be using the parameter substitution provided by the Python DB API:
https://www.python.org/dev/peps/pep-0249/#id15
To call execute using parameter substitution, you pass it two arguments. The first is the query with parameter strings which are database dependent. Psycopg2 uses "pyformat" paramstyle so your query will work as written. The second argument should be the variables you want to substitute into the query. The database driver will handle all the quoting/escaping you need. So your call to execute should be
cur.execute("""INSERT INTO items (name, description, sizes) VALUES (%s, %s, %s)""", (name, description, sizes))
I'm quite new to python and mysql.
When I try to insert some records with fields of string, I failed to insert them into mysql because it always report error like: ProgrammingError: (1064, 'You have an error in your SQL syntax.
I am quite confused.
Occasionlly, I find it works by add additional "" to the string field, like the following:
history_market_table_name = 'my_test'
f = ['field_a', 'field_b', 'field_c', 'field_d']
r = ['a', 'b', 'c', 'd']
my_time = "'" + str(datetime.now()) + "'"
target_sql = "insert into %s (time, %s, %s, %s, %s) values(%s, %s, %s, %s, %s)" % (
history_market_table_name, f[0], f[1], f[2], f[3], my_time, repr(r[0]), repr(r[1]), repr(r[2]), repr(r[3]))
Sorry the codes are quite tedious.
Moreover, there must be something wrong with my method -- it couldn't be so stupid to insert fields with string type!
Look at the my_time variable, how ridiculous.
Any one please explain how to correctly insert string into mysql with python.
I am quit new.
So please give a correct answer and detailed explaination is highly appreciated.
Change your format string to
"insert into %s (time, %s, %s, %s, %s) values(%s, '%s', '%s', '%s', '%s')"
I.e, single quotes around the otherwise-unquoted string values you're inserting -- and lose the repr calls after the %. This will still fail if you're inserting a string value which contains a ' character... there are better ways but they don't let you parameterize the table name and field names (which is a truly peculiar requirement...), only the values.
Added: as the OP requires "a direct expression for the 'target_sql' variable", here it is:
fmt_str = "insert into %s (time, %s, %s, %s, %s) values(%s, '%s', '%s', '%s', '%s')"
target_sql = fmt_str % (history_market_table_name, f[0], f[1], f[2], f[3],
my_time, r[0], r[1], r[2], r[3])
I think it's more readable split up this way. As for "some documents for such things" which the OP also requires, MySQL has such documents with every version (and the OP didn't mention which version he or she uses) and so does Python (ditto, ditto).
The "better way" (which however don't allow parameterizing table and field names) involve using "placeholders" in lieu of value in the SQL string, rather than string substitution.
Here,
sql = "insert into my_test (time, field_a, field_b, field_c, field_d) values(%s, %s, %s, %s, %s)'
(with hard-coded, not parameterized, table and field names), and later, for a given cursor opened on the MySql DB,
cursor.execute(sql, (
str(datetime.now()), r[0], r[1], r[2], r[3]))
with no worries about quoting nor about any potential for a "sql injection" attack.
I'm wondering if you can help me. I'm trying to change the value in each column if the text matches a corresponding keyword. This is the loop:
for i in range(0, 20, 1):
cur.execute("UPDATE table SET %s = 1 WHERE text rlike %s") %(column_names[i], search_terms[i])
The MySQL command works fine on its own, but not when I put it in the loop. It's giving an error at the first %s
Does anyone have any insights?
This is the error:
_mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '%s = 1 WHERE text rlike %s' at line 1")
Column names looks like
column_names = ["col1","col2","col3"...]
Search terms look like
search_terms = ["'(^| |.|-)word1[;:,. ?-]'","'(^| |.|-)word2[;:,. ?-]'",...]
The right way to do this is to give values to Python, which will quote things correctly.
adapted from voyager's post:
for i in range(0, 20, 1):
cur.execute("UPDATE table SET {} = 1 WHERE text rlike %s".format(column_names[i]),
(search_terms[i],),
)
In this case it's confusing because the column_name isn't a value, it's part of the table structure, so it's inserted using good old string formatting. The search_term is a value, so is passed to cursor.execute() for correct, safe quoting.
(Don't use string manipulation to add the quotes -- you're exposing yourself to SQL injection.)
Missing quotes and wrong parenthesis placement...
for i in range(0, 20, 1):
cur.execute("UPDATE table SET %s = 1 WHERE text rlike '%s'" %(column_names[i], search_terms[i]))
# ^ ^
# (-----------------------------------------------------------------------------------)
Please note, this is not the right way of doing this, if your string may contain quotes by itself...
What about that instead:
for i in range(0, 20, 1):
cur.execute("UPDATE table SET %s = 1 WHERE text rlike ?" % (column_names[i],),
(search_terms[i],))
This uses the % operator to set the column name, but uses an executes parameter to bind the data, letting the DB driver escape all characters that need so.
Here is my code (currently):
conn = sqlite3.connect(db)
conn.text_factory = str #bugger 8-bit bytestrings
cur = conn.cursor()
reader = csv.reader(open(csvfile, "rU"), delimiter = '\t')
for Number, Name, Message, Datetime, Type in reader:
# populate subscriber table
if str(Number)[0] == '1': # errors on this line
tmpNumber = str(Number)[1:]
Number = int(tmpNumber)
cur.execute('INSERT OR IGNORE INTO subscriber (name, phone_number) VALUES (?,?)', (Name, Number))
cur.close()
conn.close()
It returns this error on the line commented to indicate where the error lies:
IndexError: string index out of range
All of the numbers have values, but if the phone number starts with a 1 I want to remove the 1 before inserting it into the database. Why won't this work? I've converted it to a string before trying to reference the first character, so I don't understand why this isn't working.
Seems like you are getting an empty string. Try replacing your if statement with the following and see if it works.
if str(Number).startswith('1'):
(Edited to reflect #kindall 's suggestion of using startswith instead of slicing [:1]).