We Keep Coding

MySQLdb adding character b infront of strings that have been escaped - Python

I am trying to write a simple Python script to bulk add movie titles into a local database, using the MySQLdb (mysqlclient) package. I am reading the titles from a TSV file. But when go to sanitize the inputs using MySQLdb::escape_string(), I get the character b before my string. I believe this means that SQL is interpreting it as a bit value, but when I go to execute my query I get the following error:
You have an error in your SQL syntax; check the manual that
corresponds to your MariaDB server version for the right syntax to use
near 'b'Bowery to Bagdad',1955)' at line 1"
The select statement in question:
INSERT INTO movies (imdb_id, title, release_year) VALUES ('tt0044388',b'Bowery to Bagdad',1955)
def TSV_to_SQL(file_to_open):
from MySQLdb import _mysql
db=_mysql.connect(host='localhost', user='root', passwd='', db='tutorialdb', charset='utf8')
q = """SELECT * FROM user_id"""
# MySQLdb.escape_string()
# db.query(q)
# results = db.use_result()
# print(results.fetch_row(maxrows=0, how=1))
print("starting?")
with open(file_to_open, encoding="utf8") as file:
tsv = csv.reader(file, delimiter="\t")
count = 0
for line in tsv:
if count == 10:
break
# print(MySQLdb.escape_string(line[1]))
statement = "INSERT INTO movies (imdb_id, title, release_year) VALUES ('{imdb_id}',{title},{year})\n".format(
imdb_id=line[0], title=MySQLdb.escape_string(line[1]), year=line[2])
# db.query(statement)
print(statement)
count = count + 1
I know a simple solution would be to just remove the character b from the start of the string, but I was wondering if there was a more proper way, or if I missed something in documentation.

The 'b' infront of the string represents that the string is binary encoded rather than a literal string.
If you use .encode() you will be able to get what you want.
How to convert 'binary string' to normal string in Python3?

It's more common to let the connector perform the escaping automatically, by inserting placeholders in the SQL statement and passing a sequence (conventionally a tuple) of values as the second argument to cursor.execute.
conn = MySQLdb.connect(host='localhost', user='root', passwd='', db='tutorialdb', charset='utf8')
cursor = conn.cursor()
statement = """INSERT INTO movies (imdb_id, title, release_year) VALUES (%s, %s, %s)"""
cursor.execute(statement, (line[0], line[1], line[2]))
conn.commit()
The resulting code is more portable - apart from the connection it will work with all DB-API connectors*. Dropping down to low-level functions like _mysql.connect and escape_string is unusual in Python code (though you are perfectly free to code like this if you want, of course).
* Some connection packages may use a different placeholder instead of %s, but %s seems to be the favoured placeholder for MySQL connector packages.

Not enough arguments for format string error when inserting list into database SQL Python

I'm trying to insert a list into separate columns of a database
print inserter
params = ['%s' for item in inserter]
sql_query = 'INSERT INTO tablename (coloumn1, coloumn2, coloumn3, coloumn4, coloumn5, coloumn6, coloumn7) VALUES (%s,%s,%s,%s,%s,%s,%s);' % ','.join(params)
cursor.execute(sql_query)
db.commit
But keep getting the error
not enough arguments for format string
Anyone know what I am doing wrong?

Anyone know what I am doing wrong?
You are using string interpolation in a query.
This is bad, mainly for 2 reasons:
It is erroneous as you see. The python interpreter is confused between the %s for the interpolation and the %s for the sql parameters.
It makes your code vulnerable for sql injection.
You should use a parametrized query:
sql_query = '''INSERT INTO tablename (coloumn1, coloumn2, coloumn3,
coloumn4, coloumn5, coloumn6, coloumn7)
VALUES (%s,%s,%s,%s,%s,%s,%s);'''
cursor.execute(sql_query, inserter) # assuming inserter is a tuple/list of values

Programming Error in python, How do I parse values of a tuple in a table with SQL query?

In order to parse the values of variables a and b in the field username and password respectively of table admin_details I tried this method the pseudo code is something like below:
...
...
a=4
b=8
....
....
cur.execute("INSERT INTO admin_details(username, password) VALUES('%s','%s'), %(a,b)")
....
I get the value inserted in the table as username: 4 password:8
But in order to insert the characters like** a=' john' b='snow'in the admin_details field.
I tried using tuples as below
a='john'
b='snow'
tup=['a','b']
and to insert this tuple's value a and b in the table i tried all the possible ways but still I am not able to store the variables in the table.
cur.execute("INSERT INTO admin_details(username, password) VALUES('%s','%s'), % ['a','b']")
But I get this
_mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTO INTO admin_details(username) VALUES('%s'), %('entry1.get()')' at line 1")

Do not attempt to use string formatting for constructing SQL queries. Pass query parameters in the second argument to execute() - this way you'll protect yourself against sql injection problems:
a = 'john'
b = 'snow'
cur.execute("INSERT INTO admin_details(username, password) VALUES(%s, %s)", (a, b))
Note that in this case you also don't need quotes around the placeholders in the query.
See also:
Python MySQL Parameterized Queries

I think use this way to insert mysql data should be better:
insert_sql = 'INSERT INTO admin_details(username, password) VALUES("{0}","{1}")'.format(*tup)
cur.execute(insert_sql)
conn.commit()

how to store a huge string (length 50000) in mysql using python

how to store a huge string (length 50000) in mysql using python.
I have a big string of length nearly 50000 .I have to store it into mysql.
Some suggested to store the string as a blob or text type.
Can anyone help me how to convert string into blob type
def main():
stringKey=''
stringValues=''
keys=ccv.keys() //ccv is a dictionary data structure
vectors=ccv.values() //ccv is a dictionary data structure
for key in keys:
stringKey='#'.join(key for key in keys)
for value in vectors:
stringValues='$'.join(str(value) for value in vectors)
insert(stringKey,stringValues)
print 'insert successful'
def insert(k,v):
db = mysql.connector.connect(user='root', password='abhi',
host='localhost',
database='cbir')
sql= 'INSERT INTO ccv(key,vector) VALUES(%s,%s)'
args = (k,v)
cursor=db.cursor()
cursor.execute(sql,args)
db.commit()
db.close()
error:
ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'key,vector) VALUES('27:1:8#27:1:9#25:2:11#6:9:8#6:9:9#6:9:6#6:9:7#6:9:4#6:9:5#27' at line 1

You made a small error in:
'INSERT INTO ccv(key,vector) VALUES(%s,%s)'
It should be:
"INSERT INTO ccv (`key`, `vector`) VALUES(%s, %s)"
Notice the ` denoting the column names.
As Larry reminded me the values don't have to be quoted for parameterized queries.
If the fields are already set for longtext this should work without needing to convert the data to blobs.
The problem was only related to the syntax and not the data or column types.

You don't need to convert the string, you need to change the field type in your database from varchar to longtext.

You are not associating the arguments with the SQL statement.
You want
cursor.execute(sql, args)
instead of
cursor.execute(sql)

sql = "INSERT INTO ccv(`key`,`vector`) VALUES(%s,%s)"
this works fine

Python MySQL INSERT from csv

I am working on a script to parse a csv file and generate input for a MySQL table.
I import the data via csv.reader, so every row is a list of strings.
I want to iterate over the rows and put different entries into the database.
I can get the following test to work:
sql = "INSERT INTO `testSmall` (`idtestSmall`, `column1`, `column2`) VALUES (1, 'entry1', 'entry2');"
cursor.execute (sql)
So my SQL connection works and the principle SQL syntax is ok.
I can also access the entries I want to put in there, and they are correct and of the data type I expect.
However, I don't seem to be able to use the same SQL syntax with variables within the iterations:
allData = csv.reader(open('TestTable.csv', 'rb'), delimiter=',', quotechar='|')
for row in allData:
sql = "INSERT INTO `testSmall` (`idtestSmall`, `column1`, `column2`) VALUES (row[0], row[1], row[2]);"
cursor.execute (sql)
This generates a Syntax Error:
Error 1064: You have an error in your SQL syntax; check the manual that corresponds to our MySQL server version for the right syntax to use near '[0], row[1], row[2])' at line 1
But the data types are correct and the SQL syntax is the same as in the working example...
Can anyone tell me what I'm doing wrong and how to make it work?
(In the end, I want to not only insert the pure csv entries but also derived values, which is why I'm not just using mysql bulk import.)
Thanks in advance for your help!

use:
sql = "INSERT INTO `testSmall` (`idtestSmall`, `column1`, `column2`) VALUES (?, ?, ?);"
cursor.execute (sql, (row[0], row[1], row[2]))
The questionmark is a placeholder. An extra advantage of using placeholders, is that they automatically make your input 'safe', by escaping qoutes etc.
Right now, you are using the row[0], row[1], row[2] as a string with the text "row[0], row[1], row[2]", instead of telling python to use the values of these variables.
Also, if you want to use rows of multiple lengths, or if you want to be able to easily change the size of your input list, you can dynamically create the placeholders:
sql = "INSERT INTO testSmall VALUES (%s);" % ', '.join('?' for _ in row)
cursor.execute (sql, row)

The way you are doing it, row[n]s don't refer to the variable row, but they are just a piece of string sent as it is to MySQL. (I bet you come from PHP background and expect the double quotes to replace your variables with their values).
You could do this to insert the values inside the string (any string):
sql = "INSERT INTO `testSmall` (`idtestSmall`, `column1`, `column2`) VALUES (%s, %s, %s);" % row # will map each %s to the `n`th element in `row`
(this will not work, be careful, because if row[0] is abc, that string will not be enclosed in quotes, so MySQL will not interpret it as a string). Try printing the sql variable, and copy/paste it into the mysql prompt to see if it will work.
However, when used with MySQL, you better escape these, like so:
sql = "INSERT INTO `testSmall` (`idtestSmall`, `column1`, `column2`) VALUES (%s, %s, %s);"
cursor.execute(sql, row)
You can read more in the docs.