I've been trying to insert a large string into an MySQL database using pythons mysql.connector. The problem I'm having is that long strings are getting cut off at some point when using prepared statements. I'm currently using MySQL Connector/Python that is available on MySQL.com. I used the following code do duplicate the problem I'm having.
db = mysql.connector.connect(**creditials)
cursor = db.cursor()
value = []
for x in range(0, 2000):
value.append(str(x+1))
value = " ".join(value)
cursor.execute("""
CREATE TABLE IF NOT EXISTS test (
pid VARCHAR(50),
name VARCHAR(120),
data LONGTEXT,
PRIMARY KEY(pid)
)
""")
db.commit()
#this works as expected
print("Test 1")
cursor.execute("REPLACE INTO test (pid, name, data) VALUES ('try 1', 'Description', '{0}')".format(value))
db.commit()
cursor.close()
#this does not work
print("Test 2")
cursor = db.cursor(prepared=True)
cursor.execute("""REPLACE INTO test (pid, name, data) VALUE (?, ?, ?)""", ('try 2', 'Description2', value))
db.commit()
cursor.close()
Test 1 works as expected and stores all the numbers up to 2000, but test 2 get cut off right after number 65. I would rather use prepared statements than trying to sanitize incoming strings myself. Any help appreciated.
Extra information:
Computer: Windows 7 64 bit
Python: Tried on both python 3.4 and 3.3
MYSQL: 5.6.17 (Came with WAMP)
Library: MySQL Connector/Python
When MySQL Connector driver processes prepared statements, it's using a lower-level binary protocol to communicate values to the server individually. As such, it's telling the server whether the values are INTs or VARCHARs or TEXT, etc. It's not particularly smart about it, and this "behavior" is the result. In this case, it sees that the value is a Python string value and tells MySQL that it's a VARCHAR value. The VARCHAR value has a string length limit that affects the amount of data be sent to the server. What's worse, the interaction between the long value and the limited data type length can yield some strange behavior.
Ultimately, you have a few options:
Use a file-link object for your string
MySQL Connector treats files and file-like objects as BLOBs and TEXTs (depending on whether the file is open in binary or non-binary mode, respectively). You can leverage this to get the behavior you desire.
import StringIO
...
cursor = db.cursor(prepared=True)
cursor.execute("""REPLACE INTO test (pid, name, data) VALUES (?, ?, ?)""",
('try 2', 'Description', StringIO.String(value)))
cursor.close()
db.commit()
Don't use MySQL Connector prepared statements
If you don't use the prepared=True clause to your cursor creation statement, it will generate full valid SQL statements for each execution. You're not really losing too much by avoiding MySQL prepared statements in this context. You do need to pass your SQL statements in a slightly different form to get proper placeholder sanitization behavior.
cursor = db.cursor()
cursor.execute("""REPLACE INTO test (pid, name, data) VALUES (%s, %s, %s)""",
('try 2', 'Description', value))
cursor.close()
db.commit()
Use another MySQL driver
There are a couple different Python MySQL drivers:
MySQLdb
oursql
Related
I am inserting thousands of rows, timing and speed is very important. I have found through benchmarking that postgres can ingest my rows faster using execute() instead of executemany()
This works well for me:
...
def insert(self, table, columns, values):
conn = self.connectionPool.getconn()
conn.autocommit = True
try:
with conn.cursor() as cursor:
query = (
f'INSERT INTO {table} ({columns}) '
f'VALUES {values} '
f'ON CONFLICT DO NOTHING;'
).replace('[', '').replace(']', '') # Notice the replace x2 to get rid of the list brackets
print(query)
cursor.execute(query)
finally:
cursor.close()
self.connectionPool.putconn(conn)
...
self.insert('types', 'name, created_at', rows)
After the double replace, printing query returns something like this and the rows are ingested:
INSERT INTO types (name, created_at) VALUES ('TIMER', '2022-04-09 03:19:49'), ('Sequence1', '2022-04-09 03:19:49') ON CONFLICT DO NOTHING;
Is my approach secure? Is there a more pythonic implementation using execute?
No, this isn’t secure or even reliable – Python repr isn’t compatible with PostgreSQL string syntax (try some strings with single quotes, newlines, or backslashes).
Consider passing array parameters instead and using UNNEST:
cursor.execute(
"INSERT INTO types (name, created_at)"
" SELECT name, created_at FROM UNNEST (%(names)s, %(created_ats)s) AS t",
{
'names': ['TIMER', 'Sequence1', ...],
'created_ats': ['2022-04-09 03:19:49', ...],
})
This is the best solution, as the query doesn’t depend on the parameters (can be prepared and cached, statistics can be easily grouped, makes the absence of SQL injection vulnerability obvious, can easily log queries without data).
Failing that, build a query that’s only dynamic in the number of parameters, like VALUES ((%s, %s, ...), (%s, %s, ...), ...). Note that PostgreSQL has a parameter limit, so you might need to produce these in batches.
Failing that, use psycopg2.sql.Literal.
dsn_tns = cx_Oracle.makedsn('myip', '1521', service_name='myservicename')
connection = cx_Oracle.connect(user='username', password='passwd', dsn=dsn_tns)
cursor = connection.cursor()
cursor.execute("INSERT INTO tablename VALUES (7.4)")
#above works ok
cursor.execute("INSERT INTO tablename VALUES (7:4)")
#above sql raise cx_Oracle.DatabaseError: ORA-00917: missing comma
connection.commit()
connection.close()
My requirement to is to insert : to be insert in this tablename, column type of this tablename is varchar2. ( when i insert '0001' it also remove leading zeros and only insert '1' in tabename)
above code connect oracle database table and create cursor to execute sql insert query. in insert query with values 7.4 if successfully update the table but in second insert it raise error with ORA-00917.
i am using oracle version 11g and python 3.7.
Because, the value in first case can be classified into a type which is numeric which might be inserted into a string type column. Converting that into a prepared statement such as
myvar=7.4
cursor.execute("INSERT INTO tab(col) VALUES (:1)",[myvar])
that cannot be used for 7:4 which doesn't seem to be converted to a known data type even implicitly. The value should be quoted 7:4, and would be suitable to be formatted for this case such as
myvar='7:4'
cursor.execute("INSERT INTO tab(col) VALUES (:1)",[myvar.format('{}')])
As a result, prefer using this second one for both cases without having a problem.
I am receiving an error when trying to write data to a database table when using a variable for the table name that I do not get when using a static name. For some reason on the line where I insert, if I insert an integer as the column values the code runs and the table is filled, however, if I try to use a string I get a SQL syntax error
cursor = db.cursor()
cursor.execute('DROP TABLE IF EXISTS %s' %data[1])
sql ="""CREATE TABLE %s (IP TEXT, AVAILIBILITY INT)""" %data[1]
cursor.execute(sql)
for key in data[0]:
cur_ip = key.split(".")[3]
cursor.execute("""INSERT INTO %s VALUES (%s,%s)""" %(data[1],key,data[0][key]))
db.commit()
the problem is where I have %(data[1], key, data[0][key]) any ideas?
It's a little hard to analyse your problem when you don't post the actual error, and since we have to guess what your data actually is. But some general points as advise:
Using a dynamic table name is often not way DB-systems want to be used. Try thinking if the problem could be used by using a static table name and adding an additional key column to your table. Into that field you can put what you did now as a dynamic table name. This way the DB might be able to better optimize your queries, and your queries are less likely to get errors (no need to create extra tables on the fly for once, which is not a cheap thing to do. Also you would not have a need for dynamic DROP TABLE queries, which could be a security risk.
So my advice to solve your problem would be to actually work around it by trying to get rid of dynamic table names altogether.
Another problem you have is that you are using python string formatting and not parameters to the query itself. That is a security problem in itself (SQL-Injections), but also is the problem of your syntax error. When you use numbers, your expression evaluates to
INSERT INTO table_name VALUES (100, 200)
Which is valid SQL. But with strings you get
INSERT INTO table_name VALUES (Some Text, some more text)
which is not valid (since you have no quotes ' around the strings.
To get rid of your syntax problem and of the sql-injection-problem, don't add the values to the string, pass them as a list to execute():
cursor.execute("INSERT INTO table_name VALUES (%s,%s)", (key, data[0][key]))
If you must have a dynamic table name, put that in your query string first (e.g. with % formatting), and give the actual values for your query as parameters as above (since I cannot imagine that execute will accept the table name as a parameter).
To put it in some simple sample code. Right now you are trying to do it like this:
# don't do this, this won't even work!
table_name = 'some_table'
user_name = 'Peter Smith'
user_age = 47
query = "INSERT INTO %s VALUES (%s, %s)" % (table_name, user_name, user_age)
cursor.execute(query)
That creates query
INSERT INTO some_table VALUES (Peter Smith, 100)
Which cannot work, because of the unquoted string. So you needed to do:
# DON'T DO THIS, it's bad!
query = "INSERT INTO %s VALUES ('%s', %s)" % (table_name, user_name, user_age)
That's not a good idea, because you need to know where to put quotes and where not (which you will mess up at some point). Even worse, imagine a user named named Connor O'Neal. You would get a syntax error:
INSERT INTO some_table VALUES ('Connor O'Neal', 100)
(This is also the way sql-injections are used to crush your system / steal your data). So you would also need to take care of escaping the values that are strings. Getting more complicated.
Leave those problems to python and mysql, by passing the date (not the table name) as arguments to execute!
table_name = 'some_table'
user_name = 'Peter Smith'
user_age = 47
query = "INSERT INTO " + table_name + " VALUES (%s, %s)"
cursor.execute(query, (user_name, user_age))
This way you can even pass datetime objects directly. There are other ways to put the data than using %s, take a look at this examples http://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursor-execute.html (that is python3 used there, I don't know which you use - but except of the print statements it should work with python2 as well, I think).
how to store a huge string (length 50000) in mysql using python.
I have a big string of length nearly 50000 .I have to store it into mysql.
Some suggested to store the string as a blob or text type.
Can anyone help me how to convert string into blob type
def main():
stringKey=''
stringValues=''
keys=ccv.keys() //ccv is a dictionary data structure
vectors=ccv.values() //ccv is a dictionary data structure
for key in keys:
stringKey='#'.join(key for key in keys)
for value in vectors:
stringValues='$'.join(str(value) for value in vectors)
insert(stringKey,stringValues)
print 'insert successful'
def insert(k,v):
db = mysql.connector.connect(user='root', password='abhi',
host='localhost',
database='cbir')
sql= 'INSERT INTO ccv(key,vector) VALUES(%s,%s)'
args = (k,v)
cursor=db.cursor()
cursor.execute(sql,args)
db.commit()
db.close()
error:
ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'key,vector) VALUES('27:1:8#27:1:9#25:2:11#6:9:8#6:9:9#6:9:6#6:9:7#6:9:4#6:9:5#27' at line 1
You made a small error in:
'INSERT INTO ccv(key,vector) VALUES(%s,%s)'
It should be:
"INSERT INTO ccv (`key`, `vector`) VALUES(%s, %s)"
Notice the ` denoting the column names.
As Larry reminded me the values don't have to be quoted for parameterized queries.
If the fields are already set for longtext this should work without needing to convert the data to blobs.
The problem was only related to the syntax and not the data or column types.
You don't need to convert the string, you need to change the field type in your database from varchar to longtext.
You are not associating the arguments with the SQL statement.
You want
cursor.execute(sql, args)
instead of
cursor.execute(sql)
sql = "INSERT INTO ccv(`key`,`vector`) VALUES(%s,%s)"
this works fine
I am trying to insert a CLOB using the following code.
cursor = connection.cursor()
cursor.setinputsizes(HERP = cx_Oracle.CLOB)
cursor.execute("INSERT INTO myTable (FOO, BAR) VALUES (:FOO, :BAR)", FOO=val1, BAR=val2)
cursor.execute("INSERT INTO myTable2 (HERP) VALUES (:HERP)", HERP=val3)
#len(HERP) 39097
When I run the script WITHOUT cursor.setinputsizes(HERP = cx_Oracle.CLOB) it fails on the second query WITH ValueError: string data too large, when I run the script with cursor.setinputsizes(HERP = cx_Oracle.CLOB) it fails on the first query with DatabaseError: ORA-01036: illegal variable name/number. The CLOB I am trying to insert contains a code snippet (i.e. it has a lot of semi colons, commas and parenthesis), "string".decode("ascii") returns u'string', so unicode isn't the problem... right? I don't know if either of these things are problems. The field in the database is a currently a CLOB, however I have tried it with an NCLOB and the behavior did not change.
I have also tried the field as a BLOB and then used .encode("hex") on the value I was inserting, again same behavior.
I have also tried HERP = cursor.var(cx_Oracle.CLOB) instead of cursor.setinputsizes(HERP = cx_Oracle.CLOB), same issues.
I've been going through the discussions on the cx-oracle-users list, but no luck yet.
It works if I use this line cursor.execute("INSERT INTO myTable2 (HERP) VALUES (:HERP)", HERP="".join(set(val3)).encode("hex")), so I don't think it's an issue with the data's content (This is with the BLOB).
How can I use cx_Oracle to insert a CLOB into an Oracle database?
There are a few possible solutions:
Upgrade cx_Oracle. I'm not sure which version you're using, but I'm using python 2.7.2 with cx_Oracle 5.1 and I don't get any errors when inserting 150,000 chars into a CLOB column without using setinputsizes at all.
Since setinputsizes applies to every subsequent cursor use, just change it between these different cursor.execute statements.
eg:
cursor = connection.cursor()
cursor.setinputsizes(FOO=None, BAR=None)
cursor.execute("INSERT INTO myTable (FOO, BAR) VALUES (:FOO, :BAR)",
FOO=val1, BAR=val2)
cursor.setinputsizes(HERP = cx_Oracle.CLOB)
cursor.execute("INSERT INTO myTable2 (HERP) VALUES (:HERP)", HERP=val3)