I'm currently running a script to insert values (a list of tuples) into a MySQL database, using the execute many function. When I use a small number of rows (`1000), the script runs fine.
When I use around 40,000 rows, I receive the following errors:
cursor.executemany( stmt, trans_frame)
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\IPython\core\interactiveshell.py", line 2538, in run_code
exec code_obj in self.user_global_ns, self.user_ns
File "<ipython-input-1-66b44e71cf5a>", line 1, in <module>
cursor.executemany( stmt, trans_frame)
File "C:\Python27\lib\site-packages\MySQLdb\cursors.py", line 253, in executemany
r = self._query('\n'.join([query[:p], ',\n'.join(q), query[e:]]))
File "C:\Python27\lib\site-packages\MySQLdb\cursors.py", line 346, in _query
rowcount = self._do_query(q)
File "C:\Python27\lib\site-packages\MySQLdb\cursors.py", line 310, in _do_query
db.query(q)
OperationalError: (2006, 'MySQL server has gone away')
Any suggestions?
sql ='SET GLOBAL max_allowed_packet=500*1024*1024'
cursor.execute(sql)
You could try setting the max_allowed_packet parameter just for one session:
sql ='SET SESSION max_allowed_packet=500M'
cursor.execute(sql)
sql = ...
args = ...
cursor.executemany(sql, args)
If this works, you could leave the code as it is, or change your my.cnf file (knowing that that solves the executemany problem).
Related
Here is the full error
Traceback (most recent call last):
File "Desktop/Work/RDS_LAMDA_Stuff/RDS_Node_Stuff/test.py", line 26, in <module>
handler()
File "Desktop/Work/RDS_LAMDA_Stuff/RDS_Node_Stuff/test.py", line 17, in handler
cursor.execute('SELECT * FROM Reads')
File "Desktop/Work/RDS_LAMDA_Stuff/RDS_Node_Stuff/pymysql/cursors.py", line 148, in execute
result = self._query(query)
File "Desktop/Work/RDS_LAMDA_Stuff/RDS_Node_Stuff/pymysql/cursors.py", line 310, in _query
conn.query(q)
File "Desktop/Work/RDS_LAMDA_Stuff/RDS_Node_Stuff/pymysql/connections.py", line 548, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "Desktop/Work/RDS_LAMDA_Stuff/RDS_Node_Stuff/pymysql/connections.py", line 775, in _read_query_result
result.read()
File "Desktop/Work/RDS_LAMDA_Stuff/RDS_Node_Stuff/pymysql/connections.py", line 1156, in read
first_packet = self.connection._read_packet()
File "Desktop/Work/RDS_LAMDA_Stuff/RDS_Node_Stuff/pymysql/connections.py", line 725, in _read_packet
packet.raise_for_error()
File "Desktop/Work/RDS_LAMDA_Stuff/RDS_Node_Stuff/pymysql/protocol.py", line 221, in raise_for_error
err.raise_mysql_exception(self._data)
File "Desktop/Work/RDS_LAMDA_Stuff/RDS_Node_Stuff/pymysql/err.py", line 143, in raise_mysql_exception
raise errorclass(errno, errval)
pymysql.err.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'Reads' at line 1")
For context here is my code
from multiprocessing import connection
import pymysql
from pymysql.cursors import Cursor
# RDS config
endpoint = '********'
username = '********'
password = '********'
database_name = '*********'
#connection config
connection = pymysql.connect(host=endpoint,user=username,passwd=password,db=database_name)
def handler():
cursor = connection.cursor()
cursor.execute('SELECT * FROM Reads')
rows = cursor.fetchall()
for row in rows:
print("{0} {1} {2} {3}".format(row[0], row[1], row[2], row[3]))
handler()
I have 2 other tables in my database which work fine when i query them however this table seems to be the only one causing an error.
Feel free to ask for more if this is too vague
It's worth checking manual in cases like this - and Reads is a reserved word therefore must be backticked - better still don't use reserved words or you will forever have to remember to backtick in code..
In my code, I am trying to insert data into the 'user_attempts' table of my DB which has two fields: attemptID (auto-incremented) and username. Therefore when I'm passing data in I only have to pass the username as the attemptID is generated by MySQL. This my code:
username='test1'
add_userattempt=mycursor.execute('INSERT INTO user_attempt (username) VALUES %(currentuser)s', {'currentuser' :username})
mydb.commit()
However it returns this error:
Traceback (most recent call last):
File "C:\Users\User\Desktop\project\AA game things\Iteration 1\review.py", line 266, in <module>
reviewPage(screen) # the home screen function is called which essentially starts the whole program.
File "C:\Users\User\Desktop\project\AA game things\Iteration 1\review.py", line 132, in reviewPage
add_userattempt=mycursor.execute('INSERT INTO user_attempt (username) VALUES %(currentuser)s', {'currentuser' :username})
File "C:\Users\User\AppData\Local\Programs\Python\Python37-32\lib\site-packages\mysql\connector\cursor.py", line 569, in execute
self._handle_result(self._connection.cmd_query(stmt))
File "C:\Users\User\AppData\Local\Programs\Python\Python37-32\lib\site-packages\mysql\connector\connection.py", line 590, in cmd_query
result = self._handle_result(self._send_cmd(ServerCmd.QUERY, query))
File "C:\Users\User\AppData\Local\Programs\Python\Python37-32\lib\site-packages\mysql\connector\connection.py", line 478, in _handle_result
raise errors.get_exception(packet)
mysql.connector.errors.ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ''test1'' at line 1
As shown by the error you get, you have a syntax error in your SQL code.
Instead of
mycursor.execute('INSERT INTO user_attempt (username) VALUES %(currentuser)s', {'currentuser' :username})
it should be
mycursor.execute('INSERT INTO user_attempt(username) VALUES(%s)', username)
This webpage here (https://www.mysqltutorial.org/python-mysql-insert/) explains the whole procedure of inserting data into MySQL tables using python pretty well.
Why do you declare your execute statement as a variable?
Hope this helps!
I am getting the following error message when I run this program on a mac. It works fine on the original Linux computer:
File "setupPpiDb.py", line 406, in modules
createDataBase(pdbsToAnalyzeWithChains)
File "setupPpiDb.py", line 202, in createDataBase
''' %(os.path.join(RESULTS_DIR, 'PerAtomDistance.csv')))
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line 250, in execute
self.errorhandler(self, exc, value)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/MySQLdb/connections.py", line 50, in defaulterrorhandler
raise errorvalue
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line 247, in execute
res = self._query(query)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line 412, in _query
rowcount = self._do_query(q)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line 375, in _do_query
db.query(q)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/MySQLdb/connections.py", line 276, in query
_mysql.connection.query(self, query)
_mysql_exceptions.OperationalError: (1148, 'The used command is not allowed with this MySQL version')
I am running Python3 on a mac and downloaded the latest community download version MySQL Database. The complete command is the following:
cursor.execute('''
load data local infile '%s' into table interfaceDist fields terminated by ',' optionally enclosed by '"' lines terminated by '\n' ignore 1 lines (PDB,Chains,Chain,ResId,Symbol,Atom,MinDist);
''' %(os.path.join(RESULTS_DIR, 'PerAtomDistance.csv')))
I have read that people have issues the local infile command. However, it has a problem with the second line of the command and I connect to the database in the following way with local_infile=1:
MySQLdb.connect(host='localhost', user=USER, passwd=PASSWD, db=DB_NAME, local_infile=1)
Any ideas on how to fix the bug?
I've caught a problem I can't explain using pygresql for python. This is a simple SELECT query and as far as I can tell the only query that has the problem in the whole script.
Exception in thread Thread-3:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/local/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "run_daily_script.py", line 122, in DelegateAPINodes
vm_api_group = Execute(db_conn, fetch_api_group, (cust_id, nodes["node_id"]))
File ".../PreparedQueries.py", line 14, in Execute
result = db.execute(command)
File "/usr/local/lib/python2.7/site-packages/pgdb.py", line 1030, in execute
return self.executemany(operation, [parameters])
File "/usr/local/lib/python2.7/site-packages/pgdb.py", line 1065, in executemany
raise _op_error("Internal error in '%s': %s" % (sql, err))
OperationalError: Internal error in 'SELECT * FROM vm_api_group WHERE cust_id=3 AND node_id=18': insufficient data in "T" message
I ran the query via the console and it works just fine. I monitored psql logs and there were no errors from the database. It seems to be an internal operation with pygresql.
What does "T" mean and how do I fix this?
I am trying to figure out why doesn't my test programme in python work.
I can access the database just fine from MySql Workbench, and I think I did everything right with the programming part, I also went to Administrative Tools and added my database to ODBC Database Sources, here is my test programme, if anyone can figure out what's wrong:
import pypyodbc
conn = pypyodbc.connect("DSN=database")
def func():
l = []
cur = conn.cursor()
try:
cur.execute("SELECT foo FROM table")
except pypyodbc.DatabaseError:
pass
conn.commit()
for i in cur:
l.append(i)
conn.close()
cur.close()
func()
The Error I'm getting is:
Traceback (most recent call last):
File "D:/path/test.py", line 21, in <module>
func()
File "D:/path/test.py", line 14, in func
for i in cur.fetchall():
File "D:/path/test.py", line 1819, in fetchall
row = self.fetchone()
File "D:/path/test.py", line 1893, in fetchone
check_success(self, ret)
File "D:/path/test.py", line 986, in check_success
ctrl_err(SQL_HANDLE_STMT, ODBC_obj.stmt_h, ret, ODBC_obj.ansi)
File "D:/path/test.py", line 956, in ctrl_err
raise DataError(state,err_text)
pypyodbc.DataError: ('22018', '[22018] [MySQL][ODBC 5.3(a) Driver][mysqld-5.5.5-10.0.17-MariaDB]')
Error 22018 is an "invalid character" error. One common cause is trying to use the ANSI ("(a)") version of MySQL Connector/ODBC to retrieve Unicode data. In that case the solution is to use the Unicode ("(w)") version of the driver instead.