We Keep Coding

I can't get Python to work with mySQL

Okay, so I'll preface with, 'I am new to Python'. I am using PyCharm to run this programme I am building to dump to a Database (locally hosted mySQL).
Unfortunately I am running into problems getting data into the database with Python, but I am able to insert data in SQL running through the phpMyAdmin web gui. I must be missing something obvious. I am using 'mysql.connector' addon for PyCharm. Here is the code relevant.
So, at the start I import the module? like so;
import mysql.connector
And this code is referenced inside a couple while loops;
cnx = mysql.connector.connect(host ='localhost', user = 'root', passwd= '', db='weather_test01')
c = cnx.cursor()
c.execute("INSERT INTO 'stations' ('dtime', 'tmp', 'apptmp', 'dewpoint', 'relhum', 'delta_t', 'wind_dir', 'wind_spd_kmh', 'wind_gust_kmh', 'wind_spd_kts', 'wind_gust_kts', 'press_qnh', 'press_msl', 'rainsince9am') VALUES (dtime1, tmp1, apptmp1, dewpoint1, relhum1, delta_t1, wind_dir1, wind_spd_kmh1, wind_gust_kmh1, wind_spd_kts1, wind_gust_kts1, press_qnh1, press_msl1, rainsince9am1)")
cnx.commit()
Please dont eat me alive either for the way I write, I am new :)
Other important factors;
I was able to get it to enter data a couple times (but it came through to SQL as a NULL value. My guess is I was passing the wrong format of number? or something to do with the db Collation?)
I have also tried this with a number of different addons for PyCharm (All of them seem to be implemented in similar ways).
I have tried it with and without quote marks on the variable names being inserted, I have tried so much, I am at my wits end.
As for the error codes, this is what it looks like currently;
C:\Users\logge\PycharmProjects\untitled3\venv\Scripts\python.exe C:/Users/logge/PycharmProjects/untitled3/main.py
Traceback (most recent call last):
File "C:/Users/logge/PycharmProjects/untitled3/main.py", line 111, in <module>
weather(1)
File "C:/Users/logge/PycharmProjects/untitled3/main.py", line 101, in weather
c.execute("INSERT INTO 'stations' ('dtime', 'tmp', 'apptmp', 'dewpoint', 'relhum', 'delta_t', 'wind_dir', 'wind_spd_kmh', 'wind_gust_kmh', 'wind_spd_kts', 'wind_gust_kts', 'press_qnh', 'press_msl', 'rainsince9am') VALUES (dtime1, tmp1, apptmp1, dewpoint1, relhum1, delta_t1, wind_dir1, wind_spd_kmh1, wind_gust_kmh1, wind_spd_kts1, wind_gust_kts1, press_qnh1, press_msl1, rainsince9am1)")
File "C:\Users\logge\PycharmProjects\untitled3\venv\lib\site-packages\mysql\connector\cursor.py", line 561, in execute
self._handle_result(self._connection.cmd_query(stmt))
File "C:\Users\logge\PycharmProjects\untitled3\venv\lib\site-packages\mysql\connector\connection.py", line 525, in cmd_query
result = self._handle_result(self._send_cmd(ServerCmd.QUERY, query))
File "C:\Users\logge\PycharmProjects\untitled3\venv\lib\site-packages\mysql\connector\connection.py", line 427, in _handle_result
raise errors.get_exception(packet)
mysql.connector.errors.ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ''stations' ('dtime', 'tmp', 'apptmp', 'dewpoint', 'relhum', 'delta_t', 'wind_dir' at line 1
Process finished with exit code 1
The 'line 101' refered to in the error codes is talking about the line starting with 'c.execute... INSERT' in the second section of code I posted.
Here is the code I have also tried, just to see if I could get any results (I still get the same errors though):
from __future__ import print_function
from datetime import date, datetime, timedelta
import mysql.connector
dtime1 = 1
tmp1 = 5
apptmp1 = 2
dewpoint1 = 64
relhum1 = 3
delta_t1 = 4
wind_dir1 = 5
wind_spd_kmh1 = 6
wind_gust_kmh1 = 7
wind_spd_kts1 = 8
wind_gust_kts1 = 9
press_qnh1 = 10
press_msl1 = 11
rainsince9am1 =12
cnx = mysql.connector.connect(host ='localhost', user = 'root', passwd= '', db='weather_test01')
cursor = cnx.cursor()
tomorrow = datetime.now().date() + timedelta(days=1)
add_employee = ("INSERT INTO stations "
"(dtime, tmp, apptmp, dewpoint, relhum, delta_t, wind_dir, wind_spd_kmh, wind_gust_kmh, wind_spd_kts, wind_gust_kts, press_qnh, press_msl, rainsince9am) "
"VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s,)")
data_employee = (dtime1, tmp1, apptmp1, dewpoint1, relhum1, delta_t1, wind_dir1, wind_spd_kmh1, wind_gust_kmh1, wind_spd_kts1, wind_gust_kts1, press_qnh1, press_msl1, rainsince9am1)
# Insert new employee
cursor.execute(add_employee, data_employee)
emp_no = cursor.lastrowid
# Make sure data is committed to the database
cnx.commit()
cursor.close()
cnx.close()
Any help or suggestions would be greatly appreciated. I am happy to post the entire code if required also. I just didn't want to clutter this whole thing up.

I found your error. You have referenced the input data to be strings (%s). But you are putting in integers (numbers).
If you want to put the numbers as strings, in the first chunk of code (yours), you need to put in str(variable) instead of just variable. In the second chunk of code (copied from MySQL), you can declare the variables as variable = "0000" with the quotes.
If you want to work with numbers in your code, change the table structure to be INTEGER or DOUBLE instead of VARCHAR or CHAR.
I would assume that the table is present and all fields are present as there is no error connecting to the database and finding the table.

python & MySQLdb TypeError('not all arguments converted during string formatting',) [duplicate]

Upon running this script:
#! /usr/bin/env python
import MySQLdb as mdb
import sys
class Test:
def check(self, search):
try:
con = mdb.connect('localhost', 'root', 'password', 'recordsdb');
cur = con.cursor()
cur.execute( "SELECT * FROM records WHERE email LIKE '%s'", search )
ver = cur.fetchone()
print "Output : %s " % ver
except mdb.Error, e:
print "Error %d: %s" % (e.args[0],e.args[1])
sys.exit(1)
finally:
if con:
con.close()
test = Test()
test.check("test")
I get an error of:
./lookup
Traceback (most recent call last):
File "./lookup", line 27, in <module>
test.check("test")
File "./lookup", line 11, in creep
cur.execute( "SELECT * FROM records WHERE email LIKE '%s'", search )
File "/usr/local/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 187, in execute
query = query % tuple([db.literal(item) for item in args])
TypeError: not all arguments converted during string formatting
I have zero idea why. I'm trying to do parameterized querys, but it's been nothing but a pain. I'm somewhat new to Python, so it's probably an obvious problem.

Instead of this:
cur.execute( "SELECT * FROM records WHERE email LIKE '%s'", search )
Try this:
cur.execute( "SELECT * FROM records WHERE email LIKE %s", [search] )
See the MySQLdb documentation. The reasoning is that execute's second parameter represents a list of the objects to be converted, because you could have an arbitrary number of objects in a parameterized query. In this case, you have only one, but it still needs to be an iterable (a tuple instead of a list would also be fine).

You can try this code:
cur.execute( "SELECT * FROM records WHERE email LIKE %s", (search,) )
You can see the documentation

'%' keyword is so dangerous because it major cause of 'SQL INJECTION ATTACK'.
So you just using this code.
cursor.execute("select * from table where example=%s", (example,))
or
t = (example,)
cursor.execute("select * from table where example=%s", t)
if you want to try insert into table, try this.
name = 'ksg'
age = 19
sex = 'male'
t = (name, age, sex)
cursor.execute("insert into table values(%s,%d,%s)", t)

cur.execute( "SELECT * FROM records WHERE email LIKE %s", (search,) )
I do not why, but this works for me . rather than use '%s'.

The accepted answer by #kevinsa5 is correct, but you might be thinking "I swear this code used to work and now it doesn't," and you would be right.
There was an API change in the MySQLdb library between 1.2.3 and 1.2.5. The 1.2.3 versions supported
cursor.execute("SELECT * FROM foo WHERE bar = %s", 'baz')
but the 1.2.5 versions require
cursor.execute("SELECT * FROM foo WHERE bar = %s", ['baz'])
as the other answers state. I can't find the change in the changelogs, and it's possible the earlier behavior was considered a bug.
The Ubuntu 14.04 repository has python-mysqldb 1.2.3, but Ubuntu 16.04 and later have python-mysqldb 1.3.7+.
If you're dealing with a legacy codebase that requires the old behavior but your platform is a newish Ubuntu, install MySQLdb from PyPI instead:
$ pip install MySQL-python==1.2.3

I don't understand the first two answers. I think they must be version-dependent. I cannot reproduce them on MySQLdb 1.2.3, which comes with Ubuntu 14.04LTS. Let's try them. First, we verify that MySQL doesn't accept double-apostrophes:
mysql> select * from methods limit 1;
+----------+--------------------+------------+
| MethodID | MethodDescription | MethodLink |
+----------+--------------------+------------+
| 32 | Autonomous Sensing | NULL |
+----------+--------------------+------------+
1 row in set (0.01 sec)
mysql> select * from methods where MethodID = ''32'';
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '9999'' ' at line 1
Nope. Let's try the example that Mandatory posted using the query constructor inside /usr/lib/python2.7/dist-packages/MySQLdb/cursors.py where I opened "con" as a connection to my database.
>>> search = "test"
>>> "SELECT * FROM records WHERE email LIKE '%s'" % con.literal(search)
"SELECT * FROM records WHERE email LIKE ''test''"
>>>
Nope, the double apostrophes cause it to fail. Let's try Mike Graham's first comment, where he suggests leaving off the apostrophes quoting the %s:
>>> "SELECT * FROM records WHERE email LIKE %s" % con.literal(search)
"SELECT * FROM records WHERE email LIKE 'test'"
>>>
Yep, that will work, but Mike's second comment and the documentation says that the argument to execute (processed by con.literal) must be a tuple (search,) or a list [search]. You can try them, but you'll find no difference from the output above.
The best answer is ksg97031's.

According PEP8，I prefer to execute SQL in this way:
cur = con.cursor()
# There is no need to add single-quota to the surrounding of `%s`,
# because the MySQLdb precompile the sql according to the scheme type
# of each argument in the arguments list.
sql = "SELECT * FROM records WHERE email LIKE %s;"
args = [search, ]
cur.execute(sql, args)
In this way, you will recognize that the second argument args of execute method must be a list of arguments.
May this helps you.

I encountered this error while executing
SELECT * FROM table;
I traced the error to cursor.py line 195.
if args is not None:
if isinstance(args, dict):
nargs = {}
for key, item in args.items():
if isinstance(key, unicode):
key = key.encode(db.encoding)
nargs[key] = db.literal(item)
args = nargs
else:
args = tuple(map(db.literal, args))
try:
query = query % args
except TypeError as m:
raise ProgrammingError(str(m))
Given that I am entering any extra parameters, I got rid of all of "if args ..." branch. Now it works.

How to store the output(even errors/warning) of sql queries in a file when ran in python script?

I'm running sql queries using python script on windows using Python IDE.
Following is my sample code to connext to DB and execute the script:
#!/usr/bin/python
import MySQLdb
import sys
import cProfile
import re
import timeit
# Open database connection
db = MySQLdb.connect("localhost","bala","bala","mydb" )
# prepare a cursor object using cursor() method
cursor = db.cursor()
import time
start_time = time.time()
// This is simple query, I was running different query,
sqlstr = "SELECT * from accounts"
cursor.execute(sqlstr)
db.commit()
print("--- %s micro seconds ---", time.time() - start_time)
print cursor.fetchall()
After running this script it gives output on console, However I want to store it in a file or string variable at least. It might results into errors or warnings, even that also can be stored in file/string.

#!/usr/bin/python
import MySQLdb
import sys
import cProfile
import re
import timeit
file = open("filename.txt","w")
# Open database connection
db = MySQLdb.connect("localhost","bala","bala","mydb" )
# prepare a cursor object using cursor() method
cursor = db.cursor()
import time
start_time = time.time()
// This is simple query, I was running different query,
sqlstr = "SELECT * from accounts"
try :
a = cursor.execute(sqlstr)
for i in a :
file.write(i)
db.commit()
except :
file.write(a)
print("--- %s micro seconds ---", time.time() - start_time)
print cursor.fetchall()
this may work !
I have opened a file where "filename" is name of your file and opened it in write mod and written output of your query line by line in the file having file name "filename". for error I have used try and except method.

Python MySQLdb TypeError: not all arguments converted during string formatting

Upon running this script:
#! /usr/bin/env python
import MySQLdb as mdb
import sys
class Test:
def check(self, search):
try:
con = mdb.connect('localhost', 'root', 'password', 'recordsdb');
cur = con.cursor()
cur.execute( "SELECT * FROM records WHERE email LIKE '%s'", search )
ver = cur.fetchone()
print "Output : %s " % ver
except mdb.Error, e:
print "Error %d: %s" % (e.args[0],e.args[1])
sys.exit(1)
finally:
if con:
con.close()
test = Test()
test.check("test")
I get an error of:
./lookup
Traceback (most recent call last):
File "./lookup", line 27, in <module>
test.check("test")
File "./lookup", line 11, in creep
cur.execute( "SELECT * FROM records WHERE email LIKE '%s'", search )
File "/usr/local/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 187, in execute
query = query % tuple([db.literal(item) for item in args])
TypeError: not all arguments converted during string formatting
I have zero idea why. I'm trying to do parameterized querys, but it's been nothing but a pain. I'm somewhat new to Python, so it's probably an obvious problem.

Instead of this:
cur.execute( "SELECT * FROM records WHERE email LIKE '%s'", search )
Try this:
cur.execute( "SELECT * FROM records WHERE email LIKE %s", [search] )
See the MySQLdb documentation. The reasoning is that execute's second parameter represents a list of the objects to be converted, because you could have an arbitrary number of objects in a parameterized query. In this case, you have only one, but it still needs to be an iterable (a tuple instead of a list would also be fine).

You can try this code:
cur.execute( "SELECT * FROM records WHERE email LIKE %s", (search,) )
You can see the documentation

'%' keyword is so dangerous because it major cause of 'SQL INJECTION ATTACK'.
So you just using this code.
cursor.execute("select * from table where example=%s", (example,))
or
t = (example,)
cursor.execute("select * from table where example=%s", t)
if you want to try insert into table, try this.
name = 'ksg'
age = 19
sex = 'male'
t = (name, age, sex)
cursor.execute("insert into table values(%s,%d,%s)", t)

cur.execute( "SELECT * FROM records WHERE email LIKE %s", (search,) )
I do not why, but this works for me . rather than use '%s'.

The accepted answer by #kevinsa5 is correct, but you might be thinking "I swear this code used to work and now it doesn't," and you would be right.
There was an API change in the MySQLdb library between 1.2.3 and 1.2.5. The 1.2.3 versions supported
cursor.execute("SELECT * FROM foo WHERE bar = %s", 'baz')
but the 1.2.5 versions require
cursor.execute("SELECT * FROM foo WHERE bar = %s", ['baz'])
as the other answers state. I can't find the change in the changelogs, and it's possible the earlier behavior was considered a bug.
The Ubuntu 14.04 repository has python-mysqldb 1.2.3, but Ubuntu 16.04 and later have python-mysqldb 1.3.7+.
If you're dealing with a legacy codebase that requires the old behavior but your platform is a newish Ubuntu, install MySQLdb from PyPI instead:
$ pip install MySQL-python==1.2.3

I don't understand the first two answers. I think they must be version-dependent. I cannot reproduce them on MySQLdb 1.2.3, which comes with Ubuntu 14.04LTS. Let's try them. First, we verify that MySQL doesn't accept double-apostrophes:
mysql> select * from methods limit 1;
+----------+--------------------+------------+
| MethodID | MethodDescription | MethodLink |
+----------+--------------------+------------+
| 32 | Autonomous Sensing | NULL |
+----------+--------------------+------------+
1 row in set (0.01 sec)
mysql> select * from methods where MethodID = ''32'';
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '9999'' ' at line 1
Nope. Let's try the example that Mandatory posted using the query constructor inside /usr/lib/python2.7/dist-packages/MySQLdb/cursors.py where I opened "con" as a connection to my database.
>>> search = "test"
>>> "SELECT * FROM records WHERE email LIKE '%s'" % con.literal(search)
"SELECT * FROM records WHERE email LIKE ''test''"
>>>
Nope, the double apostrophes cause it to fail. Let's try Mike Graham's first comment, where he suggests leaving off the apostrophes quoting the %s:
>>> "SELECT * FROM records WHERE email LIKE %s" % con.literal(search)
"SELECT * FROM records WHERE email LIKE 'test'"
>>>
Yep, that will work, but Mike's second comment and the documentation says that the argument to execute (processed by con.literal) must be a tuple (search,) or a list [search]. You can try them, but you'll find no difference from the output above.
The best answer is ksg97031's.

According PEP8，I prefer to execute SQL in this way:
cur = con.cursor()
# There is no need to add single-quota to the surrounding of `%s`,
# because the MySQLdb precompile the sql according to the scheme type
# of each argument in the arguments list.
sql = "SELECT * FROM records WHERE email LIKE %s;"
args = [search, ]
cur.execute(sql, args)
In this way, you will recognize that the second argument args of execute method must be a list of arguments.
May this helps you.

I encountered this error while executing
SELECT * FROM table;
I traced the error to cursor.py line 195.
if args is not None:
if isinstance(args, dict):
nargs = {}
for key, item in args.items():
if isinstance(key, unicode):
key = key.encode(db.encoding)
nargs[key] = db.literal(item)
args = nargs
else:
args = tuple(map(db.literal, args))
try:
query = query % args
except TypeError as m:
raise ProgrammingError(str(m))
Given that I am entering any extra parameters, I got rid of all of "if args ..." branch. Now it works.

execute *.sql file with python MySQLdb

How can execute sql script stored in *.sql file using MySQLdb python driver. I was trying
cursor.execute(file(PATH_TO_FILE).read())
but this doesn't work because cursor.execute can run only one sql command at once. My sql script contains several sql statements instead. Also I was trying
cursor.execute('source %s'%PATH_TO_FILE)
but also with no success.

From python, I start a mysql process to execute the file for me:
from subprocess import Popen, PIPE
process = Popen(['mysql', db, '-u', user, '-p', passwd],
stdout=PIPE, stdin=PIPE)
output = process.communicate('source ' + filename)[0]

I also needed to execute a SQL file, but the catch was that there wasn't one statement per line, so the accepted answer didn't work for me.
The SQL file I wanted to execute looked like this:
-- SQL script to bootstrap the DB:
--
CREATE USER 'x'#'%' IDENTIFIED BY 'x';
GRANT ALL PRIVILEGES ON mystore.* TO 'x'#'%';
GRANT ALL ON `%`.* TO 'x'#`%`;
FLUSH PRIVILEGES;
--
--
CREATE DATABASE oozie;
GRANT ALL PRIVILEGES ON oozie.* TO 'oozie'#'localhost' IDENTIFIED BY 'oozie';
GRANT ALL PRIVILEGES ON oozie.* TO 'oozie'#'%' IDENTIFIED BY 'oozie';
FLUSH PRIVILEGES;
--
USE oozie;
--
CREATE TABLE `BUNDLE_ACTIONS` (
`bundle_action_id` varchar(255) NOT NULL,
`bundle_id` varchar(255) DEFAULT NULL,
`coord_id` varchar(255) DEFAULT NULL,
`coord_name` varchar(255) DEFAULT NULL,
`critical` int(11) DEFAULT NULL,
`last_modified_time` datetime DEFAULT NULL,
`pending` int(11) DEFAULT NULL,
`status` varchar(255) DEFAULT NULL,
`bean_type` varchar(31) DEFAULT NULL,
PRIMARY KEY (`bundle_action_id`),
KEY `I_BNDLTNS_DTYPE` (`bean_type`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
--
--
Some statements in the above file lie on a single line and some statements also span multiple lines (like the CREATE TABLE at the end). There are also a few SQL inline comment lines that begin with "--".
As suggested by ThomasK, I had to write some simple rules to join lines into a statement. I ended up with a function to execute a sql file:
def exec_sql_file(cursor, sql_file):
print "\n[INFO] Executing SQL script file: '%s'" % (sql_file)
statement = ""
for line in open(sql_file):
if re.match(r'--', line): # ignore sql comment lines
continue
if not re.search(r';$', line): # keep appending lines that don't end in ';'
statement = statement + line
else: # when you get a line ending in ';' then exec statement and reset for next statement
statement = statement + line
#print "\n\n[DEBUG] Executing SQL statement:\n%s" % (statement)
try:
cursor.execute(statement)
except (OperationalError, ProgrammingError) as e:
print "\n[WARN] MySQLError during execute statement \n\tArgs: '%s'" % (str(e.args))
statement = ""
I'm sure there's scope for improvement, but for now it's working pretty well for me. Hope someone finds it useful.

This worked for me:
with open('schema.sql') as f:
cursor.execute(f.read().decode('utf-8'), multi=True)

for line in open(PATH_TO_FILE):
cursor.execute(line)
This assumes you have one SQL statement per line in your file. Otherwise you'll need to write some rules to join lines together.

Another solution that allows to leverage on the MySQL interpreter without any parsing is to use the os.system command to run a MySQL prompt command directly inside python:
from os import system
USERNAME = "root"
PASSWORD = "root"
DBNAME = "pablo"
HOST = "localhost"
PORT = 3306
FILE = "file.sql"
command = """mysql -u %s -p"%s" --host %s --port %s %s < %s""" %(USERNAME, PASSWORD, HOST, PORT, DBNAME, FILE)
system(command)
It avoids any parsing error when for example you would have a string variable with a smiley ;-) in it or if you check for the ; as the last character, if you have comments afterward like SELECT * FROM foo_table; # selecting data

Many of the answers here have serious flaws...
First don't try to parse an open ended sql script yourself! If you think that is easily done, you aren't aware of how robust and complicated sql can be. Serious sql scripts certainly involve statements and procedure definitions spanning multiple lines. It is also common to explicitly declare and change delimiters the in middle of your scripts. You can also nest source commands within each other. For so many reasons, you want to run the script through the MySQL client and allow it to handle the heavy lifting. Trying to reinvent that is fraught peril and a huge waste of time. Maybe if you are the only one writing these scripts, and you are not writing anything sophisticated you could get away with that, but why limit yourself to such a degree? What about machine generated scripts, or those written by other developers?
The answer from #jdferreira is on the right track, but also has problems and weaknesses. The most significant is that a security hole is being opened up by sending the connection parameters to the process in that manner.
Here's a solution / example for your copy & paste pleasure. My extended discussion follows:
First, create a separate config file to save your user name and password.
db-creds.cfg
[client]
user = XXXXXXX
password = YYYYYYY
Slap the right file system permissions on that, so the python process can read from it, but no one can view that who should not be able to.
Then, use this Python (in my example case the creds file is adjacent to the py script):
#!/usr/bin/python
import os
import sys
import MySQLdb
from subprocess import Popen, PIPE, STDOUT
__MYSQL_CLIENT_PATH = "mysql"
__THIS_DIR = os.path.dirname( os.path.realpath( sys.argv[0] ) )
__DB_CONFIG_PATH = os.path.join( __THIS_DIR, "db-creds.cfg" )
__DB_CONFIG_SECTION = "client"
__DB_CONN_HOST = "localhost"
__DB_CONN_PORT = 3306
# ----------------------------------------------------------------
class MySqlScriptError( Exception ):
def __init__( self, dbName, scriptPath, stdOut, stdErr ):
Exception.__init__( self )
self.dbName = dbName
self.scriptPath = scriptPath
self.priorOutput = stdOut
self.errorMsg = stdErr
errNumParts = stdErr.split("(")
try : self.errorNum = long( errNumParts[0].replace("ERROR","").strip() )
except: self.errorNum = None
try : self.sqlState = long( errNumParts[1].split(")")[0].strip() )
except: self.sqlState = None
def __str__( self ):
return ("--- MySqlScriptError ---\n" +
"Script: %s\n" % (self.scriptPath,) +
"Database: %s\n" % (self.dbName,) +
self.errorMsg )
def __repr__( self ): return self.__str__()
# ----------------------------------------------------------------
def databaseLoginParms() :
from ConfigParser import RawConfigParser
parser = RawConfigParser()
parser.read( __DB_CONFIG_PATH )
return ( parser.get( __DB_CONFIG_SECTION, "user" ).strip(),
parser.get( __DB_CONFIG_SECTION, "password" ).strip() )
def databaseConn( username, password, dbName ):
return MySQLdb.connect( host=__DB_CONN_HOST, port=__DB_CONN_PORT,
user=username, passwd=password, db=dbName )
def executeSqlScript( dbName, scriptPath, ignoreErrors=False ) :
scriptDirPath = os.path.dirname( os.path.realpath( scriptPath ) )
sourceCmd = "SOURCE %s" % (scriptPath,)
cmdList = [ __MYSQL_CLIENT_PATH,
"--defaults-extra-file=%s" % (__DB_CONFIG_PATH,) ,
"--database", dbName,
"--unbuffered" ]
if ignoreErrors :
cmdList.append( "--force" )
else:
cmdList.extend( ["--execute", sourceCmd ] )
process = Popen( cmdList
, cwd=scriptDirPath
, stdout=PIPE
, stderr=(STDOUT if ignoreErrors else PIPE)
, stdin=(PIPE if ignoreErrors else None) )
stdOut, stdErr = process.communicate( sourceCmd if ignoreErrors else None )
if stdErr is not None and len(stdErr) > 0 :
raise MySqlScriptError( dbName, scriptPath, stdOut, stdErr )
return stdOut
If you want to test it out, add this:
if __name__ == "__main__":
( username, password ) = databaseLoginParms()
dbName = "ExampleDatabase"
print "MySQLdb Test"
print
conn = databaseConn( username, password, dbName )
cursor = conn.cursor()
cursor.execute( "show tables" )
print cursor.fetchall()
cursor.close()
conn.close()
print
print "-----------------"
print "Execute Script with ignore errors"
print
scriptPath = "test.sql"
print executeSqlScript( dbName, scriptPath,
ignoreErrors=True )
print
print "-----------------"
print "Execute Script WITHOUT ignore errors"
print
try : print executeSqlScript( dbName, scriptPath )
except MySqlScriptError as e :
print "dbName: %s" % (e.dbName,)
print "scriptPath: %s" % (e.scriptPath,)
print "errorNum: %s" % (str(e.errorNum),)
print "sqlState: %s" % (str(e.sqlState),)
print "priorOutput:"
print e.priorOutput
print
print "errorMsg:"
print e.errorMsg
print
print e
print
And for good measure, here's an example sql script to feed into it:
test.sql
show tables;
blow up;
show tables;
So, now for some discussion.
First, I illustrate how to use MySQLdb along with this external script execution, while storing the creds in one shared file you can use for both.
By using --defaults-extra-file on the command line you can SECURELY pass your connection parameters in.
The combination of either --force with stdin streaming the source command OR --execute running the command on the outside let's you dictate how the script will run. That is by ignoring errors and continuing to run, or stopping as soon as an error occurs.
The order in which the results comeback will also be preserved via --unbuffered. Without that, your stdout and stderr streams will be jumbled and undefined in their order, making it very hard to figure out what worked and what did not when comparing that to the input sql.
Using the Popen cwd=scriptDirPath let's you nest source commands within one another using relative paths. If your scripts will all be in the same directory (or a known path relative to it), doing this let's you reference those relative to where the top level script resides.
Finally, I threw in an exception class which carries all the info you could possibly want about what happened. If you are not using the ignoreErrors option, one of these exceptions will be thrown in your python when something goes wrong and script has stopped running upon that error.

At least MySQLdb 1.2.3 seems to allow this out of the box, you just have to call cursor.nextset() to cycle through the returned result sets.
db = conn.cursor()
db.execute('SELECT 1; SELECT 2;')
more = True
while more:
print db.fetchall()
more = db.nextset()
If you want to be absolutely sure the support for this is enabled, and/or disable the support, you can use something like this:
MYSQL_OPTION_MULTI_STATEMENTS_ON = 0
MYSQL_OPTION_MULTI_STATEMENTS_OFF = 1
conn.set_server_option(MYSQL_OPTION_MULTI_STATEMENTS_ON)
# Multiple statement execution here...
conn.set_server_option(MYSQL_OPTION_MULTI_STATEMENTS_OFF)

The accepted answer will encounter problems when your sql script contains empty lines and your query sentence spans multiple lines. Instead, using the following approach will solve the problem:
f = open(filename, 'r')
query = " ".join(f.readlines())
c.execute(query)

As mentioned in one of the comments, if you are sure that every command ends with a semi-colon, you can do this:
import mysql.connector
connection = mysql.connector.connect(
host=host,
user=user,
password=password
)
cursor = connection.cursor()
with open(script, encoding="utf-8") as f:
commands = f.read().split(';')
for command in commands:
cursor.execute(command)
print(command)
connection.close()

Load mysqldump file:
for line in open(PATH_TO_FILE).read().split(';\n'):
cursor.execute(line)

Are you able to use a different database driver?
If yes: what you want is possible with the MySQL Connector/Python driver by MySQL.
Its cursor.execute method supports executing multiple SQL statements at once by passing Multi=True.
Splitting the SQL statements in the file by semicolon is not necessary.
Simple example (mainly copy & paste from the second link, I just added reading the SQL from the file):
import mysql.connector
file = open('test.sql')
sql = file.read()
cnx = mysql.connector.connect(user='uuu', password='ppp', host='hhh', database='ddd')
cursor = cnx.cursor()
for result in cursor.execute(sql, multi=True):
if result.with_rows:
print("Rows produced by statement '{}':".format(
result.statement))
print(result.fetchall())
else:
print("Number of rows affected by statement '{}': {}".format(
result.statement, result.rowcount))
cnx.close()
I'm using this to import MySQL dumps (created in phpMyAdmin by exporting the whole database to a SQL file) from the *.sql file back into a database.

Here's a code snippet that will import a typical .sql that comes from an export. (I used it with exports from Sequel Pro successfully.) Deals with multi-line queries and comments (#).
Note 1: I used the initial lines from Thomas K's response but added more.
Note 2: For newbies, replace the DB_HOST, DB_PASS etc with your database connection info.
import MySQLdb
from configdb import DB_HOST, DB_PASS, DB_USER, DB_DATABASE_NAME
db = MySQLdb.connect(host=DB_HOST, # your host, usually localhost
user=DB_USER, # your username
passwd=DB_PASS, # your password
db=DB_DATABASE_NAME) # name of the data base
cur = db.cursor()
PATH_TO_FILE = "db-testcases.sql"
fullLine = ''
for line in open(PATH_TO_FILE):
tempLine = line.strip()
# Skip empty lines.
# However, it seems "strip" doesn't remove every sort of whitespace.
# So, we also catch the "Query was empty" error below.
if len(tempLine) == 0:
continue
# Skip comments
if tempLine[0] == '#':
continue
fullLine += line
if not ';' in line:
continue
# You can remove this. It's for debugging purposes.
print "[line] ", fullLine, "[/line]"
try:
cur.execute(fullLine)
except MySQLdb.OperationalError as e:
if e[1] == 'Query was empty':
continue
raise e
fullLine = ''
db.close()

How about using the pexpect library? The idea is, that you can start a process pexpect.spawn(...), and wait until the output of that process contains a certain pattern process.expect(pattern).
I actually used this to connect to the mysql client and execute some sql scripts.
Connecting:
import pexpect
process = pexpect.spawn("mysql", ["-u", user, "-p"])
process.expect("Enter password")
process.sendline(password)
process.expect("mysql>")
This way the password is not hardcoded into the command line parameter (removes security risk).
Executing even several sql scripts:
error = False
for script in sql_scripts:
process.sendline("source {};".format(script))
index = process.expect(["mysql>", "ERROR"])
# Error occurred, interrupt
if index == 1:
error = True
break
if not error:
# commit changes of the scripts
process.sendline("COMMIT;")
process.expect("mysql>")
print "Everything fine"
else:
# don't commit + print error message
print "Your scripts have errors"
Beware that you always call expect(pattern), and that it matches, otherwise you will get a timeout error. I needed this bit of code to execute several sql scripts and only commit their changes if no error occurred, but it is easily adaptable for use cases with only one script.

You can use something like this-
def write_data(schema_name: str, table_name: str, column_names: str, data: list):
try:
data_list_template = ','.join(['%s'] * len(data))
insert_query = f"insert into {schema_name}.{table_name} ({column_names}) values {data_list_template}"
db.execute(insert_query, data)
conn_obj.commit()
except Exception as e:
db.execute("rollback")
raise e