I have a little problem with Connector/python executing insert query.
I have function to insert data into database. Argument data is list of tuples:
[(2652884, 'http://www.example.com/', '5.0.5.239', 1), ..... ]
def insert_url(self, data):
query = "INSERT INTO `sp_urls` (`parent_id`, `url`, `version`, `hits`) VALUES (%d, %s, %s, %d) ON DUPLICATE KEY UPDATE url=url"
try:
cursor = self.__cnx.cursor()
cursor.executemany(query, data)
except (mysql.connector.errors.IntegrityError) as err:
print("Query syntax error:", err, file=sys.stderr)
except (mysql.connector.errors.ProgrammingError) as err:
print("Programming err:{0}".format(err))
finally:
cursor.close()
The query itself works, in self.__cnx is initialized connection.
Here's the traceback:
File "sdi/database/DbValidator.py", line 91, in process_records
self.__driver.insert_url(urldata)
File "/home/david/workspace/stickydi/sdi/database/MySQLDriver.py", line 87, in insert_url
cursor.executemany(query, data)
File "/usr/lib/python3/dist-packages/mysql/connector/cursor.py", line 492, in executemany
return self._batch_insert(operation,seq_params)
File "/usr/lib/python3/dist-packages/mysql/connector/cursor.py", line 428, in _batch_insert
fmt = m.group(1).encode(self._connection.charset)
AttributeError: 'NoneType' object has no attribute 'group'
I have one very similar method, and it works OK, I just can't see, why does executemany( ) goes wrong.
Use only %s as the parameter marker in query. Do not use %d:
query = """
INSERT INTO `sp_urls` (`parent_id`, `url`, `version`, `hits`)
VALUES (%s, %s, %s, %s) ON DUPLICATE KEY UPDATE url=url"""
The %s is the format paramstyle defined in the DB-API. It does not have the same meaning as %s in string formatting.
The correct paramstyle to use depends on the database driver. MySQLdb uses %s. Other database drivers such as oursql and sqlite3 use ?.
You should not use %d for SQL parameters. Stick to %s and let the MySQL connector handle the types:
query = """\
INSERT INTO `sp_urls` (`parent_id`, `url`, `version`, `hits`)
VALUES (%s, %s, %s, %s)
ON DUPLICATE KEY UPDATE url=url
"""
Quoting from the Python-MySQL documentation:
paramstyle
String constant stating the type of parameter marker formatting expected by the interface. Set to 'format' = ANSI C printf format codes, e.g. '...WHERE name=%s'. If a mapping object is used for conn.execute(), then the interface actually uses 'pyformat' = Python extended format codes, e.g. '...WHERE name=%(name)s'. However, the API does not presently allow the specification of more than one style in paramstyle.
Granted, using %s for SQL parameters is confusingly similar to Python string formatting, but it is not the same.
Related
When I run the below code with psycopg2:
cur.execute(
"""INSERT INTO logmsg (msg_type, file, msg) VALUES %s;""",
["Error", str(file), str(sys.exc_info()[0])])
I get the following error:
TypeError: not all arguments converted during string formatting
Can someone help me with this?
VALUES needs a list of values enclosed in brackets:
cur.execute(
"""INSERT INTO logmsg (msg_type, file, msg) VALUES (%s, %s, %s);""",
["Error", str(file), str(sys.exc_info()[0])])
Do not forget to commit the transaction.
With reference to Import MySQL dump to PostgreSQL database.
An unknown developer has offered there to use the following script to import MySQL database to PostgreSQL
import MySQLdb
#from magic import Connect #Private mysql connect information - I COMMENTED THIS LINE to use direct connection
db = MySQLdb.connect(host="localhost", # your host, usually localhost
user="USER", # your username
passwd="PASS", # your password
db="w3i") # name of the data base
import psycopg2
dbx=Connect()
DB=psycopg2.connect("dbname='w3i'")
DC=DB.cursor()
mysql='''show tables from w3i'''
dbx.execute(mysql); ts=dbx.fetchall(); tables=[]
for table in ts: tables.append(table[0])
for table in tables:
mysql='''describe w3i.%s'''%(table)
dbx.execute(mysql); rows=dbx.fetchall()
psql='drop table %s'%(table)
DC.execute(psql); DB.commit()
psql='create table %s ('%(table)
for row in rows:
name=row[0]; type=row[1]
if 'int' in type: type='int8'
if 'blob' in type: type='bytea'
if 'datetime' in type: type='timestamptz'
psql+='%s %s,'%(name,type)
psql=psql.strip(',')+')'
print psql
try: DC.execute(psql); DB.commit()
except: pass
msql='''select * from w3i.%s'''%(table)
dbx.execute(msql); rows=dbx.fetchall()
n=len(rows); print n; t=n
if n==0: continue #skip if no data
cols=len(rows[0])
for row in rows:
ps=', '.join(['%s']*cols)
psql='''insert into %s values(%s)'''%(table, ps)
DC.execute(psql,(row))
n=n-1
if n%1000==1: DB.commit(); print n,t,t-n
DB.commit()
As you can see - I changed line 2 to direct connection with MySQL
But now I have the following error
python postgres.py
Traceback (most recent call last):
File "postgres.py", line 9, in <module>
dbx=Connect()
NameError: name 'Connect' is not defined
Thanks in advance for a hint how to solve it !
EDIT : I forgot the cursor ...
EDIT2 : original script did not correctly process fields of TINYTEXT, MEDIUMTEXT or LONGTEXT type => added a conversion to PostgreSQL TEXT type
EDIT3 : the original script did not process ENUM fields, choked on non 7 bits characters, and had a wrong error management
You commented out line 2 where Connect was defined, but you left line 9 where Connect() is used untouched, so the error.
As you now explicitely connect to MySQL, you should replace dbx = Connect() with :
dbx = db.cursor()
It should now give (with the conversion of TEXT types line 28):
import MySQLdb
#from magic import Connect #Private mysql connect information - I COMMENTED THIS LINE to use direct connection
db = MySQLdb.connect(host="localhost", # your host, usually localhost
user="USER", # your username
passwd="PASS", # your password
db="w3i") # name of the data base
import psycopg2
# set client_encoding if different that PostgreSQL database default
encoding = 'Latin1'
dbx=db.cursor()
DB=psycopg2.connect("dbname='w3i'")
DC=DB.cursor()
DC.execute("set client_encoding = " + encoding)
mysql='''show tables from w3i'''
dbx.execute(mysql); ts=dbx.fetchall(); tables=[]
for table in ts: tables.append(table[0])
for table in tables:
mysql='''describe w3i.%s'''%(table)
dbx.execute(mysql); rows=dbx.fetchall()
psql='drop table %s'%(table)
DC.execute(psql); DB.commit()
psql='create table %s ('%(table)
for row in rows:
name=row[0]; type=row[1]
if 'int' in type: type='int8'
if 'blob' in type: type='bytea'
if 'datetime' in type: type='timestamptz'
if 'text' in type: type='text'
if 'enum' in type:
type = 'varchar'
print ("warning : conversion of enum to varchar %s(%s)" % (table, name))
psql+='%s %s,'%(name,type)
psql=psql.strip(',')+')'
print psql
try: DC.execute(psql); DB.commit()
except Exception as e:
print e
DB.rollback()
Above script convert enum to VARCHAR. If you have only one enum type you can try to create it PostgreSQL side :
DC.execute("DROP TYPE IF EXISTS enumtyp CASCADE")
DC.execute("CREATE TYPE enumtyp AS ENUM( ... )"
where enumtyp is the name of the type and ... is the list of (textual) values (don't forget to add an empty value if the field can be empty in MySQL)
Then you replace enum with enumtyp by replacing line type = 'varchar' with :
if 'enum' in type:
type = 'enumtyp'
as reported in the answer you cite:
from magic import Connect #Private mysql connect information
Connect() is (I assume) a method feeding parameters to connect to a specific db.
You thus have either to implement on your own this magic module, with references to your specific parameters, or to specify which connection you want to setup, namely MySQLdb.connect(...) or psycopg2.connect(...)
Im trying to run a python script on my raspberry pi, in order to store the data that I receive from the gps receiver into a sql table. While Im executing this script I'm getting an error on this part of the code:
sql = "INSERT INTO gps (n_lat, w_long, date_time) VALUES (%s, %s, %s)" % (north, west, t,)
print sql
cur.execute(sql)
print "Rows inserted: %s" % cur.rowcount
con.commit()
time.sleep(0.5)
Error:
Traceback (most recent call last):
File "gps.py", line 48, in <module>
cur.execute(sql)
File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
self.errorhandler(self, exc, value)
File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler raise errorclass, errorvalue
_mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ':01:16)' at line 1")
I don't really understand where is the problem, Have you got any ideas why whiz error occurs ?
You did not escape your input values. For numbers this is optional, but datetime might not be a number.
However, you should always escape input values for your database. The keyword here is prepared statements. Instead of parsing your input arguments into the string with pythons % operater, you should use the argument list of cursor.execute.
sql = "INSERT INTO gps (n_lat, w_long, date_time) VALUES (%s, %s, %s)"
print sql
cur.execute(sql, (north, west, t,))
print "Rows inserted: %s" % cur.rowcount
con.commit()
time.sleep(0.5)
Now the function execute will make sure, that all special characters are escaped. E.g. one of your input values might contain a single quote or similar. Using python's string parsing, this would result in something like:
"INSERT INTO gps (n_lat, w_long, date_time) VALUES ('123', '123', '1234'321')"
In best case this would result in a database error, in worst case somebody could manipulate your database with his own SQL statements (so called SQL injection).
The error in the SQL statement you have set values inside a quote:
VALUES ('%s', '%s', '%s')
I want to execute postgres query in python.The table name has to be passed as a parameter.Since the table will be created at run time. I have used dict query param style.But i am getting an error.
import psycopg2
CONNECTION_STRING = "dbname='autogist' user='postgres' password=''"
query = "INSERT INTO %(table)s " +\
"(vin_id, vin_details_id, price, mileage, dealer_id, created_on, modified_on) " +\
"VALUES (%(vin_id)s, %(vlookup_id)s, %(price)s, %(mileage)s, %(dealer_id)s,now(),now()) " +\
"RETURNING id"
params = {"table" : "dealer_vehicle_details_2010_01_02",\
"vin_id":"3",\
"vlookup_id":"403",\
"price":"403",\
"mileage":"403",\
"dealer_id":"276092"
}
conn=psycopg2.connect(CONNECTION_STRING)
cursor = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
cursor.execute(query,params)
TRACEBACK:
ERROR: An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line statement', (262, 0))
---------------------------------------------------------------------------
ProgrammingError Traceback (most recent call last)
/home/gridlex/workspace/<ipython console> in <module>()
/usr/local/lib/python2.6/dist-packages/psycopg2/extras.pyc in execute(self, query, vars)
121 self.index = {}
122 self._query_executed = 1
--> 123 return _cursor.execute(self, query, vars)
124
125 def callproc(self, procname, vars=None):
ProgrammingError: syntax error at or near "E'dealer_vehicle_details_2010_01_02'"
LINE 1: INSERT INTO E'dealer_vehicle_details_2010_01_02' (vin_id, vi...
The statement you send must be syntactically valid when PREPAREd, which a statement with placeholders for table names is not. You can't use placeholders for table names in prepared statements.
Your options are:
Substitute the table name in with regular string substitution, "double quoted". Be very careful with your quoting routine; make sure it doubles any quotes within the table name its self, so the table name double"quote becomes "double""quote". Eg. 'SELECT * FROM "%s"' % quote_ident(tablename). You'd have to roll your own quote_ident as AFAIK psycopg2 doesn't expose a function like that.
Send the table name as a query parameter to a PL/PgSQL function that uses EXECUTE ... USING to create a dynamic SQL statement using the table name. PL/PgSQL can use the quote_ident function to provide safer quoting than a home-rolled implementation.
I'm experiencing a problem inserting values into a SQLite database. The data I download from the Norwegian Parliament site data.stortinget.no. The error I get is: sqlite3.OperationalError: unrecognized token: "01T00"
Here is the method in which the error occur: (I know about the indentation error in this excerpt)
def get_perioder(cur):
DOK = "stortingsperioder"
try:
page = urllib2.urlopen(SITE+DOK)
except:
print "Failed to fetch item "+DOK
if page:
tree = ElementTree.parse(page)
root = tree.getroot()
top = list(root)[2]
elements = list(top)
for el in elements:
fra = el.find('{http://data.stortinget.no}fra').text
per_id = el.find('{http://data.stortinget.no}id').text
til = el.find('{http://data.stortinget.no}til').text
print "id: %s fra: %s til: %s" % (per_id, fra, til)
cur.execute("INSERT INTO perioder(fra, id, til) VALUES(%s,%s,%s)" % (fra, per_id, til))
else:
print "Could not load page: "+DOK
The message printed by the print just above cur.execute is:
id: 2009-2013 fra: 2009-10-01T00:00:00 til: 2013-09-30T23:59:59
The whole error trace is:
BigMac:Stortingsdata ola$ python getBasicData.py
id: 2009-2013 fra: 2009-10-01T00:00:00 til: 2013-09-30T23:59:59
Traceback (most recent call last):
File "getBasicData.py", line 169, in <module>
get_perioder(cur)
File "getBasicData.py", line 26, in get_perioder
cur.execute("INSERT INTO perioder(fra, id, til) VALUES(%s,%s,%s)" % (fra, per_id, til))
sqlite3.OperationalError: unrecognized token: "01T00"
I referred with the SQLite manual and it seems that the format is supported, so I'm wondering where the problem come from.
The proper way is to use a parametrized query.
Example:
cur.execute("""INSERT INTO perioder(fra, id, til)
VALUES (?,?,?);""", (fra, per_id, til))
There is a specific parameter "style" for each database driver.
In the case of SQLite that parameter style is ?.
Also note that the parameter values are passed as a second argument to execute().
Using string-interpolation leaves you vulnerable to all kinds of quoting issues (like the one that brought you here) and the possibility of SQL-injection attack.
For more information please read the DB-API and the database programming wiki.
If you want to store the date stamps as strings (TEXT) in SQLite, I recommend you format the text you would like to execute as follows:
cur.execute("""INSERT INTO perioder(fra, id, til)
VALUES (\"%s\",\"%s\",\"%s\")""" % (fra, per_id, til))
SQLite returns errors if the values you insert does not have inverted commas. Formatting your text with \"%s\" instead of %s will insert the string value with inverted commas in your formatted string:
"INSERT INTO perioder(fra, id, til)
VALUES ("2009-2013", "2009-10-01T00:00:00","2013-09-30T23:59:59")"