Okay, so I'll preface with, 'I am new to Python'. I am using PyCharm to run this programme I am building to dump to a Database (locally hosted mySQL).
Unfortunately I am running into problems getting data into the database with Python, but I am able to insert data in SQL running through the phpMyAdmin web gui. I must be missing something obvious. I am using 'mysql.connector' addon for PyCharm. Here is the code relevant.
So, at the start I import the module? like so;
import mysql.connector
And this code is referenced inside a couple while loops;
cnx = mysql.connector.connect(host ='localhost', user = 'root', passwd= '', db='weather_test01')
c = cnx.cursor()
c.execute("INSERT INTO 'stations' ('dtime', 'tmp', 'apptmp', 'dewpoint', 'relhum', 'delta_t', 'wind_dir', 'wind_spd_kmh', 'wind_gust_kmh', 'wind_spd_kts', 'wind_gust_kts', 'press_qnh', 'press_msl', 'rainsince9am') VALUES (dtime1, tmp1, apptmp1, dewpoint1, relhum1, delta_t1, wind_dir1, wind_spd_kmh1, wind_gust_kmh1, wind_spd_kts1, wind_gust_kts1, press_qnh1, press_msl1, rainsince9am1)")
cnx.commit()
Please dont eat me alive either for the way I write, I am new :)
Other important factors;
I was able to get it to enter data a couple times (but it came through to SQL as a NULL value. My guess is I was passing the wrong format of number? or something to do with the db Collation?)
I have also tried this with a number of different addons for PyCharm (All of them seem to be implemented in similar ways).
I have tried it with and without quote marks on the variable names being inserted, I have tried so much, I am at my wits end.
As for the error codes, this is what it looks like currently;
C:\Users\logge\PycharmProjects\untitled3\venv\Scripts\python.exe C:/Users/logge/PycharmProjects/untitled3/main.py
Traceback (most recent call last):
File "C:/Users/logge/PycharmProjects/untitled3/main.py", line 111, in <module>
weather(1)
File "C:/Users/logge/PycharmProjects/untitled3/main.py", line 101, in weather
c.execute("INSERT INTO 'stations' ('dtime', 'tmp', 'apptmp', 'dewpoint', 'relhum', 'delta_t', 'wind_dir', 'wind_spd_kmh', 'wind_gust_kmh', 'wind_spd_kts', 'wind_gust_kts', 'press_qnh', 'press_msl', 'rainsince9am') VALUES (dtime1, tmp1, apptmp1, dewpoint1, relhum1, delta_t1, wind_dir1, wind_spd_kmh1, wind_gust_kmh1, wind_spd_kts1, wind_gust_kts1, press_qnh1, press_msl1, rainsince9am1)")
File "C:\Users\logge\PycharmProjects\untitled3\venv\lib\site-packages\mysql\connector\cursor.py", line 561, in execute
self._handle_result(self._connection.cmd_query(stmt))
File "C:\Users\logge\PycharmProjects\untitled3\venv\lib\site-packages\mysql\connector\connection.py", line 525, in cmd_query
result = self._handle_result(self._send_cmd(ServerCmd.QUERY, query))
File "C:\Users\logge\PycharmProjects\untitled3\venv\lib\site-packages\mysql\connector\connection.py", line 427, in _handle_result
raise errors.get_exception(packet)
mysql.connector.errors.ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ''stations' ('dtime', 'tmp', 'apptmp', 'dewpoint', 'relhum', 'delta_t', 'wind_dir' at line 1
Process finished with exit code 1
The 'line 101' refered to in the error codes is talking about the line starting with 'c.execute... INSERT' in the second section of code I posted.
Here is the code I have also tried, just to see if I could get any results (I still get the same errors though):
from __future__ import print_function
from datetime import date, datetime, timedelta
import mysql.connector
dtime1 = 1
tmp1 = 5
apptmp1 = 2
dewpoint1 = 64
relhum1 = 3
delta_t1 = 4
wind_dir1 = 5
wind_spd_kmh1 = 6
wind_gust_kmh1 = 7
wind_spd_kts1 = 8
wind_gust_kts1 = 9
press_qnh1 = 10
press_msl1 = 11
rainsince9am1 =12
cnx = mysql.connector.connect(host ='localhost', user = 'root', passwd= '', db='weather_test01')
cursor = cnx.cursor()
tomorrow = datetime.now().date() + timedelta(days=1)
add_employee = ("INSERT INTO stations "
"(dtime, tmp, apptmp, dewpoint, relhum, delta_t, wind_dir, wind_spd_kmh, wind_gust_kmh, wind_spd_kts, wind_gust_kts, press_qnh, press_msl, rainsince9am) "
"VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s,)")
data_employee = (dtime1, tmp1, apptmp1, dewpoint1, relhum1, delta_t1, wind_dir1, wind_spd_kmh1, wind_gust_kmh1, wind_spd_kts1, wind_gust_kts1, press_qnh1, press_msl1, rainsince9am1)
# Insert new employee
cursor.execute(add_employee, data_employee)
emp_no = cursor.lastrowid
# Make sure data is committed to the database
cnx.commit()
cursor.close()
cnx.close()
Any help or suggestions would be greatly appreciated. I am happy to post the entire code if required also. I just didn't want to clutter this whole thing up.
I found your error. You have referenced the input data to be strings (%s). But you are putting in integers (numbers).
If you want to put the numbers as strings, in the first chunk of code (yours), you need to put in str(variable) instead of just variable. In the second chunk of code (copied from MySQL), you can declare the variables as variable = "0000" with the quotes.
If you want to work with numbers in your code, change the table structure to be INTEGER or DOUBLE instead of VARCHAR or CHAR.
I would assume that the table is present and all fields are present as there is no error connecting to the database and finding the table.
Related
From Python 3.9 i'm trying to do a MySql query like this
select * from brand WHERE text='L\'Orial'.
It works fine from phpMyAdmin but fails from python for all text including quote "'"
brand = "L'Orial"
where = f"text='{brand}'"
brand_pk_id = self.getPrimaryKeyIfExistInTable('brand', where)
def getPrimaryKeyIfExistInTable(self, table, where, key='id'):
try:
sql = f"SELECT {key} FROM {table} WHERE {where}"
self.cursor.execute(sql)
result = self.cursor.fetchone()
return result[key if self.bUseDictCursor else 0] if result else None
except pymysql.MySQLError as e:
logging.error(e)
return None
I can see that python escapes all quotes, which probably causes the problem, but can not figure out how to handle it properly !!
If I turn it around and use query LIKE with underscore( _ ) as wildcard:
brand = "L_Orial"
sql = f"SELECT {key} FROM {table} WHERE text LIKE '{brand}'"
It works fine, but this is not what I want !!
If I am understanding your question correctly, your problem is as follows:
Your query must exactly read:
SELECT * from brand WHERE text='L\'Orial'
But you are currently getting something like this, when you use python to execute the query:
SELECT * from brand WHERE text='L'Orial'
If this is indeed the issue, you should be able to resolve this by simply escaping the backslash that you need to have in the query. The complete python string for your query would be:
# Python String:
"SELECT * from brand WHERE text='L\\'Orial'"
# Resulting Query
SELECT * from brand WHERE text='L\'Orial'
If you wanted to automatically fix this issue for all brands that might include a ', you can simply replace the ' with \\' before making the query. Example:
brand = "L'Orial"
brand = brand.replace("'", "\\'")
# New Python string:
# "L\\'Orial"
# Output in SQL
# "L\'Orial"
Had to fire up my local instance just to make a point.
First, some prep work...
import pymysql
table = 'ps_carrier'
key = 'id_carrier'
mysql = {
"charset": "utf8",
"database": "mystore",
"host": "localhost",
"password": "secret",
"user": "justin"
}
As somebody suggested in the comments, the following
sql = "SELECT %s FROM %s WHERE %s"
where = "name='UPS'"
with pymysql.connect(**mysql) as conn:
with conn.cursor() as cur:
cur.execute(sql, (key, table, where))
Raises an error as expected since all the (string) params are quoted, even the table name!
Traceback (most recent call last):
File "<stdin>", line 3, in <module>
...
File "C:\Python38\site-packages\pymysql\err.py", line 143, in raise_mysql_exception
raise errorclass(errno, errval)
pymysql.err.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ''ps_carrier' WHERE 'name=\\'UPS\\''' at line 1")
If you can trust the inputs for the table name, the key, and the column name(s) then perhaps a simple query builder can help.
params = {'name': 'UPS'} # add more key--value pairs here
# use backticks in case we need to escape reserved words (OP uses MySQL)
where = " AND ".join(f"`{k}` = %s" for k in params.keys()) # .keys() just to be explicit
args = tuple([v for v in params.values()])
# backticks again
sql = f"SELECT `{key}` FROM `{table}` WHERE {where}"
print(sql)
print(args)
with pymysql.connect(**mysql) as conn:
with conn.cursor() as cur:
cur.execute(sql, args)
print(cur.fetchall())
If you need something more elaborate, there are a few modules such as Mysql Simple Query Builder and PyPika - Python Query Builder that you may want to look at (I've not used any of these.)
I am trying to write a script that reads a tab delimited text file and first creates a mysql table and then inserts the data into that table.
Problem: I'm stuck on writing the INSERT query because %s placeholder serves a new purpose with the mysql.connector API. Here is my code:
def insertmanyquery(tabletitle, headers, values):
'''connects to a mysql database and inserts a list of tab delimited rows into a table'''
cnxn = connect(all the connection parameters)
cursor = cnxn.cursor()
numofvalues = r"%s, " * len(headers.split(','))
numofvalues = numofvalues[:-2]
query = "INSERT INTO %s (%s) VALUES (%s)" % (tabletitle, headers, numofvalues)
cursor.executemany(query,values)
cnxn.commit()
cursor.close()
cnxn.close()
This would hopefully allow the insert query to adapt to however many columns are present in the table.
If I call the function as follows:
tabletitle = 'Bikes'
headers = 'BikeBrand, BikeName, Purpose, Price, YearPurchased'
values = ['Norco', 'Range', 'Enduro', 8,000.00, 2018]
insertmanyquery(tabletitle, headers, values)
I get the following error: mysql.connector.errors.ProgrammingError: Not all parameters were used in the SQL statement
If I just print the query instead of executing it, it looks fine:
INSERT INTO Bikes (BikeBrand, BikeName, Purpose, Price, YearPurchased) VALUES (%s, %s, %s, %s, %s)
I believe I am getting this error because the third %s in my INSERT query is being interpreted as a part of the connector INSERT syntax instead of first being interpreted in a pythonic manner and then being interpreted as connector syntax.
I am very new to coding so maybe I'm approaching this all wrong, regardless, I'd like to hear potential solutions to this problem or better ways to code this.
Thank you for your time
UPDATE:
I have tried making the query query = "INSERT INTO %s (%s)" % (tabletitle, headers) + " VALUES (" + numofvalues +");" and I still get the same error! so it doesn't have to do with using the placeholder.. >.<
import mysql.connector
conn = mysql.connector.connect(host="localhost",user="root",password="password", database="database_name")
cursor = conn.cursor()
a = "abcd"
cursor.execute("INSERT INTO table_jeux (lien_fiche) VALUES (?)", (a))
And this is the error I get when I execute the script :
ProgrammingError: 1064 (42000): Syntax error near to '?)' at line 1
Thanks for help, I really don't understand why.
What You probably want is to insert variable a into Your SQL code, but it doesn't work - and the parser gets a "?" where You want it to have "abcd"
You could try this:
cursor.execute("INSERT INTO table_jeux (lien_fiche) VALUES ({})".format(a))
or (in somewhat python2 manner):
cursor.execute("INSERT INTO table_jeux (lien_fiche) VALUES (%s)", (a,))
as described here:
https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursor-execute.html
Try using f strings,
It should look something like this:
cursor.execute(f'INSERT INTO student (student_name, student_email, courseid) VALUES ("{student_name}","{student_email}",{course_id})')
where student_name, student_email and course_id are all variables.
In your case:
cursor.execute(f'INSERT INTO table_jeux (lien_fiche) VALUES ("{a}")')
Upon running this script:
#! /usr/bin/env python
import MySQLdb as mdb
import sys
class Test:
def check(self, search):
try:
con = mdb.connect('localhost', 'root', 'password', 'recordsdb');
cur = con.cursor()
cur.execute( "SELECT * FROM records WHERE email LIKE '%s'", search )
ver = cur.fetchone()
print "Output : %s " % ver
except mdb.Error, e:
print "Error %d: %s" % (e.args[0],e.args[1])
sys.exit(1)
finally:
if con:
con.close()
test = Test()
test.check("test")
I get an error of:
./lookup
Traceback (most recent call last):
File "./lookup", line 27, in <module>
test.check("test")
File "./lookup", line 11, in creep
cur.execute( "SELECT * FROM records WHERE email LIKE '%s'", search )
File "/usr/local/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 187, in execute
query = query % tuple([db.literal(item) for item in args])
TypeError: not all arguments converted during string formatting
I have zero idea why. I'm trying to do parameterized querys, but it's been nothing but a pain. I'm somewhat new to Python, so it's probably an obvious problem.
Instead of this:
cur.execute( "SELECT * FROM records WHERE email LIKE '%s'", search )
Try this:
cur.execute( "SELECT * FROM records WHERE email LIKE %s", [search] )
See the MySQLdb documentation. The reasoning is that execute's second parameter represents a list of the objects to be converted, because you could have an arbitrary number of objects in a parameterized query. In this case, you have only one, but it still needs to be an iterable (a tuple instead of a list would also be fine).
You can try this code:
cur.execute( "SELECT * FROM records WHERE email LIKE %s", (search,) )
You can see the documentation
'%' keyword is so dangerous because it major cause of 'SQL INJECTION ATTACK'.
So you just using this code.
cursor.execute("select * from table where example=%s", (example,))
or
t = (example,)
cursor.execute("select * from table where example=%s", t)
if you want to try insert into table, try this.
name = 'ksg'
age = 19
sex = 'male'
t = (name, age, sex)
cursor.execute("insert into table values(%s,%d,%s)", t)
cur.execute( "SELECT * FROM records WHERE email LIKE %s", (search,) )
I do not why, but this works for me . rather than use '%s'.
The accepted answer by #kevinsa5 is correct, but you might be thinking "I swear this code used to work and now it doesn't," and you would be right.
There was an API change in the MySQLdb library between 1.2.3 and 1.2.5. The 1.2.3 versions supported
cursor.execute("SELECT * FROM foo WHERE bar = %s", 'baz')
but the 1.2.5 versions require
cursor.execute("SELECT * FROM foo WHERE bar = %s", ['baz'])
as the other answers state. I can't find the change in the changelogs, and it's possible the earlier behavior was considered a bug.
The Ubuntu 14.04 repository has python-mysqldb 1.2.3, but Ubuntu 16.04 and later have python-mysqldb 1.3.7+.
If you're dealing with a legacy codebase that requires the old behavior but your platform is a newish Ubuntu, install MySQLdb from PyPI instead:
$ pip install MySQL-python==1.2.3
I don't understand the first two answers. I think they must be version-dependent. I cannot reproduce them on MySQLdb 1.2.3, which comes with Ubuntu 14.04LTS. Let's try them. First, we verify that MySQL doesn't accept double-apostrophes:
mysql> select * from methods limit 1;
+----------+--------------------+------------+
| MethodID | MethodDescription | MethodLink |
+----------+--------------------+------------+
| 32 | Autonomous Sensing | NULL |
+----------+--------------------+------------+
1 row in set (0.01 sec)
mysql> select * from methods where MethodID = ''32'';
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '9999'' ' at line 1
Nope. Let's try the example that Mandatory posted using the query constructor inside /usr/lib/python2.7/dist-packages/MySQLdb/cursors.py where I opened "con" as a connection to my database.
>>> search = "test"
>>> "SELECT * FROM records WHERE email LIKE '%s'" % con.literal(search)
"SELECT * FROM records WHERE email LIKE ''test''"
>>>
Nope, the double apostrophes cause it to fail. Let's try Mike Graham's first comment, where he suggests leaving off the apostrophes quoting the %s:
>>> "SELECT * FROM records WHERE email LIKE %s" % con.literal(search)
"SELECT * FROM records WHERE email LIKE 'test'"
>>>
Yep, that will work, but Mike's second comment and the documentation says that the argument to execute (processed by con.literal) must be a tuple (search,) or a list [search]. You can try them, but you'll find no difference from the output above.
The best answer is ksg97031's.
According PEP8，I prefer to execute SQL in this way:
cur = con.cursor()
# There is no need to add single-quota to the surrounding of `%s`,
# because the MySQLdb precompile the sql according to the scheme type
# of each argument in the arguments list.
sql = "SELECT * FROM records WHERE email LIKE %s;"
args = [search, ]
cur.execute(sql, args)
In this way, you will recognize that the second argument args of execute method must be a list of arguments.
May this helps you.
I encountered this error while executing
SELECT * FROM table;
I traced the error to cursor.py line 195.
if args is not None:
if isinstance(args, dict):
nargs = {}
for key, item in args.items():
if isinstance(key, unicode):
key = key.encode(db.encoding)
nargs[key] = db.literal(item)
args = nargs
else:
args = tuple(map(db.literal, args))
try:
query = query % args
except TypeError as m:
raise ProgrammingError(str(m))
Given that I am entering any extra parameters, I got rid of all of "if args ..." branch. Now it works.
this is my first question.
I'm trying to execute a SQL query in django (south migration):
from django.db import connection
# ...
class Migration(SchemaMigration):
# ...
def transform_id_to_pk(self, table):
try:
db.delete_primary_key(table)
except:
pass
finally:
cursor = connection.cursor()
# This does not work
cursor.execute('SELECT MAX("id") FROM "%s"', [table])
# I don't know if this works.
try:
minvalue = cursor.fetchone()[0]
except:
minvalue = 1
seq_name = table + '_id_seq'
db.execute('CREATE SEQUENCE "%s" START WITH %s OWNED BY "%s"."id"', [seq_name, minvalue, table])
db.execute('ALTER TABLE "%s" ALTER COLUMN id SET DEFAULT nextval("%s")', [table, seq_name + '::regclass'])
db.create_primary_key(table, ['id'])
# ...
I use this function like this:
self.transform_id_to_pk('my_table_name')
So it should:
Find the biggest existent ID or 0 (it crashes)
Create a sequence name
Create the sequence
Update the ID field to use sequence
Update the ID as PK
But it crashes and the error says:
File "../apps/accounting/migrations/0003_setup_tables.py", line 45, in forwards
self.delegation_table_setup(orm)
File "../apps/accounting/migrations/0003_setup_tables.py", line 478, in delegation_table_setup
self.transform_id_to_pk('accounting_delegation')
File "../apps/accounting/migrations/0003_setup_tables.py", line 20, in transform_id_to_pk
cursor.execute(u'SELECT MAX("id") FROM "%s"', [table.encode('utf-8')])
File "/Library/Python/2.6/site-packages/django/db/backends/util.py", line 19, in execute
return self.cursor.execute(sql, params)
psycopg2.ProgrammingError: relation "E'accounting_delegation'" does not exist
LINE 1: SELECT MAX("id") FROM "E'accounting_delegation'"
^
I have shortened the file paths for convenience.
What does that "E'accounting_delegation'" mean? How could I get rid of it?
Thank you!
Carlos.
The problem is that you're using DB-API parameterization for things that are not SQL data. When you do something like:
cursor.execute('INSERT INTO table_foo VALUES (%s, %s)', (col1, col2))
the DB-API module (django's frontend for whatever database you are using, in this case) will know to escape the contents of 'col1' and 'col2' appropriately, and replace the %s's with them. Note that there are no quotes around the %s's. But that only works for SQL data, not for SQL metadata, such as table names and sequence names, because they need to be quoted differently (or not at all.) When you do
cursor.execute('INSERT INTO "%s" VALUES (%s, %s)', (tablename, col1, col2))
the tablename gets quoted as if you mean it to be string data to insert, and you end up with, for example, "'table_foo'". You need to separate your SQL metadata, which is part of the query, and your SQL data, which is not, like so:
sql = 'INSERT INTO TABLE "%s" VALUES (%%s, %%s)' % (tablename,)
cursor.execute(sql, (col1, col2))
Note that because the django DB-API frontend's paramstyle is 'pyformat' (it uses %s for placeholders) you need to escape those when you do the string formatting to create the SQL you want to execute. And note that this isn't secure against SQL injection attacks when you take the tablename from an insecure source and don't validate it.