I'm using a Python script that I have been using many times before to load CSV data into MySQL tables.
I modified the script for a very simple insert but it fails and I can't see why.
I've gone through the MySQL documentation of the Python connector, compared my syntax and I went through all the related articles on Stackoverflow but I can't find the reason. I've also checked the quotes I'm using as that is a common error.
Perhaps someone can help:
if row[0]:
s=row[0]
d=s[s.rfind('/')+1:len(s)-4]
cursor.execute("INSERT INTO `tab` (`did`) VALUES (%s)",(d))
I've checked print(d) and d is populated correctly.
The error I'm getting is
You have an error in your SQL syntax; check the manual that
corresponds to your MySQL server version for the right syntax to use
near '%s)' at line 1
If anyone can spot the (probably very silly) error, please help. Thanks.
The problem is that in
cursor.execute("INSERT INTO `tab` (`did`) VALUES (%s)",(d))
the (d) passed as params is a string with parentheses around it, not a tuple.
Here's how a mysql-connector cursor checks its params:
if params is not None:
if isinstance(params, dict):
for key, value in self._process_params_dict(params).items():
stmt = stmt.replace(key, value)
elif isinstance(params, (list, tuple)):
psub = _ParamSubstitutor(self._process_params(params))
stmt = RE_PY_PARAM.sub(psub, stmt)
if psub.remaining != 0:
raise errors.ProgrammingError(
"Not all parameters were used in the SQL statement")
So in your case though params is not None, it is not something accepted as params either and parameter substitution does not take place.
The fix then is simply to pass a tuple to cursor.execute() (a list works too):
cursor.execute("INSERT INTO `tab` (`did`) VALUES (%s)", (d,))
I think your string formating is wrong. It should probably be:
cursor.execute("INSERT INTO `tab` (`did`) VALUES (?)",d)
But you should check in the docs for your database library. I'm pretty sure the problem is with the placeholder in the query.
Related
I have been working with an SQL database through the mySQL library in Python. I have recently found that when I try searching for a string in my database, it is not correctly returning the output I expect. I think this may be due to my variable not being properly inserted into my SQL command.
code = data['code']
sql = "SELECT 200 FROM mytable WHERE mycolumn = '%s';"
query.execute(sql, teamCode)
print(str(query.fetchall()))
My problem is that printing query.fetchall() prints an empty list ([]) instead of the expected [[200,]] which means the program thinks the code value it is using does not exist in the SQL database, which it does.
The parameters in an execute call need to be a sequence, like a tuple. Try:
query.excute(sql, (teamCode,))
That turns it into a one-element tuple. BTW, did you really mean "code" there?
I'm fairly new to Python but have a project I am working on so please excuse any nativity on my part.
I am writing some SQL statements in Python 2.7 (Libraries not upgraded to 3 yet) but I am getting stuck on best practice procedure for them. We are using Sybase. Initially I was using
query = "UPDATE DB..TABLE SET version = '{}' WHERE name = '{}'".format(app.version, app.name)
cursor.execute(query)
But realised this after further reading that it is open to injection. So I then looked at doing the following:
query = "UPDATE DB..TABLE SET version = '%s' WHERE name = '%s'" % (app.version, app.name)
cursor.execute(query)
But got me to thinking is this not the same thing?
The parameters are also variables set by argparse, which means I have to use the '' around %s otherwise it throws up the invalid column name error. Which is frustrating for me as I also want to be able to pass NULL (None in Python) by default if any additional flags aren't set in other queries, otherwise it obviously inserts "NULL" as string.
For this particular example the 2 variables are set from a file being read by ConfigParser but I think it's still the same for argparse variables. e.g.
[SECTION]
application=name
version=1.0
I'm not quite sure how to best tackle this issue and yes yes I know "PYTHON 3 IS BETTER UPGRADE TO IT", as I said at the start, the libraries are in the process of being ported.
If you need any additional info then please advise and I will give you the best I can.
UPDATE***
Using the following Param style string I found in some sybase docs it can work but it does not pass None for NULL and throws up errors, starting to think this is a limitation of the sybase module in python.
cursor.execute("SELECT * FROM DB..table where app_name=#app_name", {"#app_name": app_name})
or
params = {"#appname": app.name. "#appver": app.version}
sql = "INSERT INTO DB..table (app_name, app_version) VALUES (#appname, #appversion)
cursor.execute(sql, params)
There is an issue though if you have a global list of params and feed that to a query that if any are None then it gives you a lot of errors again about being None, EVEN if those specific params aren't used in the query. Think I may be stuck doing IF statements for various options here for multiple inserts to bypass this None NULL issue.
Ok I have resolved this issue now.
I had to update the sybase module in order to get it to work with None > NULL.
As posted in the updated question. the below is how I was running the queries.
cursor.execute("SELECT * FROM DB..table where app_name=#app_name", {"#app_name": app_name})
or
params = {"#appname": app.name. "#appver": app.version}
sql = "INSERT INTO DB..table (app_name, app_version) VALUES (#appname, #appversion)
cursor.execute(sql, params)
But got me to thinking is this not the same thing?
Yes, those are effectively the same. They are both wide open to an injection attack.
Instead, do it this way:
query = "UPDATE DB..TABLE SET version = %s WHERE name = %s"
cursor.execute(query, [app.version, app.name])
I'm writing a program to extract a lot of data from another source and record it in a Postgres database. I need a function that takes in the destination table and a dictionary with variable fields to be added and then inserts it as appropriate. It seems like it should be simple enough, but I'm running into problems generating the insert query. The examples I've found online are either partial, outdated, or simply don't work when I modify them for my data.
Here's a simple version I've put together to work it out. I've tried a lot of variations of this, so it's probably not as clean as it should be at this point. It feels like there's something really simple that I'm just missing, but if so I'm just not seeing it.
def insert_record():
table = "test"
record = {"name": "Jack", "id": 1}
fields = record.keys()
values = ", ".join(str(n) for n in record.values())
query = sql.SQL("INSERT INTO {} ({}) VALUES ({});".format(
sql.Identifier(table),
sql.SQL(",").join(map(sql.Identifier, fields)),
sql.SQL(",").join(sql.Placeholder() * len(fields))
))
cursor = connection.cursor()
print(query.as_string(connection))
try:
cursor.execute(query, (values,))
connection.commit()
except (Exception, psycopg2.DatabaseError) as error:
print(error)
cursor.close()
This returns the error:
syntax error at or near "'test'"
LINE 1: INSERT INTO Identifier('test') (Composed([Identifier('name')...
It looks like it's not actually formatting the query for whatever reason, since the as_string function also returns the unformatted:
"INSERT INTO Identifier('test') (Composed([Identifier('name'), SQL(','), Identifier('id')])) VALUES (Composed([Placeholder(''), SQL(','), Placeholder('')]));"
Any suggestions on how to fix this, or better ways to handle dynamic queries in general?
edit: Here's my import statement
import psycopg2
from psycopg2 import extras, Error, sql
You are calling the function .format() from the object string, you must call the .format() function from sql.SQL() object class.
query = sql.SQL("INSERT INTO {} ({}) VALUES ({});").format(
sql.Identifier(table),
sql.SQL(",").join(map(sql.Identifier, fields)),
sql.SQL(",").join(sql.Placeholder() * len(fields))
)
Ref: https://www.psycopg.org/docs/sql.html?highlight=literal#module-usage
The error I get is :
sqlalchemy.exc.StatementError: (sqlalchemy.exc.InvalidRequestError) A value is required for bind parameter 'geboorteDatum' [SQL: 'INSERT into klanten (naam, geboorteDatum)VALUES(?, ?)'] [parameters: [{'naam': 'a'}]] (Background on this error at: http://sqlalche.me/e/cd3x)
It comes from the statement: connection.execute(text(sql), **kwargs), with sql being:
sql = ("INSERT into klanten "
"(naam, geboorteDatum)"
"VALUES(:naam, :geboorteDatum)")
The problem is that kwargs is missing a key-value pair (in this case for the key: 'geboorteDatum').
Is there a way to automatically use NULL in case of a missing key-value pair. It seems that this is possible when you use the ORM. But I don't use the ORM.
Attempts: I tried to convert the dict to a collection.defaultdict using tmp = (collections.defaultdict(lambda : 'NULL', klant_info)). This conversion works, however I get the same error. This is due the way that SQLAlchemy handles the dictionary. As can be seen in the method _execute_clauseelement found here, SQLAlchemy extracts all the keys, therefore using collection.defaultdict does not work.
Kind regards,
If modifying kwargs is possible, then you could run
kwargs.setdefault('geboorteDatum')
before executing the statement. It will be a no-operation, if the key is in kwargs, and add None under it if not.
UPDATE
You can use kwargs.get('geboorteDatum', None)
OR
You can check the key geboorteDatum of **kwargs befor the insert and assert it in a variable like:
geboorteDatum = if 'geboorteDatum' in kwargs else None
and then use your variable in the insert statement.
I tried to make a python command to update mysql on key duplicate
sql.run("INSERT INTO snapshots (id,username,data) VALUES ('%s','%s','%s') ON DUPLICATE KEY UPDATE data = VALUES(%s)" % (id,user.name,json.dumps(data),json.dumps(data)))
It works on data insert but on key duplicate, it throws this error
/usr/local/lib/python2.7/dist-packages/memsql/common/database.py at 166 > (1064, 'You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near \'{"achievements": {"150": 1448983913.491705, "200": 1448984742.809708, "204": 144\' at line 1')
I have tried KEY UPDATE data = '%s', KEY UPDATE data = VALUES(%s) and KEY UPDATE data = VALUES('%s') but none of them works.
What exactly did I do wrong here?
The root cause of the error is json-dumped string with not escaped " characters, caused by direct substituting the data into the query.
As #mgilson said, use cursor.execute() method and database driver will take care of escaping.