I have a dictionary that I want to add all the values to an sqlite database. All the keys in the dictionary exist in the database, and all the keys are of type string. Yet, I am having trouble with getting the values into the database. The following code is ugly, insecure, and errors whenever it comes across a string with a " in it, but it sort of works.
Query="INSERT INTO packages VALUES("
for tag in Tags:
Query=Query + '"' + Package[tag] + '", '
Query=Query[:-2]+")"
cursor.execute(Query)
How can I elegantly fix this so that it is secure and accepts inputs with " in the string? I've come across a handful of other methods. For example:
fields = Package.keys()
values = Package.values()
query = "INSERT INTO packages (%s) VALUES (%%s);" % (",".join(fields))
cursor.execute(query, values)
but it throws a type error.
TypeError: function takes at most 2 arguments (38 given)
The most elegant solution I have come across so far appears to be
sql_insert = ('INSERT INTO packages (%s) VALUES (%s)' %
(','.join('%s' % name for name in Package),
','.join('%%(%s)s' % name for name in Package)))
cursor.execute(sql_insert, Package)
but it throws an operational error, saying
sqlite3.OperationalError: near "%": syntax error
Once again, my question is how can I elegantly safely add the values from a dictionary to a database?
P.S. It may also be worthy to note that I am using Python 2.5.1.
Afaik, when query has a "?" placeholder execute() method does right escaping automatically basing on argument types. So, the following should work:
query = 'INSERT INTO packages VALUES(%s)' % ','.join(['?'] * len(Tags))
cursor.execute(query, Tags)
I have come across the same problem however i had the problem that not all dictionary entries contained all columns of the table therefore i created the following solution
keys, values = zip(*Package.items())
insert_str = "INSERT INTO packages (%s) values (%s)" % (",".join(keys),",".join(['?']*len(keys)))
cursor.execute(insert_str,values)
Related
How do I fix this error on passing a string parameter into pd.read_sql_query? Using Python 2.7.
table = 'mytable1'
query = (
"SELECT * "
"FROM ? "
)
df = pd.read_sql_query(sql=query, con=conn, params=(table))
pandas.io.sql.DatabaseError: Execution failed on sql 'SELECT * FROM ? ': near "?": syntax error
I have tried replacing ? with % and %s but it returns the same error.
The following equality example works as expected:
query = (
"SELECT * "
"FROM mytable1 "
"WHERE name = ? "
)
df = pd.read_sql_query(sql=query, con=conn, params=('cat',))
Note that the comma in params appears to be required, otherwise this error is returned: Incorrect number of bindings supplied. The current statement uses 1, and there are 7 supplied.
I have also tried params=(table,) in my problem but no luck. I know that alternatively I can do this with "FROM '{t}' ").format(t=table), but I would like to understand how to use Pandas' built-in parameters option.
The problem is that params is intended to replace values, not SQL keywords (FROM, SELECT, etc...) or tables or columns.
You can't specify table that way, you have to use a string substitution.
query = (
f"SELECT * "
"FROM {table} "
)
However, be very, very, very careful. Doing this, rather than using params opens you up to a very big family of vulnerabilities, SQL Injections.
Don't do it if you get the table names from external sources.
(Oh, and the likely reason for this limitation is that, besides their absolute necessity for security reasons, parametrized queries allow the database engine to examine the query, look at indices, statistics and all that and draw up an "execution plan". Successive calls then can reuse that execution plan and just substitute in the new variables. That could not be done if the query was changing in terms of what tables and columns were being accessed.)
I have a JSONB object that I want to insert into my database that also contains single quote characters. The example below is just trimmed to highlight my issue:
[{"name": "Moody's Analytics BBB Corp Bd ETF"}]
I have tried just dumping that JSON object to a string and inserting it, but the single ' wont allow that to work and when I try to execute (where detail is the JSON object):
cur.execute("INSERT INTO schmea.table (id, detail) VALUES (?,?)", (sec_id, detail)
I get the following syntax error:
psycopg2.errors.SyntaxError: syntax error at or near ","
LINE 1: INSERT INTO schema.table (id, detail) VALUES (?,?)
^
All of the solutions I've found so far suggest turning the JSON object into a string using json.dumps() but that wont work in my situation because of the single ' character in the name.
Any help is appreciated here.
Edit: Was told that the proper bind is %s not ? but that changes the error to:
psycopg2.ProgrammingError: can't adapt type 'dict'
edit 2:
SOLUTION:
Using the correct %s binds and using json.dumps(detail) fixed the issue.
cur.execute("INSERT INTO schmea.table (id, detail) VALUES (%s,%s)", (sec_id, json.dumps(detail))
Thanks for your help!
The way to pass bind variables to psycopg2 is with %s, not ?.
I'm writing a program to extract a lot of data from another source and record it in a Postgres database. I need a function that takes in the destination table and a dictionary with variable fields to be added and then inserts it as appropriate. It seems like it should be simple enough, but I'm running into problems generating the insert query. The examples I've found online are either partial, outdated, or simply don't work when I modify them for my data.
Here's a simple version I've put together to work it out. I've tried a lot of variations of this, so it's probably not as clean as it should be at this point. It feels like there's something really simple that I'm just missing, but if so I'm just not seeing it.
def insert_record():
table = "test"
record = {"name": "Jack", "id": 1}
fields = record.keys()
values = ", ".join(str(n) for n in record.values())
query = sql.SQL("INSERT INTO {} ({}) VALUES ({});".format(
sql.Identifier(table),
sql.SQL(",").join(map(sql.Identifier, fields)),
sql.SQL(",").join(sql.Placeholder() * len(fields))
))
cursor = connection.cursor()
print(query.as_string(connection))
try:
cursor.execute(query, (values,))
connection.commit()
except (Exception, psycopg2.DatabaseError) as error:
print(error)
cursor.close()
This returns the error:
syntax error at or near "'test'"
LINE 1: INSERT INTO Identifier('test') (Composed([Identifier('name')...
It looks like it's not actually formatting the query for whatever reason, since the as_string function also returns the unformatted:
"INSERT INTO Identifier('test') (Composed([Identifier('name'), SQL(','), Identifier('id')])) VALUES (Composed([Placeholder(''), SQL(','), Placeholder('')]));"
Any suggestions on how to fix this, or better ways to handle dynamic queries in general?
edit: Here's my import statement
import psycopg2
from psycopg2 import extras, Error, sql
You are calling the function .format() from the object string, you must call the .format() function from sql.SQL() object class.
query = sql.SQL("INSERT INTO {} ({}) VALUES ({});").format(
sql.Identifier(table),
sql.SQL(",").join(map(sql.Identifier, fields)),
sql.SQL(",").join(sql.Placeholder() * len(fields))
)
Ref: https://www.psycopg.org/docs/sql.html?highlight=literal#module-usage
I'm trying to insert a list into separate columns of a database
print inserter
params = ['%s' for item in inserter]
sql_query = 'INSERT INTO tablename (coloumn1, coloumn2, coloumn3, coloumn4, coloumn5, coloumn6, coloumn7) VALUES (%s,%s,%s,%s,%s,%s,%s);' % ','.join(params)
cursor.execute(sql_query)
db.commit
But keep getting the error
not enough arguments for format string
Anyone know what I am doing wrong?
Anyone know what I am doing wrong?
You are using string interpolation in a query.
This is bad, mainly for 2 reasons:
It is erroneous as you see. The python interpreter is confused between the %s for the interpolation and the %s for the sql parameters.
It makes your code vulnerable for sql injection.
You should use a parametrized query:
sql_query = '''INSERT INTO tablename (coloumn1, coloumn2, coloumn3,
coloumn4, coloumn5, coloumn6, coloumn7)
VALUES (%s,%s,%s,%s,%s,%s,%s);'''
cursor.execute(sql_query, inserter) # assuming inserter is a tuple/list of values
I am receiving an error when trying to write data to a database table when using a variable for the table name that I do not get when using a static name. For some reason on the line where I insert, if I insert an integer as the column values the code runs and the table is filled, however, if I try to use a string I get a SQL syntax error
cursor = db.cursor()
cursor.execute('DROP TABLE IF EXISTS %s' %data[1])
sql ="""CREATE TABLE %s (IP TEXT, AVAILIBILITY INT)""" %data[1]
cursor.execute(sql)
for key in data[0]:
cur_ip = key.split(".")[3]
cursor.execute("""INSERT INTO %s VALUES (%s,%s)""" %(data[1],key,data[0][key]))
db.commit()
the problem is where I have %(data[1], key, data[0][key]) any ideas?
It's a little hard to analyse your problem when you don't post the actual error, and since we have to guess what your data actually is. But some general points as advise:
Using a dynamic table name is often not way DB-systems want to be used. Try thinking if the problem could be used by using a static table name and adding an additional key column to your table. Into that field you can put what you did now as a dynamic table name. This way the DB might be able to better optimize your queries, and your queries are less likely to get errors (no need to create extra tables on the fly for once, which is not a cheap thing to do. Also you would not have a need for dynamic DROP TABLE queries, which could be a security risk.
So my advice to solve your problem would be to actually work around it by trying to get rid of dynamic table names altogether.
Another problem you have is that you are using python string formatting and not parameters to the query itself. That is a security problem in itself (SQL-Injections), but also is the problem of your syntax error. When you use numbers, your expression evaluates to
INSERT INTO table_name VALUES (100, 200)
Which is valid SQL. But with strings you get
INSERT INTO table_name VALUES (Some Text, some more text)
which is not valid (since you have no quotes ' around the strings.
To get rid of your syntax problem and of the sql-injection-problem, don't add the values to the string, pass them as a list to execute():
cursor.execute("INSERT INTO table_name VALUES (%s,%s)", (key, data[0][key]))
If you must have a dynamic table name, put that in your query string first (e.g. with % formatting), and give the actual values for your query as parameters as above (since I cannot imagine that execute will accept the table name as a parameter).
To put it in some simple sample code. Right now you are trying to do it like this:
# don't do this, this won't even work!
table_name = 'some_table'
user_name = 'Peter Smith'
user_age = 47
query = "INSERT INTO %s VALUES (%s, %s)" % (table_name, user_name, user_age)
cursor.execute(query)
That creates query
INSERT INTO some_table VALUES (Peter Smith, 100)
Which cannot work, because of the unquoted string. So you needed to do:
# DON'T DO THIS, it's bad!
query = "INSERT INTO %s VALUES ('%s', %s)" % (table_name, user_name, user_age)
That's not a good idea, because you need to know where to put quotes and where not (which you will mess up at some point). Even worse, imagine a user named named Connor O'Neal. You would get a syntax error:
INSERT INTO some_table VALUES ('Connor O'Neal', 100)
(This is also the way sql-injections are used to crush your system / steal your data). So you would also need to take care of escaping the values that are strings. Getting more complicated.
Leave those problems to python and mysql, by passing the date (not the table name) as arguments to execute!
table_name = 'some_table'
user_name = 'Peter Smith'
user_age = 47
query = "INSERT INTO " + table_name + " VALUES (%s, %s)"
cursor.execute(query, (user_name, user_age))
This way you can even pass datetime objects directly. There are other ways to put the data than using %s, take a look at this examples http://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursor-execute.html (that is python3 used there, I don't know which you use - but except of the print statements it should work with python2 as well, I think).