I've read the advice here about using parametrized execute call to do all the SQL escaping for you, but this seems to work only when you know the number of columns in advance.
I'm looping over CSV files, one for each table, and populating a local DB for testing purposes. Each table has different numbers of columns, so I can't simply use:
sql = "INSERT INTO TABLE_A VALUES (%s, %s)"
cursor.execute(sql, (val1, val2))
I can build up an sql statement as a string quite flexibly, but this doesn't give me the use of cursor.execute's SQL-escaping facilities, so if the input contains apostrophes or similar, it fails.
It seems like there should be a simple way to do this. Is there?
If you know the number of parameters, you can create a list of them:
count = ...
sql = "INSERT INTO ... VALUES(" + ",".join(count * ["?"]) + ")"
params = []
for i in ...:
params += ['whatever']
cursor.execute(sql, params)
Related
I am inserting thousands of rows, timing and speed is very important. I have found through benchmarking that postgres can ingest my rows faster using execute() instead of executemany()
This works well for me:
...
def insert(self, table, columns, values):
conn = self.connectionPool.getconn()
conn.autocommit = True
try:
with conn.cursor() as cursor:
query = (
f'INSERT INTO {table} ({columns}) '
f'VALUES {values} '
f'ON CONFLICT DO NOTHING;'
).replace('[', '').replace(']', '') # Notice the replace x2 to get rid of the list brackets
print(query)
cursor.execute(query)
finally:
cursor.close()
self.connectionPool.putconn(conn)
...
self.insert('types', 'name, created_at', rows)
After the double replace, printing query returns something like this and the rows are ingested:
INSERT INTO types (name, created_at) VALUES ('TIMER', '2022-04-09 03:19:49'), ('Sequence1', '2022-04-09 03:19:49') ON CONFLICT DO NOTHING;
Is my approach secure? Is there a more pythonic implementation using execute?
No, this isn’t secure or even reliable – Python repr isn’t compatible with PostgreSQL string syntax (try some strings with single quotes, newlines, or backslashes).
Consider passing array parameters instead and using UNNEST:
cursor.execute(
"INSERT INTO types (name, created_at)"
" SELECT name, created_at FROM UNNEST (%(names)s, %(created_ats)s) AS t",
{
'names': ['TIMER', 'Sequence1', ...],
'created_ats': ['2022-04-09 03:19:49', ...],
})
This is the best solution, as the query doesn’t depend on the parameters (can be prepared and cached, statistics can be easily grouped, makes the absence of SQL injection vulnerability obvious, can easily log queries without data).
Failing that, build a query that’s only dynamic in the number of parameters, like VALUES ((%s, %s, ...), (%s, %s, ...), ...). Note that PostgreSQL has a parameter limit, so you might need to produce these in batches.
Failing that, use psycopg2.sql.Literal.
I'm trying to transfer a user input from a python code to a table in postgresql
What I want to do is place an input() in this code and make it's value go to the comment (#) in the code.
conn = psycopg2.connect(
host="localhost",
database="Twitterzuil",
user="postgres",
password="")
cur = conn.cursor()
cur.execute("INSERT INTO Bericht2 (name) VALUES (#THIS IS WHERE I WANT THE INPUT TO GO)");
conn.commit()
I have no idea how, I'm really a beginner in all this so any help is appreciated
I believe what you are asking about is called string interpolation. Using f-style format, this might look like
new_name = "'bob'" # need single quotes for SQL strings
sql = f"INSERT INTO Bericht2 (name) VALUES ({new_name})" # => sql == "INSERT INTO Bericht2 (name) VALUES ('bob')"
cur.execute(sql)
Note the f at the start of the string, when you do this expressions inside {} pairs get replaced with their python values (tutorial). There are also string formatting approaches involving % substitution and the .format method on strings.
If you are doing anything beyond the basics you should look into using the SQLAlchemy package; here's the link to their insert api. Using SQLAlchemy will help reduce the risks that can come with manually constructing SQL queries.
Example from "Inserting Rows with SQLAlchemy"
from sqlalchemy import insert
stmt = insert(user_table).values(name='spongebob', fullname="Spongebob Squarepants")
with engine.connect() as conn:
result = conn.execute(stmt)
conn.commit()
I'm trying to insert a list into separate columns of a database
print inserter
params = ['%s' for item in inserter]
sql_query = 'INSERT INTO tablename (coloumn1, coloumn2, coloumn3, coloumn4, coloumn5, coloumn6, coloumn7) VALUES (%s,%s,%s,%s,%s,%s,%s);' % ','.join(params)
cursor.execute(sql_query)
db.commit
But keep getting the error
not enough arguments for format string
Anyone know what I am doing wrong?
Anyone know what I am doing wrong?
You are using string interpolation in a query.
This is bad, mainly for 2 reasons:
It is erroneous as you see. The python interpreter is confused between the %s for the interpolation and the %s for the sql parameters.
It makes your code vulnerable for sql injection.
You should use a parametrized query:
sql_query = '''INSERT INTO tablename (coloumn1, coloumn2, coloumn3,
coloumn4, coloumn5, coloumn6, coloumn7)
VALUES (%s,%s,%s,%s,%s,%s,%s);'''
cursor.execute(sql_query, inserter) # assuming inserter is a tuple/list of values
I am receiving an error when trying to write data to a database table when using a variable for the table name that I do not get when using a static name. For some reason on the line where I insert, if I insert an integer as the column values the code runs and the table is filled, however, if I try to use a string I get a SQL syntax error
cursor = db.cursor()
cursor.execute('DROP TABLE IF EXISTS %s' %data[1])
sql ="""CREATE TABLE %s (IP TEXT, AVAILIBILITY INT)""" %data[1]
cursor.execute(sql)
for key in data[0]:
cur_ip = key.split(".")[3]
cursor.execute("""INSERT INTO %s VALUES (%s,%s)""" %(data[1],key,data[0][key]))
db.commit()
the problem is where I have %(data[1], key, data[0][key]) any ideas?
It's a little hard to analyse your problem when you don't post the actual error, and since we have to guess what your data actually is. But some general points as advise:
Using a dynamic table name is often not way DB-systems want to be used. Try thinking if the problem could be used by using a static table name and adding an additional key column to your table. Into that field you can put what you did now as a dynamic table name. This way the DB might be able to better optimize your queries, and your queries are less likely to get errors (no need to create extra tables on the fly for once, which is not a cheap thing to do. Also you would not have a need for dynamic DROP TABLE queries, which could be a security risk.
So my advice to solve your problem would be to actually work around it by trying to get rid of dynamic table names altogether.
Another problem you have is that you are using python string formatting and not parameters to the query itself. That is a security problem in itself (SQL-Injections), but also is the problem of your syntax error. When you use numbers, your expression evaluates to
INSERT INTO table_name VALUES (100, 200)
Which is valid SQL. But with strings you get
INSERT INTO table_name VALUES (Some Text, some more text)
which is not valid (since you have no quotes ' around the strings.
To get rid of your syntax problem and of the sql-injection-problem, don't add the values to the string, pass them as a list to execute():
cursor.execute("INSERT INTO table_name VALUES (%s,%s)", (key, data[0][key]))
If you must have a dynamic table name, put that in your query string first (e.g. with % formatting), and give the actual values for your query as parameters as above (since I cannot imagine that execute will accept the table name as a parameter).
To put it in some simple sample code. Right now you are trying to do it like this:
# don't do this, this won't even work!
table_name = 'some_table'
user_name = 'Peter Smith'
user_age = 47
query = "INSERT INTO %s VALUES (%s, %s)" % (table_name, user_name, user_age)
cursor.execute(query)
That creates query
INSERT INTO some_table VALUES (Peter Smith, 100)
Which cannot work, because of the unquoted string. So you needed to do:
# DON'T DO THIS, it's bad!
query = "INSERT INTO %s VALUES ('%s', %s)" % (table_name, user_name, user_age)
That's not a good idea, because you need to know where to put quotes and where not (which you will mess up at some point). Even worse, imagine a user named named Connor O'Neal. You would get a syntax error:
INSERT INTO some_table VALUES ('Connor O'Neal', 100)
(This is also the way sql-injections are used to crush your system / steal your data). So you would also need to take care of escaping the values that are strings. Getting more complicated.
Leave those problems to python and mysql, by passing the date (not the table name) as arguments to execute!
table_name = 'some_table'
user_name = 'Peter Smith'
user_age = 47
query = "INSERT INTO " + table_name + " VALUES (%s, %s)"
cursor.execute(query, (user_name, user_age))
This way you can even pass datetime objects directly. There are other ways to put the data than using %s, take a look at this examples http://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursor-execute.html (that is python3 used there, I don't know which you use - but except of the print statements it should work with python2 as well, I think).
I am working on a script to parse a csv file and generate input for a MySQL table.
I import the data via csv.reader, so every row is a list of strings.
I want to iterate over the rows and put different entries into the database.
I can get the following test to work:
sql = "INSERT INTO `testSmall` (`idtestSmall`, `column1`, `column2`) VALUES (1, 'entry1', 'entry2');"
cursor.execute (sql)
So my SQL connection works and the principle SQL syntax is ok.
I can also access the entries I want to put in there, and they are correct and of the data type I expect.
However, I don't seem to be able to use the same SQL syntax with variables within the iterations:
allData = csv.reader(open('TestTable.csv', 'rb'), delimiter=',', quotechar='|')
for row in allData:
sql = "INSERT INTO `testSmall` (`idtestSmall`, `column1`, `column2`) VALUES (row[0], row[1], row[2]);"
cursor.execute (sql)
This generates a Syntax Error:
Error 1064: You have an error in your SQL syntax; check the manual that corresponds to our MySQL server version for the right syntax to use near '[0], row[1], row[2])' at line 1
But the data types are correct and the SQL syntax is the same as in the working example...
Can anyone tell me what I'm doing wrong and how to make it work?
(In the end, I want to not only insert the pure csv entries but also derived values, which is why I'm not just using mysql bulk import.)
Thanks in advance for your help!
use:
sql = "INSERT INTO `testSmall` (`idtestSmall`, `column1`, `column2`) VALUES (?, ?, ?);"
cursor.execute (sql, (row[0], row[1], row[2]))
The questionmark is a placeholder. An extra advantage of using placeholders, is that they automatically make your input 'safe', by escaping qoutes etc.
Right now, you are using the row[0], row[1], row[2] as a string with the text "row[0], row[1], row[2]", instead of telling python to use the values of these variables.
Also, if you want to use rows of multiple lengths, or if you want to be able to easily change the size of your input list, you can dynamically create the placeholders:
sql = "INSERT INTO testSmall VALUES (%s);" % ', '.join('?' for _ in row)
cursor.execute (sql, row)
The way you are doing it, row[n]s don't refer to the variable row, but they are just a piece of string sent as it is to MySQL. (I bet you come from PHP background and expect the double quotes to replace your variables with their values).
You could do this to insert the values inside the string (any string):
sql = "INSERT INTO `testSmall` (`idtestSmall`, `column1`, `column2`) VALUES (%s, %s, %s);" % row # will map each %s to the `n`th element in `row`
(this will not work, be careful, because if row[0] is abc, that string will not be enclosed in quotes, so MySQL will not interpret it as a string). Try printing the sql variable, and copy/paste it into the mysql prompt to see if it will work.
However, when used with MySQL, you better escape these, like so:
sql = "INSERT INTO `testSmall` (`idtestSmall`, `column1`, `column2`) VALUES (%s, %s, %s);"
cursor.execute(sql, row)
You can read more in the docs.