Using Python and psycopg2 I am trying to build a dynamic SQL query to insert rows into tables.
The variables are:
1. Table name
2. Variable list of column names
3. Variable list of values, ideally entering multiple rows in one statement
The problems I have come across are the treatment of string literals from Python to SQL and psycopg2 trying to avoid you exposing your code to SQL injection attacks.
Using the SQL module from psycopg2, I have resolved dynamically adding the Table name and List of columns. However I am really struggling with adding the VALUES. Firstly the values are put into the query as %(val)s and seem to be passed literally like this to the database, causing an error.
Secondly, I would then like to be able to add multiple rows at once.
Code below. All help much appreciated :)
import psycopg2 as pg2
from psycopg2 import sql
conn = pg2.connect(database='my_dbo',user='***',password='***')
cols = ['Col1','Col2','Col3']
vals = ['val1','val2','val3']
#Build query
q2 = sql.SQL("insert into my_table ({}) values ({})") \
.format(sql.SQL(',').join(map(sql.Identifier, cols)), \
sql.SQL(',').join(map(sql.Placeholder,vals)))
When I print this string as print(q2.as_string(conn)) I get:
insert into my_table ("Col1","Col2","Col3") values %(val1)s,%(val2)s,%(val3)s
And then when i try and a execute such a string I get the following error:
ProgrammingError: syntax error at or near "%"
LINE 1: ... ("Col1","Col2","Col3") values (%(val1)s...
^
Ok I solved this. Firstly use Literal rather than Placeholder, secondly put your row values together as tuples within a tuple, loop through adding each tuple to a list as literals and then drop in at the end when building the query.
Related
I have a question regarding performing insert into a table in BigQuery using the DBAPI .
The simple insert of 1 value I was able to preform , my question is how to preform insert of list of values.
I couldn't find an example on how to do it, I want to preform something like :
query ="insert into test.test values (%s)"
self._cursor.execute(query,(('4'),('3')))
But I am getting an error
Thanks,
Nir
You can try the command executemany. This method is used to insert multiple rows in a single operation.
Your code should look something like:
query ="insert into test.test values (%s)"
self._cursor.executemany(query,[('4'),('3')])
Also what is the error that you are getting?
I am trying to query an Oracle DB table from sqlalchemy with
session.query(PAY).filter_by(REGION_CODE=record['REGION_CODE'],
AOID=record['AOID'],
COMPANY_CODE=record['COMPANY_CODE']).all()
it returns 1 value, however if do a raw sql
select * from schema.table_name where REGION_CODE = '{record['REGION_CODE']}'
and AOID = '{record['AOID']}' and COMPANY_CODE = '{record['COMPANY_CODE']}
The database has 2 records I get two results. The results have identical values but are 2 different records.
Any one have any idea why the query only returns the single value?
It is possible that SQLAlchemy is prepending a "distinct" clause to the query which will eliminate duplicate rows. You can find out for sure by setting the environment variable DPI_DEBUG_LEVEL to the value 16 before running the two examples. This will show the SQL statements that are being executed and you can confirm my suspicions!
I'm working on a bit of python code to run a query against a redshift (postgres) SQL database, and I'm running into an issue where I can't strip off the surrounding single quotes from a variable I'm passing to the query. I'm trying to drop a number of tables from a list. This is the basics of my code:
def func(table_list):
drop_query = 'drop table if exists %s' #loaded from file
table_name = table_list[0] #table_name = 'my_db.my_table'
con=psycopg2.connect(dbname=DB, host=HOST, port=PORT, user=USER, password=PASS)
cur=con.cursor()
cur.execute(drop_query, (table_name, )) #this line is giving me trouble
#cleanup statements for the connection
table_list = ['my_db.my_table']
when func() gets called, I am given the following error:
syntax error at or near "'my_db.my_table'"
LINE 1: drop table if exists 'my_db.my_table...
^
Is there a way I can remove the surrounding single quotes from my list item?
for the time being, I've done it (what think is) the wrong way and used string concatenation, but know this is basically begging for SQL-injection.
This is not how psycopg2 works. You are using a string operator %s to replace with a string. The reason for this is to tokenize your string safely to avoid SQL injection, psycopg2 handles the rest.
You need to modify the query before it gets to the execute statement.
drop_query = 'drop table if exists {}'.format(table_name)
I warn you however, do not allow these table names to be create by outside sources, or you risk SQL injection.
However a new version of PSYCOPG2 kind of allows something similar
http://initd.org/psycopg/docs/sql.html#module-psycopg2.sql
from psycopg2 import sql
cur.execute(
sql.SQL("insert into {} values (%s, %s)").format(sql.Identifier('my_table')),[10, 20]
)
This question already has answers here:
imploding a list for use in a python MySQLDB IN clause
(8 answers)
Closed 1 year ago.
I want to insert a list in my database but I can't.
Here is an example of what I need:
variable_1 = "HELLO"
variable_2 = "ADIOS"
list = [variable_1,variable_2]
INSERT INTO table VALUES ('%s') % list
Can something like this be done? Can I insert a list as a value?
When I try it, an error says that is because of an error in MySQL syntax
The answer to your original question is: No, you can't insert a list like that.
However, with some tweaking, you could make that code work by using %r and passing in a tuple:
variable_1 = "HELLO"
variable_2 = "ADIOS"
varlist = [variable_1, variable_2]
print "INSERT INTO table VALUES %r;" % (tuple(varlist),)
Unfortunately, that style of variable insertion leaves your code vulnerable to SQL injection attacks.
Instead, we recommend using Python's DB API and building a customized query string with multiple question marks for the data to be inserted:
variable_1 = "HELLO"
variable_2 = "ADIOS"
varlist = [variable_1,variable_2]
var_string = ', '.join('?' * len(varlist))
query_string = 'INSERT INTO table VALUES (%s);' % var_string
cursor.execute(query_string, varlist)
The example at the beginning of the SQLite3 docs shows how to pass arguments using the question marks and it explains why they are necessary (essentially, it assures correct quoting of your variables).
Your question is not clear.
Do you want to insert the list as a comma-delimited text string into a single column in the database? Or do you want to insert each element into a separate column? Either is possible, but the technique is different.
Insert comma-delimited list into one column:
conn.execute('INSERT INTO table (ColName) VALUES (?);', [','.join(list)])
Insert into separate columns:
params = ['?' for item in list]
sql = 'INSERT INTO table (Col1, Col2. . .) VALUES (%s);' % ','.join(params)
conn.execute(sql, list)
both assuming you have established a connection name conn.
A few other suggestions:
Try to avoid INSERT statements that do not list the names and order of the columns you're inserting into. That kind of statement leads to very fragile code; it breaks if you add, delete, or move columns around in your table.
If you're inserting a comma-separted list into a single-field, that generally violates principals of database design and you should use a separate table with one value per record.
If you're inserting into separate fields and they have names like Word1 and Word2, that is likewise an indication that you should be using a separate table instead.
Never use direct string substitution to create SQL statements. It will break if one of the values is, for example o'clock. It also opens you to attacks by people using SQL injection techniques.
You can use json.dumps to convert a list to json and write the json to db.
For example:
insert table example_table(column_name) values(json.dumps(your_list))
I have the following query for inserting values into sqlite3 db, which is made in python code;
insert into 'SplCharTest' ('Id','name','address','telefon') values (1.0, u'cust01', u'addr01 strasse 48908', 2131234213.0)
I am getting the following error when I try to execute this query;
sqlite3CursorObj.execute(dataInsertQueryForSheet)
sqlite3.OperationalError: near "'cust01'": syntax error
I noticed that the presence of u (unicode) in front of 'cust01' is making the problem. Could anybody please tell me how to insert unicode value into sqlite db. Simple usage of 'str' is not possible as I am getting the unicode values from an excel file using openpyxl package and there could be special characters.
Below given is my python code
dataInsertQueryForSheet = "insert into '"+tableNameForSheet+"' ("+columnNamesCSV+") values (%s)" % unicode(rowDataList)[1:(rowDataStrLen-1)]
sqlite3CursorObj.execute(dataInsertQueryForSheet)
Here variable tableNameForSheet contains the table name, columnNamesCSV contains all the columns names in CSV format and rowDataList contains all values to be inserted as an array list.
Any help will be very much appreciated. Thanks in advance.
SQLite always uses Unicode.
The easiest way to use strings while avoiding formatting problems or SQL injection attacks is to use parameters:
sql = "insert into "+tableNameForSheet+" ("+columnNamesCSV+") values (?,?,?,?)"
parameters = [1, u'cust01', u'addr01 strasse 48908', "2131234213"]
cursor.execute(sql, parameters)