We Keep Coding

how to specify table name using %s [duplicate]

I have the following code, using pscyopg2:
sql = 'select %s from %s where utctime > %s and utctime < %s order by utctime asc;'
data = (dataItems, voyage, dateRangeLower, dateRangeUpper)
rows = cur.mogrify(sql, data)
This outputs:
select 'waterTemp, airTemp, utctime' from 'ss2012_t02' where utctime > '2012-05-03T17:01:35+00:00'::timestamptz and utctime < '2012-05-01T17:01:35+00:00'::timestamptz order by utctime asc;
When I execute this, it falls over - this is understandable, as the quotes around the table name are illegal.
Is there a way to legally pass the table name as a parameter, or do I need to do a (explicitly warned against) string concatenation, ie:
voyage = 'ss2012_t02'
sql = 'select %s from ' + voyage + ' where utctime > %s and utctime < %s order by utctime asc;'
Cheers for any insights.

According to the official documentation:
If you need to generate dynamically an SQL query (for instance
choosing dynamically a table name) you can use the facilities
provided by the psycopg2.sql module.
The sql module is new in psycopg2 version 2.7. It has the following syntax:
from psycopg2 import sql
cur.execute(
sql.SQL("insert into {table} values (%s, %s)")
.format(table=sql.Identifier('my_table')),
[10, 20])
More on: https://www.psycopg.org/docs/sql.html#module-usage
[Update 2017-03-24: AsIs should NOT be used to represent table or fields names, the new sql module should be used instead: https://stackoverflow.com/a/42980069/5285608 ]
Also, according to psycopg2 documentation:
Warning: Never, never, NEVER use Python string concatenation (+) or string parameters interpolation (%) to pass variables to a SQL query string. Not even at gunpoint.

Per this answer you can do it as so:
import psycopg2
from psycopg2.extensions import AsIs
#Create your connection and cursor...
cursor.execute("SELECT * FROM %(table)s", {"table": AsIs("my_awesome_table")})

The table name cannot be passed as a parameter, but everything else can. Thus, the table name should be hard coded in your app (Don't take inputs or use anything outside of the program as a name). The code you have should work for this.
On the slight chance that you have a legitimate reason to take an outside table name, make sure that you don't allow the user to directly input it. Perhaps an index could be passed to select a table, or the table name could be looked up in some other way. You are right to be wary of doing this, however. This works, because there are relatively few table names around. Find a way to validate the table name, and you should be fine.
It would be possible to do something like this, to see if the table name exists. This is a parameterised version. Just make sure that you do this and verify the output prior to running the SQL code. Part of the idea for this comes from this answer.
SELECT 1 FROM information_schema.tables WHERE table_schema = 'public' and table_name=%s LIMIT 1

This is a workaround I have used in the past
query = "INSERT INTO %s (col_1, col_2) VALUES (%%s, %%s)" % table_name
cur.execute(query, (col_1_var, col_2_var))
Hope it help :)

This is a small addition to #Antoine Dusséaux's answer. If you want to pass two (unquoted) parameters in a SQL query, you can do it as follows: -
query = sql.SQL("select {field} from {table} where {pkey} = %s").format(
field=sql.Identifier('my_name'),
table=sql.Identifier('some_table'),
pkey=sql.Identifier('id'))
As per the documentation,
Usually you should express the template of your query as an SQL
instance with {}-style placeholders and use format() to merge the
variable parts into them, all of which must be Composable subclasses.
You can still have %s-style placeholders in your query and pass values
to execute(): such value placeholders will be untouched by format()
Source: https://www.psycopg.org/docs/sql.html#module-usage
Also, please keep this in mind while writing queries.

I have created a little utility for preprocessing of SQL statements with variable table (...) names:
from string import letters
NAMECHARS = frozenset(set(letters).union('.'))
def replace_names(sql, **kwargs):
"""
Preprocess an SQL statement: securely replace table ... names
before handing the result over to the database adapter,
which will take care of the values.
There will be no quoting of names, because this would make them
case sensitive; instead it is ensured that no dangerous chars
are contained.
>>> replace_names('SELECT * FROM %(table)s WHERE val=%(val)s;',
... table='fozzie')
'SELECT * FROM fozzie WHERE val=%(val)s;'
"""
for v in kwargs.values():
check_name(v)
dic = SmartDict(kwargs)
return sql % dic
def check_name(tablename):
"""
Check the given name for being syntactically valid,
and usable without quoting
"""
if not isinstance(tablename, basestring):
raise TypeError('%r is not a string' % (tablename,))
invalid = set(tablename).difference(NAMECHARS)
if invalid:
raise ValueError('Invalid chars: %s' % (tuple(invalid),))
for s in tablename.split('.'):
if not s:
raise ValueError('Empty segment in %r' % tablename)
class SmartDict(dict):
def __getitem__(self, key):
try:
return dict.__getitem__(self, key)
except KeyError:
check_name(key)
return key.join(('%(', ')s'))
The SmartDict object returns %(key)s for every unknown key, preserving them for the value handling. The function could check for the absence of any quote characters, since all quoting now should be taken care of ...

If you want to pass the table name as a parameter, you can use this wrapper:
class Literal(str):
def __conform__(self, quote):
return self
#classmethod
def mro(cls):
return (object, )
def getquoted(self):
return str(self)
Usage: cursor.execute("CREATE TABLE %s ...", (Literal(name), ))

You can just use the module format for the table name and then use the regular paramaterization for the execute:
xlist = (column, table)
sql = 'select {0} from {1} where utctime > %s and utctime < %s order by utctime asc;'.format(xlist)
Keep in mind if this is exposed to the end user, you will not be protected from SQL injection unless you write for it.

Surprised no one has mentioned doing this:
sql = 'select {} from {} where utctime > {} and utctime < {} order by utctime asc;'.format(dataItems, voyage, dateRangeLower, dateRangeUpper)
rows = cur.mogrify(sql)
format puts in the string without quotations.

Sqlite3 / Python -- increment column by 1 using '?' and UPDATE, getting syntax error [duplicate]

I want to dynamically choose what table to use in a SQL query, but I just keep getting error however I am trying to format this. Also tried %s instead of ?.
Any suggestions?
group_food = (group, food)
group_food_new = (group, food, 1)
with con:
cur = con.cursor()
tmp = cur.execute("SELECT COUNT(Name) FROM (?) WHERE Name=?", group_food)
if tmp == 0:
cur.execute("INSERT INTO ? VALUES(?, ?)", group_food_new)
else:
times_before = cur.execute("SELECT Times FROM ? WHERE Name=?", group_food)
group_food_update = (group, (times_before +1), food)
cur.execute("UPDATE ? SET Times=? WHERE Name=?", group_food_update)

You cannot use SQL parameters to be placeholders in SQL objects; one of the reasons for using a SQL parameters is to escape the value such that the database can never mistake the contents for a database object.
You'll have to interpolate the database objects separately; escape your identifiers by doubling any " double quote parameters and use
cur.execute('SELECT COUNT(Name) FROM "{}" WHERE Name=?'.format(group.replace('"', '""')), (food,))
and
cur.execute('INSERT INTO "{}" VALUES(?, ?)'.format(group.replace('"', '""')), (food, 1))
and
cur.execute('UPDATE "{}" SET Times=? WHERE Name=?'.format(group.replace('"', '""')),
(times_before + 1, food))
The ".." double quotes are there to properly demark an identifier, even if that identifier is also a valid keyword; any existing " characters in the name must be doubled; this also helps de-fuse SQL injection attempts.
However, if your object names are user-sourced, you'll have to do your own (stringent) validation on the object names to prevent SQL injection attacks here. Always validate them against existing objects in that case.
You should really consider using a project like SQLAlchemy to generate your SQL instead; it can take care of validating object names and rigorously protect you from SQL injection risks. It can load your table definitions up front so it'll know what names are legal:
from sqlalchemy import create_engine, func, select, MetaData
engine = create_engine('sqlite:////path/to/database')
meta = MetaData()
meta.reflect(bind=engine)
conn = engine.connect()
group_table = meta.tables[group] # can only find existing tables
count_statement = select([func.count(group_table.c.Name)], group_table.c.Name == food)
count, = conn.execute(count_statement).fetchone()
if count:
# etc.

What are the values of group and food? The guidelines say to make the question as such others can benefit from it, for that to be the case here we need the values of group and food.
It seems you use the Python String formatter instead of SQL parameters for table names even though https://docs.python.org/3/library/sqlite3.html#module-sqlite3 says using the String formatter in unsafe.
# Never do this -- insecure!
symbol = 'RHAT'
c.execute("SELECT * FROM stocks WHERE symbol = '%s'" % symbol)
# Do this instead
t = ('RHAT',)
c.execute('SELECT * FROM stocks WHERE symbol=?', t)
Using %s as a placeholder then putting % outside the string with the variables in Python2 has been Replaced in Python3 with .format() function with the variables as arguments.

I've found building the SQL query up as a text string and then passing this string into the c.execute() function works.
querySelect = "SELECT * FROM " + str(your_table_variable)
queryWhere = " WHERE " + str(variableName) + " = " str(variableValue)
query = querySelect + queryWhere
c.execute(query)
I don't know the security situation around it though (re injection) and I'm sure there are probably better ways of doing this.

Alternating SQL queries

I'm looking for a way to implement alternating SQL queries - i.e. a function that allows me to filter entries based on different columns. Take the following example:
el=[["a","b",1],["a","b",3]]
def save_sql(foo):
with sqlite3.connect("fn.db") as db:
cur=db.cursor()
cur.execute("CREATE TABLE IF NOT EXISTS et"
"(var1 VARCHAR, var2 VARCHAR, var3 INT)")
cur.executemany("INSERT INTO et VALUES "
"(?,?,?)", foo)
db.commit()
def load_sql(v1,v2,v3):
with sqlite3.connect("fn.db") as db:
cur=db.cursor()
cur.execute("SELECT * FROM et WHERE var1=? AND var2=? AND var3=?", (v1,v2,v3))
return cur.fetchall()
save_sql(el)
Now if I were to use load_sql("a","b",1), it would work. But assume I want to only query for the first and third column, i.e. load_sql("a",None,1) (the None is just intended as a placeholder) or only the last column load_sql(None,None,5), this wouldn't work.
This could of course be done with if statements checking which variables were supplied in the function call, but in tables with larger amounts of columns, this might get messy.
Is there a good way to do this?

What if load_sql() would accept an arbitrary number of keyword arguments, where keyword argument names would correspond to column names. Something along these lines:
def load_sql(**values):
with sqlite3.connect("fn.db") as db:
cur = db.cursor()
query = "SELECT * FROM et"
conditions = [f"{column_name} = :{column_name}" for column_name in values]
if conditions:
query = query + " WHERE " + " AND ".join(conditions)
cur.execute(query, values)
return cur.fetchall()
Note that here we trust keyword argument names to be valid and existing column names (and string-format them into the query) which may potentially be used as an SQL injection attack vector.
As a side note, I cannot stop but think that this feels like a reinventing-the-wheel step towards an actual ORM. Look into lightweight PonyORM or Peewee abstraction layers between Python and a database.

It will inevitably get messy if you want your SQL statements to remain sanitized/safe, but as long as you control your function signature it can remain reasonably safe, e.g.:
def load_sql(var1, var2, var3):
fields = dict(field for field in locals().items() if field[1] is not None)
query = "SELECT * FROM et"
if fields: # if at least one field is not None:
query += " WHERE " + " AND ".join((k + "=?" for k in fields.keys()))
with sqlite3.connect("fn.db") as db:
cur = db.cursor()
cur.execute(query, fields.values())
return cur.fetchall()
You can replace the function signature with load_sql(**kwargs) and then use kwargs.items() instead of locals.items() so that you can pass arbitrary column names, but that can be very dangerous and is certainly not recommended.

sqlite3 error when attempting to insert info, incorrect number of bindings supplied

I am trying to create a table in sqlite with some data in a .txt file. I am using python, and this is my code:
import sqlite3
conn = sqlite3.connect('TEST_Inventory.sqlite')
cur = conn.cursor()
cur.execute('''
DROP TABLE IF EXISTS InventoryData''')
cur.execute('''
CREATE TABLE `InventoryData` (
`INV_ID` INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
`Inventory_Name` TEXT)''')
fname = ('NameINV.txt')
fh = open(fname)
for line in fh:
name = line.split()
print name
cur.execute('''INSERT INTO InventoryData (Inventory_Name)
VALUES ( '?' )''', (name,))
cur.commit()
However, every time I try to run this code I get this error:
sqlite3.ProgrammingError: Incorrect number of bindings supplied. The current statement uses 0, and there are 1 supplied.
What exactly is going on here?

Sql statements can be constructed in 2 ways.
By concatenating string (unsafe, easy for sql injection attacks)
ex: sql_string = 'INSERT INTO TABLE_1 (COL1, COL2) VALUES(\'' + value1 + '\',\'' + value2 + '\')'
By using placeholders and bind variables (recommended)
ex: sql_string = 'INSERT INTO TABLE_1(COL1,COL2) VALUES (?,?)' and supply bind variables as (value1,value2)
When you use bind variables, there is no need to quote them (to identify them as string/text data). Bind variables will be substituted at database, during execution of statement.
To answer precisely, when you use single-quote around placeholder '?', it will be treated as a text and program doesn't know where to bind the data you supplied while executing query. Just use ? without quotes, so that, parser finds placeholders to substitute with data and code will execute.
cur.execute('''INSERT INTO InventoryData (Inventory_Name)
VALUES ( ? )''', (name,))

Don't quote the placeholder; quoting is taken care of by the database driver when replacing the placeholder with the value:
cur.execute('''INSERT INTO InventoryData (Inventory_Name)
VALUES ( ? )''', (name,))

Using a WHERE ___ IN ___ statement

I'm trying to figure out how to properly use a WHERE _ IN _ statement
Definition:
c.execute('''CREATE TABLE IF NOT EXISTS tab (
_id integer PRIMARY KEY AUTOINCREMENT,
obj text NOT NULL
) ;''')
I'm trying to do something like this:
list_of_vars=['foo','bar']
statement="SELECT * FROM tab WHERE obj IN (?)"
c.execute(statement,"'"+"','".join(list_of_vars)+"'")
Alternatively, I've also tried this, which directly evaluates to the above
statement="SELECT * FROM tab WHERE obj IN (?)"
c.execute(statement,"'foo','bar'")
The error I am getting is:
sqlite3.ProgrammingError: Incorrect number of bindings supplied. The current statement uses 1, and there are 9 supplied
This is giving me an error. When I do it this way, it works, but this is not recommended as it is vulnerable to a SQL injection attack.
statement="SELECT * FROM tab WHERE obj IN ("+"'"+"','".join(statement)+"'"+")

You need to create enough parameters to match your list of vars:
statement = "SELECT * FROM tab WHERE obj IN ({0})".format(', '.join(['?'] * len(list_of_vars)))
c.execute(statement, list_of_vars)
Note that you pass in list_of_vars as the parameter values list. Using the ', '.join() we generate a string of ? characters separated by commas, then use .format() to insert that into the statement.
For a long list of variables, it may be more efficient to use a temporary table to hold those values, then use a JOIN against the temporary table rather than an IN clause with bind parameters.

FYI, pymysql with MySQL user.
query ="SELECT * FROM tab WHERE obj IN %s"
cursor.execute(query, (['foo','bar'],))
same as
cursor.execute(query, (list_of_var,))
I'm not sure about sqlite3,
this may work,
query ="SELECT * FROM tab WHERE obj IN ?"
cursor.execute(query, (['foo','bar'],))
or
cursor.execute(query, (list_of_vars,))