Let's say we have a SQL statement that just needs to be completed with the parameters before getting executed against the DB. For instance:
sql = '''
SELECT id, price, date_out
FROM sold_items
WHERE date_out BETWEEN ? AND ?
'''
database_cursor.execute(sql, (start_date, end_date))
How do I get the string that is parsed and executed?, something like this:
SELECT id, price, date_out
FROM sold_items
WHERE date_out BETWEEN 2010-12-05 AND 2011-12-01
In this simple case it's not very important, but I have other SQL Statements much more complicated, and for debugging purposes I would like to execute them myself in my sqlite manager and check the results.
Thanks in advance
UPDATE. I learned from this web page that since Python 3.3 you can trigger printing of executed SQL with
connection.set_trace_callback(print)
Should you want to revert to silent processing, use
connection.set_trace_callback(None)
You can use another function instead of print.
SQLite never actually substitutes parameters into the SQL query string itself; the parameters' values are read directly when it executes the command.
(Formatting those values only to parse them again into the same values would be useless overhead.)
But if you want to find out how the parameters would be written in SQL, you can use the quote function; something like this:
import re
def log_and_execute(cursor, sql, *args):
s = sql
if len(args) > 0:
# generates SELECT quote(?), quote(?), ...
cursor.execute("SELECT " + ", ".join(["quote(?)" for i in args]), args)
quoted_values = cursor.fetchone()
for quoted_value in quoted_values:
s = s.replace('?', quoted_value, 1)
#s = re.sub(r'(values \(|, | = )\?', r'\g<1>' + quoted_value, s, 1)
print "SQL command: " + s
cursor.execute(sql, args)
(This code will fail if there is a ? that is not a parameter, i.e., inside a literal string. Unless you use the re.sub version, which will only match a ? after 'values (' ', ' or ' = '. The '\g<1>' puts back the text before the ? and using '\g<>' avoids clashes with quoted_values that start with a number.)
I've written a function that just fills in the question marks with the arguments.
It's weird that everyone sends you in the direction of using positional arguments, but no one thought about the need to log or preview or check queries in their totality.
Anyways, the code below assumes
That there are no '?' tokens in the query that do not serve as postional argument tokens. I'm not sure whether that is always the case.
That the value of the argument will be wrapped in quotes. This will not be the case when you use an argument for a table name for example. This usecase is unlikely though.
def compile_query(query, *args):
# test for mismatch in number of '?' tokens and given arguments
number_of_question_marks = query.count('?')
number_of_arguments = len(args)
# When no args are given, an empty tuple is passed
if len(args) == 1 and (not args[0]):
number_of_arguments = 0
if number_of_arguments != number_of_question_marks:
return f"Incorrect number of bindings supplied. The current statement uses {number_of_question_marks}, and there are {number_of_arguments} supplied."
# compile query
for a in args:
query = query.replace('?', "'"+str(a)+"'", 1)
return query
Suggested usage
query = "INSERT INTO users (name, password) VALUES (?, ?)"
# sensitive query, we need to log this for security
query_string = compile_query(query, username, password_hash)
fancy_log_function(query_string)
# execute
cursor.execute(query, username, password_hash)
What about string formatting it?
sql = """
SELECT id, price, date_out
FROM sold_items
WHERE date_out BETWEEN {0} AND {1} """.format(start_date, end_date)
"""
database_cursor.execute(sql)
Related
I have the following code, using pscyopg2:
sql = 'select %s from %s where utctime > %s and utctime < %s order by utctime asc;'
data = (dataItems, voyage, dateRangeLower, dateRangeUpper)
rows = cur.mogrify(sql, data)
This outputs:
select 'waterTemp, airTemp, utctime' from 'ss2012_t02' where utctime > '2012-05-03T17:01:35+00:00'::timestamptz and utctime < '2012-05-01T17:01:35+00:00'::timestamptz order by utctime asc;
When I execute this, it falls over - this is understandable, as the quotes around the table name are illegal.
Is there a way to legally pass the table name as a parameter, or do I need to do a (explicitly warned against) string concatenation, ie:
voyage = 'ss2012_t02'
sql = 'select %s from ' + voyage + ' where utctime > %s and utctime < %s order by utctime asc;'
Cheers for any insights.
According to the official documentation:
If you need to generate dynamically an SQL query (for instance
choosing dynamically a table name) you can use the facilities
provided by the psycopg2.sql module.
The sql module is new in psycopg2 version 2.7. It has the following syntax:
from psycopg2 import sql
cur.execute(
sql.SQL("insert into {table} values (%s, %s)")
.format(table=sql.Identifier('my_table')),
[10, 20])
More on: https://www.psycopg.org/docs/sql.html#module-usage
[Update 2017-03-24: AsIs should NOT be used to represent table or fields names, the new sql module should be used instead: https://stackoverflow.com/a/42980069/5285608 ]
Also, according to psycopg2 documentation:
Warning: Never, never, NEVER use Python string concatenation (+) or string parameters interpolation (%) to pass variables to a SQL query string. Not even at gunpoint.
Per this answer you can do it as so:
import psycopg2
from psycopg2.extensions import AsIs
#Create your connection and cursor...
cursor.execute("SELECT * FROM %(table)s", {"table": AsIs("my_awesome_table")})
The table name cannot be passed as a parameter, but everything else can. Thus, the table name should be hard coded in your app (Don't take inputs or use anything outside of the program as a name). The code you have should work for this.
On the slight chance that you have a legitimate reason to take an outside table name, make sure that you don't allow the user to directly input it. Perhaps an index could be passed to select a table, or the table name could be looked up in some other way. You are right to be wary of doing this, however. This works, because there are relatively few table names around. Find a way to validate the table name, and you should be fine.
It would be possible to do something like this, to see if the table name exists. This is a parameterised version. Just make sure that you do this and verify the output prior to running the SQL code. Part of the idea for this comes from this answer.
SELECT 1 FROM information_schema.tables WHERE table_schema = 'public' and table_name=%s LIMIT 1
This is a workaround I have used in the past
query = "INSERT INTO %s (col_1, col_2) VALUES (%%s, %%s)" % table_name
cur.execute(query, (col_1_var, col_2_var))
Hope it help :)
This is a small addition to #Antoine Dusséaux's answer. If you want to pass two (unquoted) parameters in a SQL query, you can do it as follows: -
query = sql.SQL("select {field} from {table} where {pkey} = %s").format(
field=sql.Identifier('my_name'),
table=sql.Identifier('some_table'),
pkey=sql.Identifier('id'))
As per the documentation,
Usually you should express the template of your query as an SQL
instance with {}-style placeholders and use format() to merge the
variable parts into them, all of which must be Composable subclasses.
You can still have %s-style placeholders in your query and pass values
to execute(): such value placeholders will be untouched by format()
Source: https://www.psycopg.org/docs/sql.html#module-usage
Also, please keep this in mind while writing queries.
I have created a little utility for preprocessing of SQL statements with variable table (...) names:
from string import letters
NAMECHARS = frozenset(set(letters).union('.'))
def replace_names(sql, **kwargs):
"""
Preprocess an SQL statement: securely replace table ... names
before handing the result over to the database adapter,
which will take care of the values.
There will be no quoting of names, because this would make them
case sensitive; instead it is ensured that no dangerous chars
are contained.
>>> replace_names('SELECT * FROM %(table)s WHERE val=%(val)s;',
... table='fozzie')
'SELECT * FROM fozzie WHERE val=%(val)s;'
"""
for v in kwargs.values():
check_name(v)
dic = SmartDict(kwargs)
return sql % dic
def check_name(tablename):
"""
Check the given name for being syntactically valid,
and usable without quoting
"""
if not isinstance(tablename, basestring):
raise TypeError('%r is not a string' % (tablename,))
invalid = set(tablename).difference(NAMECHARS)
if invalid:
raise ValueError('Invalid chars: %s' % (tuple(invalid),))
for s in tablename.split('.'):
if not s:
raise ValueError('Empty segment in %r' % tablename)
class SmartDict(dict):
def __getitem__(self, key):
try:
return dict.__getitem__(self, key)
except KeyError:
check_name(key)
return key.join(('%(', ')s'))
The SmartDict object returns %(key)s for every unknown key, preserving them for the value handling. The function could check for the absence of any quote characters, since all quoting now should be taken care of ...
If you want to pass the table name as a parameter, you can use this wrapper:
class Literal(str):
def __conform__(self, quote):
return self
#classmethod
def mro(cls):
return (object, )
def getquoted(self):
return str(self)
Usage: cursor.execute("CREATE TABLE %s ...", (Literal(name), ))
You can just use the module format for the table name and then use the regular paramaterization for the execute:
xlist = (column, table)
sql = 'select {0} from {1} where utctime > %s and utctime < %s order by utctime asc;'.format(xlist)
Keep in mind if this is exposed to the end user, you will not be protected from SQL injection unless you write for it.
Surprised no one has mentioned doing this:
sql = 'select {} from {} where utctime > {} and utctime < {} order by utctime asc;'.format(dataItems, voyage, dateRangeLower, dateRangeUpper)
rows = cur.mogrify(sql)
format puts in the string without quotations.
I am implementing a student database project which has multiple tables such as student,class,section etc
I wrote a delete_table function which takes parameters table name and value to delete a row from a specific table but there seems to be some sort of syntax error in my code :
def delete_tables(tab_name,attr,value):
c.execute("delete from table=:tab_name where attribute=:attr is value=:value ",{'tab_name':tab_name, 'attr': attr, 'value': value})
input :
delete_tables('section','sec_name','S1')
error text :
c.execute("delete from table=:tab_name where attribute=:attr is value=:value ",{'tab_name':tab_name, 'attr': attr, 'value': value})
sqlite3.OperationalError: near "table": syntax error
I've tried all mentioned answers and what y'all are suggesting is that it'll also be insecure even if it works out. So Do i Have to write functions to delete every table individually instead of going for one single function, and is there any other alternative to this problem where I need not keep on writing n functions for n number of tables?????
Thanks in Advance :))
The problem is that you can't use parametrized queries (that :tab_name) on things others than values (? not sure I am using the right term): table names, column names and SQL keywords are forbidden.
where age > :max_age is OK.
where :some_col > :max_age is not.
where age :comparison_operator :max_age is not OK.
Now, you can build your own query using string concatenation or f strings, but... 🧨 this is a massive, massive SQL injection risk. See Bobby Tables Not to mention that concatenating values into SQL query strings quickly runs into issues when you have to deal with characters, numbers or None. (None => NULL, characters need quotes, numbers dont).
You could possibly build a query using string substitutions that accept only known values for the table and column names and then drives the delete criteria value using a parametrized query on :value.
(While this seems restrictive, letting a random caller determine which tables to delete is just not safe in the least).
Something like:
delete_tables(tab_name,attr,value):
safe_tab_name = my_dict_of_known_table_names[tab_name]
safe_attr = my_dict_of_known_column_names[attr]
# you have to `=`, not `is` here👇
qry = f"delete from {safe_tab_name} where {safe_attr} = :value "
# not entirely sure about SQLite's bind/parametrized syntax.
# look it up if needed.
c.execute(qry, dict(value = value))
Assuming a user only enters value directly, that at least is protected from SQL injection.
You need to have a look at what will be the exact SQL command that will be executed in the python method.
For the method call delete_tables('section', 'sec_name', 'S1') the SQL command that will be generated will be
delete from table=section where attribute=sec_name is value=S1
This will be an invalid command in SQL. The correct command should be
delete from section where sec_name='S1'
So you need to change your python function accordingly. The changes that need to be done should be as follows:
def delete_tables(tab_name, attr, value):
c.execute("delete from :tab_name where :attr = ':value'",
{'tab_name': tab_name, 'attr': attr, 'value':value})
def delete_tables(tab_name, attr, value):
c.execute("delete from " + tab_name + "where " + attr + " = " + value)
I think something like that will work, the issue is that you are trying to modify an attribute but its name is always attribute, for that you would like to make it a parameter in order to properly handle it.
Hope it helped.
Edit:
Check this SQLite python
What the c.execute does is to 'execute' a SQL query, so, you can make something like c.execute("select * from clients") if you have a clients table.
execute makes a query and brings you the result set (if it is the case), so if you want to delete from your table using a normal SQL query you would type in the console delete from clients where client_id = 12 and that statement will delete the client with id equal to 12.
Now, if you are using SQLite in python, you will do
c.execute("delete from clients where client_id = 12")
but as you wish it to be for any table and any field (attribute) it turns in the table name, the field name and the value of that field being variables.
tableName = "clients"
field = "client_id"
value = "12" #must be string because you would have to cast it from int in the execute
"""
if value is a varchar you must write
value = "'12'" because the '' are needed.
"""
c.execute("delete from " + tableName + " where " + field + " = " + value)
and in the top of that, as you want it to be a function
def delete_tables(tableName, field, value):
c.execute("delete from " + tableName+ "where " + field + " = " + value)
Edit 2:
aaron's comment is true, it is not secure, the next step you would do is
def delete_tables(tab_name, attr, value):
#no ':value' (it limits the value to characters)
c.execute("delete from :tab_name where :attr = :value",
{'tab_name': tab_name, 'attr': attr, 'value':value})
It is from Vatsal's answer
I'm trying to execute a raw sql query and safely pass an order by/asc/desc based on user input. This is the back end for a paginated datagrid. I cannot for the life of me figure out how to do this safely. Parameters get converted to strings so Oracle can't execute the query. I can't find any examples of this anywhere on the internet. What is the best way to safely accomplish this? (I am not using the ORM, must be raw sql).
My workaround is just setting ASC/DESC to a variable that I set. This works fine and is safe. However, how do I bind a column name to the ORDER BY? Is that even possible? I can just whitelist a bunch of columns and do something similar as I do with the ASC/DESC. I was just curious if there's a way to bind it. Thanks.
#default.route('/api/barcodes/<sort_by>/<sort_dir>', methods=['GET'])
#json_enc
def fetch_barcodes(sort_by, sort_dir):
#time.sleep(5)
# Can't use sort_dir as a parameter, so assign to variable to sanitize it
ord_dir = "DESC" if sort_dir.lower() == 'desc' else 'ASC'
records = []
stmt = text("SELECT bb_request_id,bb_barcode,bs_status, "
"TO_CHAR(bb_rec_cre_date, 'MM/DD/YYYY') AS bb_rec_cre_date "
"FROM bars_barcodes,bars_status "
"WHERE bs_status_id = bb_status_id "
"ORDER BY :ord_by :ord_dir ")
stmt = stmt.bindparams(ord_by=sort_by,ord_dir=ord_dir)
rs = db.session.execute(stmt)
records = [dict(zip(rs.keys(), row)) for row in rs]
DatabaseError: (cx_Oracle.DatabaseError) ORA-01036: illegal variable name/number
[SQL: "SELECT bb_request_id,bb_barcode,bs_status, TO_CHAR(bb_rec_cre_date, 'MM/DD/YYYY') AS bb_rec_cre_date FROM bars_barcodes,bars_status WHERE bs_status_id = bb_status_id ORDER BY :ord_by :ord_dir "] [parameters: {'ord_by': u'bb_rec_cre_date', 'ord_dir': 'ASC'}]
UPDATE Solution based on accepted answer:
def fetch_barcodes(sort_by, sort_dir, page, rows_per_page):
ord_dir_func = desc if sort_dir.lower() == 'desc' else asc
query_limit = int(rows_per_page)
query_offset = (int(page) - 1) * query_limit
stmt = select([column('bb_request_id'),
column('bb_barcode'),
column('bs_status'),
func.to_char(column('bb_rec_cre_date'), 'MM/DD/YYYY').label('bb_rec_cre_date')]).\
select_from(table('bars_barcode')).\
select_from(table('bars_status')).\
where(column('bs_status_id') == column('bb_status_id')).\
order_by(ord_dir_func(column(sort_by))).\
limit(query_limit).offset(query_offset)
result = db.session.execute(stmt)
records = [dict(row) for row in result]
response = json_return()
response.addRecords(records)
#response.setTotal(len(records))
response.setTotal(1001)
response.setSuccess(True)
response.addMessage("Records retrieved successfully. Limit: " + str(query_limit) + ", Offset: " + str(query_offset) + " SQL: " + str(stmt))
return response
You could use Core constructs such as table() and column() for this instead of raw SQL strings. That'd make your life easier in this regard:
from sqlalchemy import select, table, column, asc, desc
ord_dir = desc if sort_dir.lower() == 'desc' else asc
stmt = select([column('bb_request_id'),
column('bb_barcode'),
column('bs_status'),
func.to_char(column('bb_rec_cre_date'),
'MM/DD/YYYY').label('bb_rec_cre_date')]).\
select_from(table('bars_barcodes')).\
select_from(table('bars_status')).\
where(column('bs_status_id') == column('bb_status_id')).\
order_by(ord_dir(column(sort_by)))
table() and column() represent the syntactic part of a full blown Table object with Columns and can be used in this fashion for escaping purposes:
The text handled by column() is assumed to be handled like the name of a database column; if the string contains mixed case, special characters, or matches a known reserved word on the target backend, the column expression will render using the quoting behavior determined by the backend.
Still, whitelisting might not be a bad idea.
Note that you don't need to manually zip() the row proxies in order to produce dictionaries. They act as mappings as is, and if you need dict() for serialization reasons or such, just do dict(row).
I'm creating SQL commands from given input:
def do_add(self, table, full_sql_command="INSERT INTO {} VALUES ({})"):
""" Add a column to a specified table name """
add_command = raw_input("INSERT INTO {} VALUES ".format(table))
self.__create_sql_command(full_sql_command, table, add_command.split(" "))
def __create_sql_command(self, full, tablename, addable):
print full.format(tablename, ', '.join(addable))
What I need this to output is INSERT INTO <table-name> VALUES ('<val-1>', '<val-2>', '<etc..>')
As of right now I get the following output:
INSERT INTO inventory VALUES test test test
# <= INSERT INTO inventory VALUES (test, test, test)
How can I get the quotes around the values to add?
Instead of wrapping your arguments in quotes, use a parametrized query. It will in effect add the quotes as needed, transparently and with no chance of problems due to e.g. quotes in the input itself, either intentional (SQL injection attacks) or accidental.
This is an example of a parametrized query.
cursor.execute("INSERT INTO table VALUES (?, ?, ?)", (var1, var2, var3))
As you can see you need to pass two separate arguments to execute(), the query template and the values, so it will require some rejigging of your code.
Note: Different database modules accept different placeholders for the parametrized query. sqlite3, for example, accepts ? as above, but also allows named parameters (passed in a dictionary instead of a tuple):
cursor.execute("INSERT INTO table VALUES (:who, :age)", {"who":"Bill", "age": 33})
I am no SQL guy but if your enter table data separated by a space then a little for loop will do your job,
Say:
l=raw_input('Enter Table contents separated by space.').split(' ')
l1='('
for i in l:l1=l1+i+', '
l1=l1[:len(l1-3)] #for removing extra comma and space left after loop
l1=l1+')'
Then use l1 where ever you want. You can use private attributes or something, for more protection!
If anyone is curious, this is what I ended up doing:
Never do this in production it is extremely vulnerable to SQL injection
def __create_sql_command(self, full, tablename, addable):
return full.format(tablename, ', '.join(map(lambda x: "'" + x + "'", addable)))
#<= CREATE TABLE test ('test1', 'test2', 'test3')
I hope this has not been asked previously, I was not sure what keywords to use.
Suppose I want to write a function that can take a less than or equal to statement for a query...
import MySQLdb
def query1(date,le):
'''
query1('2013-01',<= )
>>> 10
'''
query = '''
select *
from table
where number {x} 1
and date = {dt}
'''.format(dt=date,x=le)
cursor.execute(query)
rslt = cursor.fetchall()
return rslt
Then what is the best way to do this?
You can just pass the comparison operator as a string to your function:
query1('2013-01', '<=')
This will insert the string for the operator in to the query, resulting in
select *
from table
where number <= 1
and date = 2013-01
Please note that directly building SQL queries by inserting strings is a potential vector for SQL injections. If you allow users to supply their own date strings, the user could inject some SQL code in and run malicious code. Look in to query parameterisation for more information.
If you wanted to guard against SQL injection, you should do something like the following. The allowed operators list is carefully whitelisted, so only valid and safe operators can be used. This is used to build the query. The date is then injected in to the query by the cursor.execute() command. MySQLdb then handles constructing a safe query from your data, and will not allow a malicious user to inject their own SQL in place of the date string.
import MySQLdb
def query1(date, comp):
query = '''
select *
from table
where number {comp} 1
and date = %s
'''.format(comp=sql_comp_operator(comp))
cursor.execute(query, (date, ))
return cursor.fetchall()
def sql_comp_operator(comp):
operators = {
'lt': '<',
'lte': '<',
'gt': '>',
'gte': '>=',
}
if comp in operators:
return operators[comp]
else:
raise ValueError("Unknown comparison operator '{}'".format(comp))
query1('2013-01', 'lte')
Ideally, you'd want to use an ORM to prevent SQL injection attacks (I prefer SQLAlchemy/Elixir) which would let you do stuff like:
q = session.query(User).\
filter(User.id <= 1).\
filter(User.date_of_birth == date)
Sounds like you want "le" to be a function/lambda that you can pass in, but I don't know of any way to convert that lambda to a string for putting in your query. For example, you could call it like:
query1('2013-01-01', lambda x,y: x <= y)
But there's no real way I know of to convert that in your query to "<=". If you pass it in as a string, however, you can used format with named blocks, by passing in a dictionary with keys with the same names as those blocks, like this:
sql = """
select *
from table
where number {operation} 1
and date = '{date}'
"""
data = {
"operation": "<=",
"date": "2013-01-01"
}
query = sql.format(**data)