I'm trying to create a python script that constructs valid sqlite queries. I want to avoid SQL Injection, so I cannot use '%s'. I've found how to execute queries, cursor.execute('sql ?', (param)), but I want how to get the parsed sql param. It's not a problem if I have to execute the query first in order to obtain the last query executed.
If you're trying to transmit changes to the database to another computer, why do they have to be expressed as SQL strings? Why not pickle the query string and the parameters as a tuple, and have the other machine also use SQLite parameterization to query its database?
If you're not after just parameter substitution, but full construction of the SQL, you have to do that using string operations on your end. The ? replacement always just stands for a value. Internally, the SQL string is compiled to SQLite's own bytecode (you can find out what it generates with EXPLAIN thesql) and ? replacements are done by just storing the value at the correct place in the value stack; varying the query structurally would require different bytecode, so just replacing a value wouldn't be enough.
Yes, this does mean you have to be ultra-careful. If you don't want to allow updates, try opening the DB connection in read-only mode.
Use the DB-API’s parameter substitution. Put ? as a placeholder wherever you want to use a value, and then provide a tuple of values as the second argument to the cursor’s execute() method.
# Never do this -- insecure!
symbol = 'hello'
c.execute("SELECT * FROM stocks WHERE symbol = '%s'" % symbol)
# Do this instead
t = (symbol,)
c.execute('SELECT * FROM stocks WHERE symbol=?', t)
print c.fetchone()
More reference is in the manual.
I want how to get the parsed 'sql param'.
It's all open source so you have full access to the code doing the parsing / sanitization. Why not just reading this code and find out how it works and if there's some (possibly undocumented) implementation that you can reuse ?
Related
I have a function that executes many SQL queries with different dates.
What I want is to pass all dates and other query variables as function parameters and then just execute the function. I have figured out how to do this for datetime variables as below. But I also have a query that looks at specific campaign_names in a database and pulls those as strings. I want to be able to pass those strings as function parameters but I haven't figured out the correct syntax for this in the SQL query.
def Camp_eval(start_date,end_1M,camp1,camp2,camp3):
query1 = f"""SELECT CONTACT_NUMBER, OUTCOME_DATE
FROM DATABASE1
where OUTCOME_DATE >= (to_date('{start_date}', 'dd/mm/yyyy'))
and OUTCOME_DATE < (to_date('{end_1M}', 'dd/mm/yyyy'))"""
query2 = """SELECT CONTACT_NUMBER
FROM DATABASE2
WHERE (CAMP_NAME = {camp1} or
CAMP_NAME = {camp2} or
CAMP_NAME = {camp3})"""
Camp_eval('01/04/2022','01/05/2022','Camp_2022_04','Camp_2022_05','Camp_2022_06')
The parameters start_date and end_1M work fine with the {} brackets but the camp variables, which are strings don't return any results even though there are results in the database with those conditions if I were to write them directly in the query.
Any help would be appreciated!!
Please, do not use f-strings for creating SQL queries!
Most likely, any library you use for accessing a database already has a way of creating queries: SQLite docs (check code examples).
Another example: cur.execute("SELECT * FROM tasks WHERE priority = ?", (priority,)).
Not only this way is safer (fixes SQL Injection problem mentioned by #d-malan in comments), but it also eliminates the need to care about how data is represented in SQL - the library will automatically cast dates, strings, etc. in what they need to be casted into. Therefore, your problem can be fixed by using proper instruments.
I am new to PyMySQL and just tried to execute a query:
c.execute('''INSERT INTO mysql_test1 (
data,
duration,
audio,
comments
) VALUES (
?,
?,
?,
?
);
''', [
comments_var,
duration_var,
audio_var,
comments_var
]
);
However, it threw the following error:
TypeError: not all arguments converted during string formatting
I noticed that something must be wrong with my variables and read up on how to properly deal with them in PyMySQL, expecting methods for parameter substitution, but to my surprise I could not find anything. Instead, every thread I found used string operations (e.g. here, here, here and here (with a comment claiming that string operations would be standard with PyMySQL).
This is interesting to me because I have previously only dealt with SQLite where the DBAPI documentation explicitly warns to use string operations with variables:
SQL operations usually need to use values from Python variables. However, beware of using Python’s string operations to assemble queries, as they are vulnerable to SQL injection attacks.
The documentation exemplifies this with the following code snippet:
Never do this -- insecure!
symbol = 'RHAT'
cur.execute("SELECT * FROM stocks WHERE symbol = '%s'" % symbol)
Instead, use the DB-API’s parameter substitution.
When reading the PyMySQL docs, I could not find any mention of such dangers. It only confirmed my previous findings:
If args is a list or tuple, %s can be used as a placeholder in the query. If args is a dict, %(name)s can be used as a placeholder in the query.
Why is using string operations in sqlite3 considered vulnerable to SQL injection attacks and at the same time not questioned in pymysql?
It's a pity that the designers of pymysql chose to use %s as the parameter placeholder. It confuses many developers because that's the same as the %s used in string-formatting functions. But it's not doing the same thing in pymysql.
It's not just doing a simple string substitution. Pymysql will apply escaping to the values before interpolating them into the SQL query. This prevents special characters from changing the syntax of the SQL query.
In fact, you can get into trouble with pymysql too. The following is unsafe:
cur.execute("SELECT * FROM stocks WHERE symbol = '%s'" % symbol)
Because it interpolates the variable symbol into the string before passing it as an argument to execute(). The only argument is then a finished SQL string with the variable(s) formatted into it.
Whereas this is safe:
cur.execute("SELECT * FROM stocks WHERE symbol = %s", (symbol,))
Because it passes the list consisting of the symbol variable as a second argument. The code in the execute() function applies escaping to each element in the list, and interpolates the resulting value into the SQL query string. Note the %s is not delimited by single-quotes. The code of execute() takes care of that.
I have been working with an SQL database through the mySQL library in Python. I have recently found that when I try searching for a string in my database, it is not correctly returning the output I expect. I think this may be due to my variable not being properly inserted into my SQL command.
code = data['code']
sql = "SELECT 200 FROM mytable WHERE mycolumn = '%s';"
query.execute(sql, teamCode)
print(str(query.fetchall()))
My problem is that printing query.fetchall() prints an empty list ([]) instead of the expected [[200,]] which means the program thinks the code value it is using does not exist in the SQL database, which it does.
The parameters in an execute call need to be a sequence, like a tuple. Try:
query.excute(sql, (teamCode,))
That turns it into a one-element tuple. BTW, did you really mean "code" there?
I am trying to run a SQL script in Python where I am passing a variable in the where clause as below:
cursor.execute(f"""select * from table where type = variable_value""")
In the above query, variable_value has the value that I am trying to use in the where clause. I am however getting an error psycopg2.errors.UndefinedColumn: column "variable_value" does not exist in table
As per psycopg2 documentation the execute function takes variables as an extra parameter.
cursor.execute("""select * from table where type = %(value)s """, {"value": variable_value})
More examples in psycopg2 user manual..
Also please read carefully the section about SQL injection - the gist is, you should not quote parameters in your query, the execute function will take care of that to prevent the injection of harmful SQL.
Also to explain the error you are getting - the query you're sending is comparing two identifiers (type and variable_value). The table does not contain variable_value column, hence the error.
I believe, you intended to use string interpolation to construct the query, but you forgot the {}. It would work like this:
cursor.execute(f"""select * from table where type = '{variable_value}'""")
⚠️ but because of previously mentioned SQL injection, it is not a recommended way!.
can somebody please recomend me some python DBAL library that will best suit my requirements. I would like to write my sql statements directly, most of the logics will be in db stored procedures (postgresql), so I only need to invoke db procedures, pass arguments to them and fetch the results. The library should help me with quoting (preventing sql inject).
I played with sqlalchemy, but i think that there is no quoting helper when writing sql statement directly to engine.execute method.
Thank you
You should have given sqlalchemy a deeper look; It does a fine job of quoting placeholders:
>>> engine = sqlalchemy.create_engine("sqlite:///:memory:")
>>> engine.execute("select ?", 5).fetchall()
[(5,)]
>>> engine.execute("select ?", "; drop table users; --").fetchall()
[(u'; drop table users; --',)]
psycopg2 (via DB-API) will automatically quote to prevent SQL injection, IF you use it properly. (The python way is wrong; you have to pass the parameters as arguments to the query command itself.)
WRONG:
cur.execute('select * from table where last="%s" and first="%s"'
% (last, first))
RIGHT:
cur.execute('select * from table where last=%s and first=%s',
(last, first))
Note: you don't use %, and you don't put quotes around your values.
The syntax is slightly different for MySQLdb and sqlite3. (For example, sqlite uses ? instead of %s.)
Also, for psycopg2, always use %s even if you're dealing with numbers or some other type.