We Keep Coding

Python adds single quote to my query parameters

I'm trying to execute thi query:
SELECT '23.34.67.0/22' CONCAT(DAY_31, 'hello') DAY_31 FROM Jule
using pymysql. My code is:
cursor.execute("SELECT %s CONCAT(%s, %s) %s FROM Jule", (p, 'DAY_' + _day, as_tmp, 'DAY_' + _day))
But python adds single quote and return a syntax error
"You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '('DAY_31', 'hello') 'DAY_31' FROM Jule' at line 1"
DAY_31 is a column of Jule Schema

If memory serves, ? in place of %s might do the trick.

The reason that you end up with quotes is because cursor.execute adds these around values you pass in as arguments. This is totally appropriate for your second argument, because if the value 'hello' was inserted into the query as-is you would end up with a query like this:
SELECT '23.34.67.0/22' CONCAT(DAY_31, hello) DAY_31 FROM Jule
and you would have errors telling you that MySQL can't identify what hello should refer to.
Obviously though, this is not appropriate for situations where you want to pass in field names, or any other part of the query that is not a primitive string value. In these cases, you will need to splice them into your string before executing the query. One way you could do that is with f-strings, but there are other alternatives as well. Here's your cursor.execute line with the field names spliced in using f-strings:
cursor.execute(f"SELECT %s, CONCAT({'DAY_'+_day}, %s) {'DAY_'+_day} FROM Jule", (p, as_tmp))
Notice that I've removed the 'DAY_'+_day from the arguments list as well.
Important note:
While this should work like this (although I think you also needed an extra comma after SELECT '23.34.67.0/22', which I've added in the example above), it's very important that if day has a value that originates from outside of your application (e.g. passed in by a user in a form field) that you make sure it is exactly in the format you want before splicing it in to your query. Checking that the string value is an integer could be one way to do that. The reason that it is important is that without this, your application could be prone to SQL injection, which would potentially allow users to run arbitrary SQL on your database. If the value of day is calculated solely by your application, you shouldn't need to worry about this.

Actually what is happening has nothing to do with cursor.execute. It is just the way you are formatting the string in python.
We should first note that %s %d can be specific to SQL, and that these can also be used in python for string formatting.
So taking this into account, I think you have no need to use %s for string formatting (which on top does not help with readability). And since you are cursor.executeing straight to your database, you could format your string straight away in python.
Small check to see whats happening:
p = "23.34.67.0/22"
as_tmp = "hello"
_day = "3"
print("SELECT %s CONCAT(%s, %s) %s FROM Jule", (p, 'DAY_' + _day, as_tmp, 'DAY_' + _day))
# output
# SELECT %s CONCAT(%s, %s) %s FROM Jule ('23.34.67.0/22', 'DAY_3', 'hello', 'DAY_3')
# ^^^^ so this is what is being sent in cursor.execute (with all the quotes and so on)
If you format with an f-string you will increase readability, and you should get rid of your problem with the quotes
print(f"SELECT '{p}' CONCAT(DAY_{_day}, '{as_tmp}') DAY_{_day} FROM Jule")
# output
# SELECT '23.34.67.0/22' CONCAT(DAY_3, 'hello') DAY_3 FROM Jule
So the solution could be:
cursor.execute(f"SELECT '{p}' CONCAT(DAY_{_day}, '{as_tmp}') DAY_{_day} FROM Jule")

MySql read_sql python query with variable #

I am aware that queries in Python can be parameterized using either ? or %s in execute query here or here
However I have some long query that would use some constant variable defined at the beginning of the query
Set #my_const = 'xyz';
select #my_const;
-- Query that use #my_const 40 times
select ... coalesce(field1, #my_const), case(.. then #my_const)...
I would like to do the least modif possible to the query from Mysql. So that instead of modifying the query to
pd.read_sql(select ... coalesce(field1, %s), case(.. then %s)... , [my_const, my_const, my_const, ..]
,I could write something along the line of the initial query. Upon trying the following, however, I am getting a TypeError: 'NoneType' object is not iterable
query_str = "Set #null_val = \'\'; "\
" select #null_val"
erpur_df = pd.read_sql(query_str, con = db)
Any idea how to use the original variable defined in Mysql query ?

The reason
query_str = "Set #null_val = \'\'; "\
" select #null_val"
erpur_df = pd.read_sql(query_str, con = db)
throws that exception is because all you are doing is setting null_value to '' and then selecting that '' - what exactly would you have expected that to give you? EDIT read_sql only seems to execute one query at a time, and as the first query returns no rows it results in that exception.
If you split them in to two calls to read_sql then it will in fact return you the value of your #null value in the second call. Due to this behaviour read_sql is clearly not a good way to do this. I strongly suggest you use one of my suggestions below.
Why are you wanting to set the variable in the SQL using '#' anyway?
You could try using the .format style of string formatting.
Like so:
query_str = "select ... coalesce(field1, {c}), case(.. then {c})...".format(c=my_const)
pd.read_sql(query_str)
Just remember that if you do it this way and your my_const is a user input then you will need to sanitize it manually to prevent SQL injection.
Another possibility is using a dict of params like so:
query_str = "select ... coalesce(field1, %(my_const)s, case(.. then %(my_const)s)..."
pd.read_sql(query_str, params={'my_const': const_value})
However this is dependent on which database driver you use.
From the pandas.read_sql docs:
Check your database driver documentation for which of the five syntax
styles, described in PEP 249’s paramstyle, is supported. Eg. for
psycopg2, uses %(name)s so use params={‘name’ : ‘value’}

Escaping single and double quotes for mysql in python

I am using mysql.connector python library with python 2.7
I have a unicode string which may or may not contain single and double quotes.
Here are the things I tried for my escape function:
def escape(string):
#string.MySQL.escape_string()
#string = string.decode('string_escape')
#string = string.encode('unicode-escape').replace("'", "''")
#string = string.encode('unicode-escape').replace('"', '\"')
#string = string.encode('unicode-escape').replace("'", u"\u2019")
#string = string.encode('unicode-escape').replace('''"''', u"\u201D")
#string = string.encode('unicode-escape').replace('''''', u"\u201D")
return string
Nothing seems to have worked. I tried using this function but still gives mysql syntax error.
I need something within mysql.connector library which escapes the single and double quotes without breaking the unicode as well as mysql query.
Here is an example of SQL query I am using:
"""SELECT * FROM messages WHERE msg_id = '{msg_id}'""".format(**db_dict)
Let me know if any more details needed
EDIT: Example SQL query updated

MySQLdb officially declares to use the format paramstyle, but it also supports the pyformat style*, so if you want to use parameters from a dict, you can use:
db_dict = {'msg_id': "1'2'3", ...}
cursor.execute("SELECT * FROM messages WHERE msg_id = %(msg_id)s", db_dict)
Using string manipulation to create sql queries only leads to sql injection vulnerabilities, so you should never do it.
*... most db connectors that use python string formatting behind the screen do the same, they specify one of format or pyformat as paramstyle but actually support both. The dbapi2 doesnt't allow to specify two values here, but it doesn't forbid to support multiple parmstyles either. If you write code that potentially uses an unknowon dbapi2 connector it's enough that you can query a supported paramstyle, being able to know all would be nice but it's not necessary.

cursor.execute('SELECT * FROM messages WHERE msg_id = %s', (db_dict['msg_id'],)) is what you want to run here. Standard string escapes aren't supported by python's database interface, and, per #bobince's comment, are a security hole to boot.

TypeError: not all arguments converted during string formatting string python

You probably have seen it before, but I'm basically trying to avoid MySQL Injection, so I'm formatting my query as follows using Python:
if "username" in form:
username = form["username"].value
else:
success = 0
error = "User Name is Missing"
cur.execute("SELECT COUNT(*) FROM users WHERE screenName=':1'",[username])
results = int(cur.fetchall()[0][0])
This throws an error saying:
<type 'exceptions.TypeError'>: not all arguments converted during string formatting
args = ('not all arguments converted during string formatting',)
message = 'not all arguments converted during string formatting'
Any idea what's wrong?
Thanks

You don't specify the exact library you're using, but assuming it's Python DB API compliant, you probably want to change the execute line to this:
cur.execute("SELECT COUNT(*) FROM users WHERE screenName=%s",[username])
Edit in response to comment from OP:
The use of %s is the current standard in terms of preventing SQL Injection. I'm not sure what the post in the answer you linked is getting at there... A couple things to keep in mind about that are that the thread has been closed as "not constructive", and that answer is also ~7 years old, so it's quite possible that there were issues back then that that answer may have been referencing that no longer apply.
But %s means that the library (in the execute method) handles all the escaping and quoting, and is the way to prevent injection. (As opposed to, say, using regular interpolation, say, via format, which would leave it exposed to injection.)
Note that is very specifically not '%s' and then using foo % bar in the execute call, but an unquoted %s and passing the parameters as the 2nd argument to execute.
For example, I use psycopg2, which is fully DB API compliant, and its current doc describes using %s to prevent injection.

Python + Sqlite 3. How to construct queries?

I'm trying to create a python script that constructs valid sqlite queries. I want to avoid SQL Injection, so I cannot use '%s'. I've found how to execute queries, cursor.execute('sql ?', (param)), but I want how to get the parsed sql param. It's not a problem if I have to execute the query first in order to obtain the last query executed.

If you're trying to transmit changes to the database to another computer, why do they have to be expressed as SQL strings? Why not pickle the query string and the parameters as a tuple, and have the other machine also use SQLite parameterization to query its database?

If you're not after just parameter substitution, but full construction of the SQL, you have to do that using string operations on your end. The ? replacement always just stands for a value. Internally, the SQL string is compiled to SQLite's own bytecode (you can find out what it generates with EXPLAIN thesql) and ? replacements are done by just storing the value at the correct place in the value stack; varying the query structurally would require different bytecode, so just replacing a value wouldn't be enough.
Yes, this does mean you have to be ultra-careful. If you don't want to allow updates, try opening the DB connection in read-only mode.

Use the DB-API’s parameter substitution. Put ? as a placeholder wherever you want to use a value, and then provide a tuple of values as the second argument to the cursor’s execute() method.
# Never do this -- insecure!
symbol = 'hello'
c.execute("SELECT * FROM stocks WHERE symbol = '%s'" % symbol)
# Do this instead
t = (symbol,)
c.execute('SELECT * FROM stocks WHERE symbol=?', t)
print c.fetchone()
More reference is in the manual.

I want how to get the parsed 'sql param'.
It's all open source so you have full access to the code doing the parsing / sanitization. Why not just reading this code and find out how it works and if there's some (possibly undocumented) implementation that you can reuse ?