We Keep Coding

Read a txt/sql file as formatted string - f'{}' in Python

I have sql file which has table names as formatted string
query.sql
SELECT * FROM {table_name} WHERE LOAD_DT = '{load_date}'
How to read the sql file as f-string to pass it to pd.read_sql() method?
table_name = PRODUCTS
load_date = '15-08-2020'
# n other local variables
with open('query.sql','r') as file:
sql_str = file.read()
Note: I do not prefer .format(table_name,load_date) or .format(**locals()) as I have a custom function to read various sql files and don't want to send a param list of format variables every time, the reason being if the format list is huge it will be laborious while preparing the sql file with positional arguments and chances of mistakes are high

You can use .format method of string:
sql_str = "SELECT * FROM {table_name} WHERE LOAD_DT = '{load_date}'"
sql_str.format(table_name="PRODUCTS", load_date="15-08-2020")
You can also pass all local variables into the .format method:
table_name = "PRODUCTS"
load_date = "15-08-2020"
sql_str.format(**locals())
It is also possible to achieve desired result using eval, which is quite dangerous:
table_name = "PRODUCTS"
load_date = "15-08-2020"
sql_str = "SELECT * FROM {table_name} WHERE LOAD_DT = '{load_date}'"
sql_str_f = f"f\"{sql_str}\""
result = eval(sql_str_f)

Pass a python variable into an SQL query

I am working on Databricks and I am trying to pass a python variable into a SQL query:
series_name "LogTest"
query = """SELECT * FROM results_table
WHERE name = $series_name
"""
spark.sql(query).toPandas()
I tried with $ but it does not work.
How do I do this?
Regards

In this case, your variable and queries are just strings. You can do:
query = f"""SELECT * FROM results_table
WHERE name = '{series_name}'
"""
... in python 3. Remember that your query string needs the single quotes around the inserted variable. However, for certain variables, you may need to pass the variable directly to the spark module so it can interpret it. I use pyodbc a lot and would do this:
query = f"""SELECT * FROM results_table
WHERE name = ?"""
cursor.execute(query, series_name)

Following will also work,
series_name = "LogTest"
spark.sql("SELECT * FROM results_table WHERE name = " + series_name).toPandas()

we can also try the following.
series_name = "LogTest"
query = "SELECT * FROM results_table WHERE name = {}".format(series_name)
spark.sql(query).toPandas() #

compose mysql query in python

I want to fetch all rows from MySQL table with
query = "SELECT * FROM %s WHERE last_name=%s"
cursor.execute(query, ("employees","Smith"))
but I'm getting
You have an error in your SQL syntax. When I try
query = "SELECT * FROM employees WHERE last_name=%s"
cursor.execute(query, ("Smith",))
all is fine.
Documentation says
cursor.execute(operation, params=None, multi=False)
The parameters found in the tuple or dictionary params are bound to the variables in the operation.link on docs

The first will generate an SQL like this:
SELECT * FROM 'employees' WHERE last_name='smith'
The parameters are SQL quoted.
If you really need to have a table name as param, you must proceed in 2 steps:
table_name = 'employees'
query_tpl = "SELECT * FROM {} WHERE last_name=%s"
query = query_tpl.format(table_name)
cursor.execute(query, ("Smith",))

you need to add the quote symbol. So the query will be like
SELECT * FROM employees WHERE last_name='Smith'
Change both your query to
query = "SELECT * FROM %s WHERE last_name='%s'"
query = "SELECT * FROM employees WHERE last_name='%s'"

You can't use a parameter for the table name in the execute call.
But you can use Python string interpolation for that:
query = "SELECT * FROM %s WHERE last_name=%s" %("employees","Smith")
cursor.execute(query)

You can't use a table name as a parameter. you are generating invalid sql with your code that is putting quotes around each string. the table name cannot have quotes around it.
sql you are generating
select * from 'employees' where last_name = 'Smith'
What sql you want
select * from employees where last_name = 'Smith'
you would have to format the string first like the example below.
query = "SELECT * from {} wherre last_name ='{}'"
cursor.execute(query.format("employees","Smith"))
using code like this does open up the possibility of SQL injection. so please bear that in mind.

query="SELECT * FROM %s WHERE name=%s",(employees,smith)
cursor.execute(query)
rows = cursor.fetchall()
Try this one. Hopefully it works for you.

Python/pg8000 WHERE IN statement

What is the correct method to have the tuple (names) be available via %s in the SQL statement?
names = ('David', 'Isaac')
sql = 'SELECT * from names WHERE name IN %s'
cur.execute(sql,(names,))
The response in https://stackoverflow.com/a/28117658/5879128 works for psycopg2 but does not work in pg8000.
Thanks!

Generate the proper number of placeholders. In pg8000 the placeholder defaults to %s.
Interpolate placeholders to the query string.
Execute SQL-injection-safe query.
Like so:
sql = 'SELECT * from names WHERE name IN ({})'.format( ','.join(['%s']*len(names)) )
# results in -> 'SELECT * from names WHERE name IN (%s,%s)'
cur.execute(sql,(names,))

Using placeholders in a database query doesn't work

I am querying a mysql database version 5.6.13 using python 2.7.
This works:
whichCustomer = str(1934)
qry = ("SELECT * FROM customers WHERE customerid = " + whichCustomer)
cursor.execute(qry)
The query also works:
qry = ("SELECT * FROM customers WHERE customerid = 1934")
cursor.execute(qry)
BUT, when I try to use string substitution the query fails:
whichCustomer = 1934
qry = ("SELECT * FROM customers WHERE customerid = %d")
cursor.execute(qry, (whichCustomer))
Is there something I am missing. The full try/execute code follows:
try:
import mysql.connector
print 'Module mysql initialized'
print 'Attempting connection to cheer database'
cnx = mysql.connector.connect(user='notsure',
password='notsure',
host='localhost',
database='notreal')
cursor = cnx.cursor()
whichCustomer = str(1934)
qry = ("SELECT * FROM customers WHERE customerid = " + whichCustomer)
cursor.execute(qry)
recx = cursor.fetchone()
print recx[1]
cnx.close()
print 'Successful connection to notreal database'
except:
print 'Error initialzing mysql databsasr'

You need to use %s for SQL parameters, and the second argument must be a sequence, like a tuple:
whichCustomer = 1934
qry = ("SELECT * FROM customers WHERE customerid = %s")
cursor.execute(qry, (whichCustomer,))
Note the comma in the second parameter; without a comma, that parameter is not a tuple and just the 1934 integer value is passed in instead.
Although both Python string interpolation placeholders and SQL parameters use closely related syntax, they are not the same thing. As such, SQL parameters for positional values are always expressed as %s regardless of the type.