I'm using PyMYSQL to query data from a MySQL database. An example query that I want to use is:
SELECT count(*) AS count
FROM example
WHERE item = %s
LIMTI 1
which would be run using
query = "SELECT COUNT(*) as count FROM example WHERE item = %s LIMIT 1"
cur.execute(query,(None))
or
cur.execute(query,(10))
The issue is that item can have a value of NULL which results in
WHERE item = NULL
Which doesn't work in MySQL. Rather it should read
WHERE item IS NULL
However the value can be integer also. How can I make PyMySQL adjust the query to handle both situations?
you can make a simple function to catch the None situation:
def null_for_sql(a):
return "is NULL" if not a else "= " + str(a)
and then call the query like that:
query = "SELECT COUNT(*) as count FROM example WHERE item %s LIMIT 1"
cur.execute(query,null_for_sql(None))
Note there is no "=" after "item" in the query
Edit after comments:
def sql_none_safe_generator(query,parameter):
if parameter:
return query.replace(%param, "= " + str(parameter))
else:
return query.replace(%param, is NULL)
query = "SELECT COUNT(*) as count FROM example WHERE item %param LIMIT 1"
cur.execute(sql_none_safe_generator(query,a))
Related
Im using python3 and postgres 11.5.
This is the script :
a = cursor.execute("SELECT tablename FROM pg_catalog.pg_tables limit 5")
for table in a:
cursor.execute("SELECT * FROM pg_prewarm(public.%s)", [table[0]])
a query gets some table names , and the loop query should run table name as the %s.
but for some reason i get the arg table[0] with // /n in the query and its messing it up.
if i print a results i get table names as tuple:
[('sa1591354519',), ('sa1591397719',), ('sa1591397719',)]
so [table[0]] is a string.
the error i get:
1574683839 [16177], ERR, execute ({'Error while connecting to PostgreSQL': SyntaxError('syntax error at or near "\'sa1591440919\'"\nLINE 1: SELECT * FROM pg_prewarm(public.\'sa1591440919\')\n ^\n')},)
what can i do ?
The errors don't have anything to do with the newlines you see, which are just an artifact of the error message. If you were to print out the error, would see:
syntax error at or near "'sa1591440919'"
LINE 1: SELECT * FROM pg_prewarm(public.'sa1591440919')
^
In other words, Postgres doesn't like the table name you're passing because it contains quotes. This is happening because you're trying to treat the table names like a normal query parameter, which causes psycopg to quote them...but that's not what you want in this case.
Just replace your use of query templating with normal Python string substitution:
a = cursor.execute("SELECT tablename FROM pg_catalog.pg_tables limit 5")
for table in a:
cursor.execute("SELECT * FROM pg_prewarm(public.%s)" % (table[0]))
But this won't actually work, because cursor.execute doesn't return a value, so a will be None. You would need to do something like:
cursor.execute("SELECT tablename FROM pg_catalog.pg_tables limit 5")
a = cursor.fetchall()
for table in a:
...
I am querying data from bigquery using get_data_from_bq method mentioned below in a loop:
def get_data_from_bq(product_ids):
format_strings = ','.join([("\"" + str(_id) + "\"") for _id in product_ids])
query = "select productId, eventType, count(*) as count from [xyz:xyz.abc] where productId in (" + format_strings + ") and eventTime > CAST(\"" + time_thresh +"\" as DATETIME) group by eventType, productId order by productId;"
query_job = bigquery_client.query(query, job_config=job_config)
return query_job.result()
While for the first query(iteration) data returned is correct, all the subsequent queries are throwing the below-mentioned exception
results = query_job.result()
File "/home/ishank/.local/lib/python2.7/site-packages/google/cloud/bigquery/job.py", line 2415, in result
super(QueryJob, self).result(timeout=timeout)
File "/home/ishank/.local/lib/python2.7/site-packages/google/cloud/bigquery/job.py", line 660, in result
return super(_AsyncJob, self).result(timeout=timeout)
File "/home/ishank/.local/lib/python2.7/site-packages/google/api_core/future/polling.py", line 120, in result
raise self._exception
google.api_core.exceptions.BadRequest: 400 Cannot explicitly modify anonymous table xyz:_bf4dfedaed165b3ee62d8a9efa.anon1db6c519_b4ff_dbc67c17659f
Edit 1:
Below is a sample query which is throwing the above exception. Also, this is running smoothly in bigquery console.
select productId, eventType, count(*) as count from [xyz:xyz.abc] where productId in ("168561","175936","161684","161681","161686") and eventTime > CAST("2018-05-30 11:21:19" as DATETIME) group by eventType, productId order by productId;
I had the exact same issue. The problem is not the query itself, it's that you are most likely reusing the same QueryJobConfig. When you perform a query, unless you set a destination, BigQuery stores the result in an anonymous table which is stated in the QueryJobConfig object. If you reuse this configuration, BigQuery tries to store the new result in the same anonymous table, hence the error.
I don't particularly like this behaviour, to be honest.
You should rewrite your code like that:
def get_data_from_bq(product_ids):
format_strings = ','.join([("\"" + str(_id) + "\"") for _id in product_ids])
query = "select productId, eventType, count(*) as count from [xyz:xyz.abc] where productId in (" + format_strings + ") and eventTime > CAST(\"" + time_thresh +"\" as DATETIME) group by eventType, productId order by productId;"
query_job = bigquery_client.query(query, job_config=QueryJobConfig())
return query_job.result()
Hope this helps!
Edited:
Federico Bertola is correct on the solution and the temporary table that is written to by BigQuery see this link.
I did not get an error with my sample code querying from a public table last time, but I can reproduce the error today, so it is possible this symptom can appear intermittent. I can confirm the error is resolved with Federico’s suggestion.
You can get the “super(QueryJob, self).result(timeout=timeout)” error when the query string lacks quotes around the parameters in the query. It seems you have made a similar mistake with the parameter format_strings in your query. You can fix this problem by ensuring there is quotes escaped around the parameter:
(" + myparam + ")
, should be written as
(\"" + myparam + "\")
You should examine your query string where you use parameters, and start with a simpler query such as
select productId, eventType, count(*) as count from `xyz:xyz.abc`
, and grow your query as you go.
For the record, here is what worked for me:
from google.cloud import bigquery
client = bigquery.Client()
job_config = bigquery.QueryJobConfig()
def get_data_from_bq(myparam):
query = "SELECT word, SUM(word_count) as count FROM `publicdata.samples.shakespeare` WHERE word IN (\""+myparam+"\") GROUP BY word;"
query_job = client.query(query, job_config=job_config)
return query_job.result()
mypar = "raisin"
x = 1
while (x<9):
iterator = get_data_from_bq(mypar)
print "==%d iteration==" % x
x += 1
I am working in Django 1.8. I am currently running the following unsafe query using django.db.connection:
cursor = connection.cursor()
param = 'cost' # Actually obtained as a GET parameter
query = "SELECT date, pct_id, SUM(%s) as val " % param
query += "FROM mytable "
query += "GROUP BY pct_id, date ORDER BY date"
cursor.execute(query)
This works, but is obviously vulnerable to SQL injection. Instead I want to escape param before passing it into the query string.
So I'm trying this:
query = "SELECT date, pct_id, SUM(%s) as val "
query += "FROM mytable "
query += "GROUP BY pct_id, date ORDER BY date"
cursor.execute(query, param)
However, this gives me the following error:
TypeError: not all arguments converted during string formatting
How can I safely escape this parameter?
I have this Python code:
def get_employees(conditions, fields):
cursor.execute("SELECT employeeID FROM employees WHERE name=%s, budget=%s,
%year=%s,(some of conditions))
Is there any way to get employeeIDs if I set in conditions not all parameters, etc. only name and year?
If conditions were a dictionary, you could construct a query string:
def get_employees(conditions):
query = 'select employeeid from employees'
if conditions:
query += ' where ' + ' and '.join(key + ' = %s' for key in conditions.keys())
cursor.execute(query, conditions.values())
(I should note that here I am assuming that conditions does not have user-supplied keys. If there are user-supplied keys, this is definitely vulnerable to SQL injection.)
usually it is done via dynamic sql building, like this:
sql = "SELECT employeeID FROM employees WHERE 1=1";
if condition has name
sql += " and name='" .escape(name) . "'"
if condition has somefield
sql += " and somefield='" .escape(somefield) . "'"
etc
execute final sql
I am running Django + MySql on OpenShift server and have this problem.
Code:
from django.db import connection
...
cursor = connection.cursor()
somedate = calculateSomeDate()
query = "SELECT id FROM levelbuffer WHERE START > '%s' ORDER BY START ASC LIMIT 1" % somedate
print "First select: " + query
cursor.execute(query)
sql_result = cursor.fetchone()
if not sql_result == None:
gameId = int(sql_result[0])
cursor.fetchall() #To clear anything left behind (this wasn't here before, but no changes appeared after adding)
query = "SELECT gamescores.skore, users.name FROM gamescores JOIN users ON gamescores.userId = users.id where gamescores.gameId = " + str(gameId) + " ORDER BY skore ASC"
cursor.execute(query);
#cursor.execute("SELECT gamescores.skore, users.name FROM gamescores JOIN users ON gamescores.userId = users.id where gamescores.gameId = %s ORDER BY skore ASC", (gameId))
print "Second select " + query
row = cursor.fetchone()
The weird thing is, that the row is sometimes None and sometimes it contains wanted values. I cant figure out, what i am missing. The first query works fine every time.
When I try to perform the second query printed in log on phpmyadmin - works always fine.