Related
Today i am using the sql_debug(True) that helps me to see the queries but without the values.
How could i see how ponyorm translate the query with values ?
Thank you very much.
This is an example of query i'm using.
with db_session:
access = select(p for p in Access if raw_sql('( lower(first_name) = lower($first_name) and lower(last_name) = lower($last_name) ) '
'or ( lower(first_name) = lower($last_name) and lower(last_name) = lower($first_name) ) '
'or (lower(facebook_url) = lower($facebook_url)) '
'or (lower(twitter_url) = lower($twitter_url)) '
'or (lower(linkedin_url) = lower($linkedin_url)) '))
.order_by(desc(Access.twitter_url),desc(Access.facebook_url),desc(Access.linkedin_url),
desc(Access.facebook_url))
print(access.get_sql())
I use
logging.getLogger(__name__).debug('SQL:\n\n\t\t\t%s\n', '\n'.join(unicode(x) for x in request._construct_sql_and_arguments()[:2]).replace('\n', '\n\t\t\t'))
for that.
For example,
19:30:01.902 data.py:231 [DEBUG] SQL:
SELECT "x"."_id", "x"."filename", "x"."_created", "x"."_updated"
FROM "reports" "x"
WHERE "x"."_id" <= ?
AND "x"."_created" >= ?
(50, '2019-04-17 19:30:01.900028')
will be printed out.
You can use set_sql_debug(debug=True, show_values=True).
Reference here.
There is a method called get_sql()
query_obj = select(c for c in Category if c.name.startswith('v'))
sql = query_obj.get_sql()
print(sql)
output:
SELECT "c"."id", "c"."name"
FROM "category" "c"
WHERE "c"."name" LIKE 'v%%'
code continue:
for obj in query_obj:
print('id:', obj.id, 'name:', obj.name)
output:
id: 1 name: viki
here is a link to the docs https://docs.ponyorm.com/api_reference.html#Query.get_sql
You can log the sql or simply print it.
Update:
OP updated the question:
If the sql query has a variable like $name it is passed as a sql parameter.
first_name = 'viki'
query = select(c for c in Category if raw_sql('( lower(name) = lower($first_name))'))
query.get_sql()
so get_sql() will return the value with a placeholder, and the output will look like this:
'SELECT "c"."id", "c"."name", "c"."age"\nFROM "Category" "c"\nWHERE ( lower(name) = lower(?))'
If we want no placeholders should be there in the query then we can avoid passing direct sql to query and instead build it separately in python.
Like this:
query = select(c for C in Category if c.name == 'viki')
query.get_sql()
output:
'SELECT "c"."id", "c"."name", "c"."age"\nFROM "Category" "c"\nWHERE "c"."name" = \'viki\''
I have a query similar to below:
def connection():
pcon = pg8000.connect(host='host', port=1234, user='user', password='password', database = 'database')
return pcon, pcon.cursor()
pcon, pcur = connection()
query = """ SELECT * FROM db WHERE (db.foo LIKE 'string-%' OR db.foo LIKE 'bar-%')"""
db = pd.read_sql_query(query, pcon)
However when I try to run the code I get:
DatabaseError: '%'' not supported in a quoted string within the query string
I have tried escaping the symbol with \ and an additional % with no luck. How can I get pg8000 to treat this as a wildcard properly?
"In Python, % usually refers to a variable that follows the string. If you want a literal percent sign, then you need to double it. %%"
-- Source
LIKE 'string-%%'
Otherwise, if that doesn't work, PostgreSQL also supports underscores for pattern matching.
'abc' LIKE 'abc' true
'abc' LIKE 'a%' true
'abc' LIKE '_b_' true
But, as mentioned in the comments,
An underscore (_) in pattern stands for (matches) any single character; a percent sign (%) matches any sequence of zero or more characters
According to the source code, though, it would appear the problem is the single quote following the % in your LIKE statement.
if next_c == "%":
in_param_escape = True
else:
raise InterfaceError(
"'%" + next_c + "' not supported in a quoted "
"string within the query string")
So if next_c == "'" instead of next_c == "%", then you would get your error
'%'' not supported in a quoted string within the query string
With a recent version of pg8000 you shouldn't have any problems with a % in a LIKE. For example:
>>> import pg8000.dbapi
>>>
>>> con = pg8000.dbapi.connect(user="postgres", password="cpsnow")
>>> cur = con.cursor()
>>> cur.execute("CREATE TEMPORARY TABLE book (id SERIAL, title TEXT)")
>>> for title in ("Ender's Game", "The Magus"):
... cur.execute("INSERT INTO book (title) VALUES (%s)", [title])
>>>
>>> cur.execute("SELECT * from book WHERE title LIKE 'The %'")
>>> cur.fetchall()
([2, 'The Magus'],)
I have a series of SQL select statements that I need to run from a Python script using MySQLdb. One of the variables I would like to pass to the select statements is called unit_ids. I've tried treating unit_ids as a string as well as a tuple of strings. Initially, backslashes were being inserted into the string. After looking around online, I've been able to avoid the backslashes, but now extra quotation marks are being inserted instead. Here is my current code:
connection = MySQLdb.connect('localhost', 'root', '*****', 'test')
cur = connection.cursor
unit_ids = ('0A1', '0A2', '0A3', '0A4')
attr = 'sample'
cur.execute("""SELECT COUNT(*) FROM test WHERE attribute = %s AND unit_id IN %r""", (a, tuple(unit_ids)))
Using cur._last_executed, I can see that the actual SQL statement that was performed was:
SELECT COUNT(*) FROM test WHERE attribute = 'sample' AND unit_id IN ("'0A1'", "'0A2'", "'0A3'", "'0A4'")
Any ideas on what I need to change in order to have ('0A1', '0A2', '0A3', '0A4') remain unchanged in the SQL statement?
UPDATE: Here's the exact output I get even when using %s:
>>> cn = MySQLdb.connect('localhost', 'root', '***', '***')
>>> c = cn.cursor()
>>> unit_ids = ('0A1', '0A2', '0A3', '0A4')
>>> a = 'foo'
>>> c.execute("""select count(*) from model_test where attribute = %s and unit_id in %s""", (a, unit_ids))
1L
>>> print(c._last_executed)
select count(*) from model_test where attribute = 'foo' and unit_id in ("'0A1'", "'0A2'", "'0A3'", "'0A4'")
At this point, I think I may just need to create separate variables for each element of unit_ids (eg unit_id1 = '0A1'). I'm using Python 2.7.9 and MySQL Server 5.6, by the way.
UPDATE 2: #thebjorn solved it: my version of MySQLdb was outdated. After upgrading, there were no more extra quotation marks being inserted into the SQL statement.
You don't need any magic, just do it the regular mysql way:
connection = MySQLdb.connect('localhost', 'root', '*****', 'test')
cur = connection.cursor()
unit_ids = ('0A1', '0A2', '0A3', '0A4')
attr = 'sample'
cur.execute("""SELECT COUNT(*) FROM test WHERE attribute = %s AND unit_id IN %s""", (a, unit_ids))
the only wrinkle I can see is if a was not included and unit_ids only had one item, then the tuple syntax could trip you up. If you make unit_ids into a list then the syntax wouldn't be quite as awkward:
unit_ids = ('0A1',)
cur.execute("SELECT COUNT(*) FROM test WHERE unit_id IN %s", (unit_ids,))
which when inlined becomes:
cur.execute("SELECT COUNT(*) FROM test WHERE unit_id IN %s", (('0A1',),))
vs. using lists (one parameter, that parameter is a one item list):
cur.execute("SELECT COUNT(*) FROM test WHERE unit_id IN %s", [['0A1']])
You can use %s for all mysql parameters (a few others can be used too, but not %r -- this isn't string interpolation).
Update: you must be doing something differently from me.. here is the output of cursor._last_executed
>>> cn = MySQLdb.connect('server', 'user', 'password', 'database')
>>> c = cn.cursor()
>>> unit_ids = ('0A1', '0A2', '0A3', '0A4')
>>> c.execute("select count(*) from foo where id in %s", (unit_ids,))
Traceback (most recent call last):
File ...
_mysql_exceptions.ProgrammingError: (1146, "Table 'djangodevelop.foo' doesn't exist")
>>> c._last_executed
"select count(*) from foo where id in ('0A1','0A2','0A3','0A4')"
>>>
Do not use %r in unit_id IN %r. Python-Mysql database API only supports %s as place holder when building SQL queries.
From the docs
In the preceding example, we store the SELECT statement in the variable query. Note that we are using unquoted %s-markers where dates should have been. Connector/Python converts hire_start and hire_end from Python types to a data type that MySQL understands and adds the required quotes. In this case, it replaces the first %s with '1999-01-01', and the second with '1999-12-31'.
You can see similar warning and usage in Python-MySql docs.
cur.execute("""
SELECT COUNT(*) FROM test WHERE attribute = %s AND unit_id IN %s
""", (a, ('0A1', '0A2', '0A3', '0A4')))
What is the best way to insert a Python dictionary with many keys into a Postgres database without having to enumerate all keys?
I would like to do something like...
song = dict()
song['title'] = 'song 1'
song['artist'] = 'artist 1'
...
cursor.execute('INSERT INTO song_table (song.keys()) VALUES (song)')
from psycopg2.extensions import AsIs
song = {
'title': 'song 1',
'artist': 'artist 1'
}
columns = song.keys()
values = [song[column] for column in columns]
insert_statement = 'insert into song_table (%s) values %s'
# cursor.execute(insert_statement, (AsIs(','.join(columns)), tuple(values)))
print cursor.mogrify(insert_statement, (AsIs(','.join(columns)), tuple(values)))
Prints:
insert into song_table (artist,title) values ('artist 1', 'song 1')
Psycopg adapts a tuple to a record and AsIs does what would be done by Python's string substitution.
You can also insert multiple rows using a dictionary. If you had the following:
namedict = ({"first_name":"Joshua", "last_name":"Drake"},
{"first_name":"Steven", "last_name":"Foo"},
{"first_name":"David", "last_name":"Bar"})
You could insert all three rows within the dictionary by using:
cur = conn.cursor()
cur.executemany("""INSERT INTO bar(first_name,last_name) VALUES (%(first_name)s, %(last_name)s)""", namedict)
The cur.executemany statement will automatically iterate through the dictionary and execute the INSERT query for each row.
PS: This example is taken from here
Something along these lines should do it:
song = dict()
song['title'] = 'song 1'
song['artist'] = 'artist 1'
cols=song.keys();
vals = [song[x] for x in cols]
vals_str_list = ["%s"] * len(vals)
vals_str = ", ".join(vals_str_list)
cursor.execute("INSERT INTO song_table ({cols}) VALUES ({vals_str})".format(
cols = cols, vals_str = vals_str), vals)
The key part is the generated string of %s elements, and using that in format, with the list passed directly to the execute call, so that psycopg2 can interpolate each item in the vals list (thus preventing possible SQL Injection).
Another variation, passing the dict to execute, would be to use these lines instead of vals, vals_str_list and vals_str from above:
vals_str2 = ", ".join(["%({0})s".format(x) for x in cols])
cursor.execute("INSERT INTO song_table ({cols}) VALUES ({vals_str})".format(
cols = cols, vals_str = vals_str2), song)
The new sql module was created for this purpose and added in psycopg2 version 2.7. According to the documentation:
If you need to generate dynamically an SQL query (for instance choosing dynamically a table name) you can use the facilities provided by the psycopg2.sql module.
Two examples are given in the documentation: http://initd.org/psycopg/docs/sql.html
names = ['foo', 'bar', 'baz']
q1 = sql.SQL("insert into table ({}) values ({})").format(
sql.SQL(', ').join(map(sql.Identifier, names)),
sql.SQL(', ').join(sql.Placeholder() * len(names)))
print(q1.as_string(conn))
insert into table ("foo", "bar", "baz") values (%s, %s, %s)
q2 = sql.SQL("insert into table ({}) values ({})").format(
sql.SQL(', ').join(map(sql.Identifier, names)),
sql.SQL(', ').join(map(sql.Placeholder, names)))
print(q2.as_string(conn))
insert into table ("foo", "bar", "baz") values (%(foo)s, %(bar)s, %(baz)s)
Though string concatenation would produce the same result, it should not be used for this purpose, according to psycopg2 documentation:
Warning: Never, never, NEVER use Python string concatenation (+) or string parameters interpolation (%) to pass variables to a SQL query string. Not even at gunpoint.
an other approach for query to mySQL or pgSQL from dictionary is using construction %(dic_key)s, it will be replaced by value from dictionary coresponding by dic_key like {'dic_key': 'dic value'}
working perfect, and prevent sqlInjection
tested: Python 2.7
see below:
# in_dict = {u'report_range': None, u'report_description': None, 'user_id': 6, u'rtype': None, u'datapool_id': 1, u'report_name': u'test suka 1', u'category_id': 3, u'report_id': None}
cursor.execute('INSERT INTO report_template (report_id, report_name, report_description, report_range, datapool_id, category_id, rtype, user_id) VALUES ' \
'(DEFAULT, %(report_name)s, %(report_description)s, %(report_range)s, %(datapool_id)s, %(category_id)s, %(rtype)s, %(user_id)s) ' \
'RETURNING "report_id";', in_dict)
OUT:
INSERT INTO report_template (report_id, report_name, report_description, report_range, datapool_id, category_id, rtype, user_id) VALUES (DEFAULT, E'test suka 1', NULL, NULL, 1, 3, NULL, 6) RETURNING "report_id";
Using execute_values https://www.psycopg.org/docs/extras.html is faster and has a fetch argument to return something. Next there is some code that might help.
columns is a string like col_name1, col_name2
template is the one that allows the matching, a string like %(col_name1)s, %(col_name2)
def insert(cur: RealDictCursor,
table_name: str,
values: list[dict],
returning: str = ''
):
if not values:
return []
query = f"""SELECT
column_name AS c
FROM
information_schema.columns
WHERE
table_name = '{table_name}'
AND column_default IS NULL;"""
cur.execute(query)
columns_names = cur.fetchall()
fetch = False
if returning:
returning = f'RETURNING {returning}'
fetch = True
columns = ''
template = ''
for col in columns_names:
col_name = col['c']
for val in values:
if col_name in val:
continue
val[col_name] = None
columns += f'{col_name}, '
template += f'%({col_name})s, '
else:
columns = columns[:-2]
template = template[:-2]
query = f"""INSERT INTO {table_name}
({columns})
VALUES %s {returning}"""
return execute_values(cur, query, values,
template=f'({template})', fetch=fetch)
Python has certain inbuilt features such as join and list using which one can generate the query. Also,the python dictionary offers keys() and values() which can be used to extract column name and column values respectively. This is the approach I used and this should work.
song = dict()
song['title'] = 'song 1'
song['artist'] = 'artist 1'
query = '''insert into song_table (''' +','.join(list(song.keys()))+''') values '''+ str(tuple(song.values()))
cursor.execute(query)
How to convert
cursor.execute("SELECT strftime('%m.%d.%Y %H:%M:%S', timestamp, 'localtime'), temp FROM data WHERE timestamp>datetime('now','-1 hours')")
# fetch all or one we'll go for all.
results = cursor.fetchall()
for row in results[:-1]:
row=results[-1]
rowstr="['{0}',{1}]\n".format(str(row[0]),str(row[1]))
temp_chart_table+=rowstr
result
['01.15.2015 21:38:52',21.812]
into dictionary output in form of:
[{timestamp:'01.15.2015 21:38:52',temp:21.812}]
Edit
This is fetchone sample I currenyly use and it works fine:
def get_avg():
conn=sqlite3.connect(dbname)
curs=conn.cursor()
curs.execute("SELECT ROUND(avg(temp), 2.2) FROM data WHERE timestamp>datetime('now','-1 hour') AND timestamp<=datetime('now')")
rowavg=curs.fetchone()
#print rowavg
#rowstrmin=format(str(rowavg[0]))
#return rowstrmin
**d = [{"avg":rowavg[0]}]**
return d
conn.close()
#print get_avg()
schema = {"avg": ("number", "avg")}
data = get_avg()
# Loading it into gviz_api.DataTable
data_table = gviz_api.DataTable(schema)
data_table.LoadData(data)
json = data_table.ToJSon()
#print results
#print "Content-type: application/json\n\n"
print "Content-type: application/json"
print
print json
Then I make jQuery call and pass it into javascript and found help for that in here
ajax json query directly to python generated html gets undefined
As I can see you are using format to write in the form of a string.
Note from the docs
it is not possible to use { and } as fill char while using the str.format() method
To make it look like a dictionary you can do
"[{timestamp:'%s',temp:%s}]\n"%(str(row[0]),str(row[1]))
But if you want to make it a dictionary then you will have to do
row_dic = [{'timestamp':row[0],'temp':row[1]}]
Try this instead:
cursor.execute("SELECT strftime('%m.%d.%Y %H:%M:%S', timestamp, 'localtime'), temp FROM data WHERE timestamp>datetime('now','-1 hours')")
# fetch all or one we'll go for all.
results = cursor.fetchall()
temp_chart_table = []
for row in results:
temp_chart_table.append({'timestamp': row[0], 'temp': row[1]})
In most of the python database adapters you can use a DictCursor to retrieve records using an interface similar to the Python dictionaries instead of the tuples.
Using psycopg2:
>>> dict_cur = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
>>> dict_cur.execute("INSERT INTO test (num, data) VALUES(%s, %s)",
... (100, "abc'def"))
>>> dict_cur.execute("SELECT * FROM test")
>>> rec = dict_cur.fetchone()
>>> rec['id']
1
>>> rec['num']
100
>>> rec['data']
"abc'def"
Using MySQLdb:
>>> import MySQLdb
>>> import MySQLdb.cursors
>>> myDb = MySQLdb.connect(user='andy47', passwd='password', db='db_name', cursorclass=MySQLdb.cursors.DictCursor)
>>> myCurs = myDb.cursor()
>>> myCurs.execute("SELECT columna, columnb FROM tablea")
>>> firstRow = myCurs.fetchone()
{'columna':'first value', 'columnb':'second value'}
def stuffToDict(stuff):
return {"timestamp":stuff[0],"temp":stuff[1]}
That would be a dictionary. The sample output you showed is a list of dictionaries, which can be achieved by putting square brackets around the dictionary. I don't know why you'd want that, though. Also, because of the missing quotes, it wasn't legal python syntax.
Use MySQLdb's cursor library.
import MySQLdb
import MySQLdb.cursors
conn = MySQLdb.connect(host=db_host, user=db_user, passwd=db_passwd, db=db_schema, port=db_port, cursorclass=MySQLdb.cursors.DictCursor)
cursor = conn.cursor()
cursor.execute("SELECT timestamp, localtime, temp FROM data WHERE timestamp>datetime('now','-1 hours')")
# fetch all or one we'll go for all.
results = cursor.fetchall()
Then you have access to the results as a dictionary:
>>> results['timestamp']
14146587
>>> results['localtime']
20:08:07
>>> results['temp']
temp_variable_whatever