String-encoding SQL statement - python

I have a very strange issue. I have a SQL statement that works locally, but the same statement does not work on a remote ubuntu machine (same mysql version). I think it has to do with how the SQL string is being encoded by the driver. Here is the statement I have:
group_ids = ('43ede7a1e1f048872c025867602dc54d', '43ede7a1e1f048872c025867602dc54d', '7a8ec12901c43606aee041f1e6d5b2d4', '0f57f4ad
cursor = connection.cursor()
cursor.execute( '''
SELECT
DISTINCT c.group_id
FROM
main_cue c
LEFT OUTER JOIN
main_passongroup p
ON
(c.group_id=p.group_id AND p.user_id=%s)
WHERE
c.group_id in %s
ORDER BY
p.timestamp ASC, c.id DESC''', (user.pk, group_ids))
results = cursor.fetchall()
print '>>> 1', results
On my local machine, it seems to be properly SQL-encoding the statement, but on the remote server, it is only working if I hard code the exact sql statement.
Is there a better way to encode the SQL statement?

This seemed to be an issue with an outdated version of MySQLdb, which was not properly encoding the string.
I was able to solve this by uninstalling MySQLdb and then reinstalling the newer version, (MySQL-python==1.2.5).

Related

How can I create a database with MySQL using query parameters? [duplicate]

I'm using Python + MySQL and want to use parameterized query. I'm stuck. I've encountered an error and can't figure out how to solve it. I've spent a day, checked dozens of articles, used various options (sinle quotes, double quotes, prepared statements) and still no luck.
Requirements: use Parameterized Query
Here is basic demo of the issue:
#!/usr/bin/python3
import mysql.connector as mysql
conn = mysql.connect(host=server, user=username, passwd=password, autocommit=True)
try:
create_database_query = "CREATE DATABASE %s;"
db_name = "BOOKS"
cursor = conn.cursor()
print(f"Creating {db_name} database... ", end='')
cursor.execute(create_database_query, (db_name,))
print("Success")
except mysql.Error as error:
print("Parameterized query failed {}".format(error))
Output:
Creating BOOKS database... Parameterized query failed 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ''BOOKS'' at line 1
So it looks like it uses too many quotes (2 single quotes on each side). The code above works fine if I change the following line:
create_database_query = "CREATE DATABASE %s;"
and put backtick around %s
The problem that now it creates a database but with invalid chars - 'BOOKS' (quotes are now part of db name). Duh...
If I use prepared statements then the same issue occurs but slightly different error message:
1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '?' at line 1
Environment:
MacOS Catalina
Python 3.8
PyCharm 2019.3 IDE
MySQL 8.0.19
mysql-connector-python module 8.0.19
What is going on? Any ideas?
Thanks
You can't use query parameters for identifiers (like a database name or table name or column name).
Query parameters can be used only in place of a constant value — a quoted string, quoted date/time, or a numeric value. Not identifiers, expressions, SQL keywords, etc.
To combine a database name with your CREATE DATABASE statement, you have to format it into the string in a way that forms the full statement before it is sent to MySQL.
db_name = "BOOKS"
create_database_query = "CREATE DATABASE %s;" % db_name
cursor.execute(create_database_query)
Because this creates a risk of SQL injection when you format variables into your string, it's up to you to make sure the db_name is safe.
Update: Thanks to #Parfait for the reminder about current best practices of string-formatting.
Prefer:
db_name = "BOOKS"
create_database_query = "CREATE DATABASE {};".format(db_name)
Or F-strings:
db_name = "BOOKS"
create_database_query = f"CREATE DATABASE {db_name};"
(In other words, Python has become Ruby ;-)

AppScreener says that my SQL code has some SQL Injection vulnerabilities

I am developing a project where I will have a backup of some forums and information coming from Lotus Notes. I am using Flask to run the backend. After check my code with the code scanner AppScreener, it says that my SQL code has some SQL Injection vulnerabilities but I can't undestand why.
This is the AppScreener result:
for usuario in data:
print("Usuario", usuario)
status = usuario['estatus_usuario']
mail = usuario['email_usuario']
cursor = conn.cursor()
cursor.execute( "UPDATE administrador_usuarios SET estatus_usuario=%s
WHERE email=%s",(status,mail)) # ---> this is the line where according to AppScreener is the vulnerability present
conn.commit()
conn.close()
return json.dumps({"response":"ok"})
Could you tell me what I can do?
It really depends on the DBAPI you're using for sql (pyodbc, pymysql, sqlite, etc). In most of these, I think the %s notation got depreciated a while ago. Best to use ? as people have commented (referenced links).

SQLite3 syntax error on tested sql script

I'm using python 3.6.4 and sqlite3 2.6.0 to query the nearest consecutive dates in my table in a sqlite 3.27.2 file.
I've tried to get the actual sql string with vscode debugger and test it with DB Browser for SQLite. It works as I expect.
Here's the code:
sql = 'WITH \
dates(cast_date) AS (\
SELECT DISTINCT play_date\
FROM TimeTable\
),\
groups AS (\
SELECT\
date(cast_date, \'-\'||(ROW_NUMBER() OVER (ORDER BY cast_date))||\' days\') AS grp,\
cast_date\
FROM dates\
)\
SELECT\
MIN(cast_date) AS date_start,\
MAX(cast_date) AS date_end\
FROM groups GROUP BY grp ORDER BY 2 DESC LIMIT 1'
cursor = conn.cursor()
result = []
try:
cursor.execute(sql)
result = cursor.fetchone()
except sqlite3.OperationalError:
FileLogger.exception('Exception at '+__file__+' '+__name__)
An exception occurs:
cursor.execute(sql)
sqlite3.OperationalError: near "OVER": syntax error
Window functions support was first added to SQLite with release version 3.25.0 (2018-09-15), according to official documentation.
When using Python, you are using Python SQLite3 client library (which is distributed with Python) instead of your system SQLite3 installation. For Python 2.7, the version is 3.11.0, which is below your required version.
You may try using a newer SQLite3 client library, as suggested by these answers.

sql INSERT in python (postgres, cursor, execute)

I had no problem with SELECTing data in python from postgres database using cursor/execute. Just changed the sql to INSERT a row but nothing is inserted to DB. Can anyone let me know what should be modified? A little confused because everything is the same except for the sql statement.
<!-- language: python -->
#app.route("/addcontact")
def addcontact():
# this connection/cursor setting showed no problem so far
conn = pg.connect(conn_str)
cur = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
sql = f"INSERT INTO jna (sid, phone, email) VALUES ('123','123','123')"
cur.execute(sql)
return redirect("/contacts")
first look at your table setup and make sure your variables are named right in the right order, format and all that, if your not logging into the specific database on the sql server it won't know where the table is, you might need to send something like 'USE databasename' before you do your insert statement so your computer is in the right place in the server.
I might not be up to date with the language but is that 'f' supposed to be right before the quotes? if thats in ur code that'd probably throw an error unless it has a use im not aware of or its not relevant to the problem.
You have to commit your transaction by adding the line below after execute(sql)
conn.commit()
Ref: Using INSERT with a PostgreSQL Database using Python

question about postgresql bind variables

I was looking at the question and decided to try using the bind variables. I use
sql = 'insert into abc2 (interfield,textfield) values (%s,%s)'
a = time.time()
for i in range(10000):
#just a wrapper around cursor.execute
db.executeUpdateCommand(sql,(i,'test'))
db.commit()
and
sql = 'insert into abc2 (intfield,textfield) values (%(x)s,%(y)s)'
for i in range(10000):
db.executeUpdateCommand(sql,{'x':i,'y':'test'})
db.commit()
Looking at the time taken for the two sets, above it seems like there isn't much time difference. In fact, the second one takes longer. Can someone correct me if I've made a mistake somewhere? using psycopg2 here.
The queries are equivalent in Postgresql.
Bind is oracle lingo. When you use it will save the query plan so the next execution will be a little faster. prepare does the same thing in Postgres.
http://www.postgresql.org/docs/current/static/sql-prepare.html
psycopg2 supports an internal 'bind', not prepare with cursor.executemany() and cursor.execute()
(But don't call it bind to pg people. Call it prepare or they may not know what you mean:)
IMPORTANT UPDATE :
I've seen into source of all python libraries to connect to PostgreSQL in FreeBSD ports and can say, that only py-postgresql does real prepared statements! But it is Python 3+ only.
also py-pg_queue is funny lib implementing official DB protocol (python 2.4+)
You've missed answer for that question about prepared statements to use as many as possible. "Binded variables" are better form of this, let's see:
sql_q = 'insert into abc (intfield, textfield) values (?, ?)' # common form
sql_b = 'insert into abc2 (intfield, textfield) values (:x , :y)' # should have driver and db support
so your test should be this:
sql = 'insert into abc2 (intfield, textfield) values (:x , :y)'
for i in range (10000):
cur.execute(sql, x=i, y='test')
or this:
def _data(n):
for i in range (n):
yield (i, 'test')
sql = 'insert into abc2 (intfield, textfield) values (? , ?)'
cur.executemany(sql, _data(10000))
and so on.
UPDATE:
I've just found interest reciple how to transparently replace SQL queries with prepared and with usage of %(name)s
As far as I know, psycopg2 has never supported server-side parameter binding ("bind variables" in Oracle parlance). Current versions of PostgreSQL do support it at the protocol level using prepared statements, but only a few connector libraries make use of it. The Postgres wiki notes this here. Here are some connectors that you might want to try: (I haven't used these myself.)
pg8000
python-pgsql
py-postgresql
As long as you're using DB-API calls, you probably ought to consider cursor.executemany() instead of repeatedly calling cursor.execute().
Also, binding parameters to their query in the server (instead of in the connector) is not always going to be faster in PostgreSQL. Note this FAQ entry.

Categories