psycopg2 difference between AsIs and sql module - python

To choose dynamically a table name in a query I used to use AsIs() from psycopg2.extensions ( http://initd.org/psycopg/docs/extensions.html#psycopg2.extensions.AsIs ), with the following syntax:
cur.execute("SELECT * FROM %s WHERE id = %s;", (AsIs('table_name'), id))
However, the documentation now recommends to use the new psycopg2.sql module available in version 2.7 ( http://initd.org/psycopg/docs/sql.html#module-psycopg2.sql ) with the following syntax:
from psycopg2 import sql
cur.execute(
sql.SQL("SELECT * FROM {} WHERE id = %s;")
.format(sql.Identifier('table_name')), (id, )
What's the difference between those two options besides the fact that objects exposed by the sql module can be passed directly to execute()?

AsIs is... as it is. It won't perform any escape of the table name, if it contains characters need quoting. The objects in the sql module instead know what is an identifier.
More subtly, AsIs is for parameter values only: if currently works is mostly an implementation accident and in the future the behaviour may change. Query values should not be used to represent variable parts of the query, such as table or field names.

Related

sqlite3.OperationalError: near "index": syntax error

I am trying to connect to a database with python using sqlite3 module and i gets an error - sqlite3.OperationalError: near "index": syntax error
I searched some solutions for this but i did not got the solution. I
am new to sqlite3
def insert_into_db(url, title, description, keywords):
con = sqlite3.connect('index.db')
c = con.cursor()
create = r'''CREATE TABLE IF NOT EXISTS index (id INTEGER NOT NULL AUTO_INCREMENT,url VARCHAR,description TEXT,keywords TEXT);INSERT INTO index(url, title, description, keywords)VALUES('{}','{}',{}','{}');'''.format(url, title,description, keywords)
c.execute(create)
con.commit()
con.close()
help me to get rid of this error :(
INDEX is a keyword in SQLite3. Thus, it'll be parsed as a keyword. There are several ways around this, though.
According to the documentation, you could use backticks or quote marks to specify it as a table name. For example,
CREATE TABLE IF NOT EXISTS `index` ...
or
CREATE TABLE IF NOT EXISTS "index" ...
may work.
You can pass arguments to your sql statement from the execute() command. Thus,
create = r'''CREATE TABLE ... VALUES(?,?,?,?);''' # use ? for placeholders
c.execute(create, (url, title, description, keywords)) # pass args as tuple
This is more secure compared to formatting your arguments directly with Python.
Note also that SQLite's syntax for autoinc is AUTOINCREMENT without the underscore and they require the field to also be an INTEGER PRIMARY KEY.
You can not name a table index. INDEX is a reserved keyword.
The documentation states:
The SQL standard specifies a large number of keywords which may not be used as the names of tables, indices, columns, databases, user-defined functions, collations, virtual table modules, or any other named object.

To distinguish variable name from DB field name in Python

Code
import psycopg2
import psycopg2.extras
def store_values_to_pg94(file_size, connection):
cursor = connection.cursor()
cursor.execute(
INSERT INTO measurements
(file_size)
VALUES (file_size)
where the field name of the column in PostgreSQL 9.4 is file_size but the variable name is also file_size in Python.
How can you avoid this kind of conflict?
The code from your example does not compile, i.e. it is not valid python. cursor.execute() accepts a string and an optional set of arguments that are expanded into the string according to the database module's paramstyle.
In fact, when using the DBAPI, your problem does not even surface, because SQL syntax and python syntax are completely separate.
If you are using psycopg2 to access PostgreSQL, paramstyle is pyformat, hence
cursor.execute(
'INSERT INTO measurements (file_size) VALUES (%(file_size)s);',
file_size=file_size # pass variable file_size as named parameter file_size
);
As you can see, you specify the SQL query and leave placeholders for the values that you want to pass in from python code.
If paramstyle == 'pyformat', it is also possible to use positional parameters:
cursor.execute(
'INSERT INTO measurements (file_size) VALUES (%s);',
file_size # pass variable file_size as positional parameter
);
Note that different database access modules implement different paramstyle's and need different parameter placeholders (or a suitable wrapper).
All parameter substitution implementations are expected to properly escape their arguments according to the rules of the database backend, which is a nice and important feature as you don't have to manually escape your data.

Lightweight DBAL for python

can somebody please recomend me some python DBAL library that will best suit my requirements. I would like to write my sql statements directly, most of the logics will be in db stored procedures (postgresql), so I only need to invoke db procedures, pass arguments to them and fetch the results. The library should help me with quoting (preventing sql inject).
I played with sqlalchemy, but i think that there is no quoting helper when writing sql statement directly to engine.execute method.
Thank you
You should have given sqlalchemy a deeper look; It does a fine job of quoting placeholders:
>>> engine = sqlalchemy.create_engine("sqlite:///:memory:")
>>> engine.execute("select ?", 5).fetchall()
[(5,)]
>>> engine.execute("select ?", "; drop table users; --").fetchall()
[(u'; drop table users; --',)]
psycopg2 (via DB-API) will automatically quote to prevent SQL injection, IF you use it properly. (The python way is wrong; you have to pass the parameters as arguments to the query command itself.)
WRONG:
cur.execute('select * from table where last="%s" and first="%s"'
% (last, first))
RIGHT:
cur.execute('select * from table where last=%s and first=%s',
(last, first))
Note: you don't use %, and you don't put quotes around your values.
The syntax is slightly different for MySQLdb and sqlite3. (For example, sqlite uses ? instead of %s.)
Also, for psycopg2, always use %s even if you're dealing with numbers or some other type.

select from sqlite table where rowid in list using python sqlite3 — DB-API 2.0

The following works:
>>> cursor.execute("select * from sqlitetable where rowid in (2,3);")
The following doesn't:
>>> cursor.execute("select * from sqlitetable where rowid in (?) ", [[2,3]] )
sqlite3.InterfaceError: Error binding parameter 0 - probably unsupported type.
Is there a way to pass in a python list without having to format it into a string first ?
Unfortunately not. Each value must be given its own parameter mark (?).
Since the argument list can (presumably) have arbitrary length, you must use string formating to build the correct number of parameter marks. Happily, that isn't so hard:
args=[2,3]
sql="select * from sqlitetable where rowid in ({seq})".format(
seq=','.join(['?']*len(args)))
cursor.execute(sql, args)
In Python 3.6 you can also build queries with the f strings:
args=[2, 3]
query = f"SELECT * FROM sqlitetable WHERE rowid in ({','.join(['?']*len(args))})"
cursor.execute(query, args)
SQLite natively supports only the types TEXT, INTEGER, REAL, BLOB and NULL. If you want to use other types you must add support for them yourself. The detect_types parameter and the using custom converters registered with the module-level register_converter() function allow you to easily do that.
As described before, SQLite supports only a limited set of types natively.
To use other Python types with SQLite, you must adapt them to one of the sqlite3 module’s supported types for SQLite: one of NoneType, int, float, str, bytes.
https://docs.python.org/3.6/library/sqlite3.html#using-adapters-to-store-additional-python-types-in-sqlite-databases

Parameterized queries with psycopg2 / Python DB-API and PostgreSQL

What's the best way to make psycopg2 pass parameterized queries to PostgreSQL? I don't want to write my own escpaing mechanisms or adapters and the psycopg2 source code and examples are difficult to read in a web browser.
If I need to switch to something like PyGreSQL or another python pg adapter, that's fine with me. I just want simple parameterization.
psycopg2 follows the rules for DB-API 2.0 (set down in PEP-249). That means you can call execute method from your cursor object and use the pyformat binding style, and it will do the escaping for you. For example, the following should be safe (and work):
cursor.execute("SELECT * FROM student WHERE last_name = %(lname)s",
{"lname": "Robert'); DROP TABLE students;--"})
From the psycopg documentation
(http://initd.org/psycopg/docs/usage.html)
Warning Never, never, NEVER use Python string concatenation (+) or string parameters interpolation (%) to pass variables to a SQL query string. Not even at gunpoint.
The correct way to pass variables in a SQL command is using the second argument of the execute() method:
SQL = "INSERT INTO authors (name) VALUES (%s);" # Note: no quotes
data = ("O'Reilly", )
cur.execute(SQL, data) # Note: no % operator
Here are a few examples you might find helpful
cursor.execute('SELECT * from table where id = %(some_id)d', {'some_id': 1234})
Or you can dynamically build your query based on a dict of field name, value:
query = 'INSERT INTO some_table (%s) VALUES (%s)'
cursor.execute(query, (my_dict.keys(), my_dict.values()))
Note: the fields must be defined in your code, not user input, otherwise you will be susceptible to SQL injection.
I love the official docs about this:
https://www.psycopg.org/psycopg3/docs/basic/params.html

Categories