I'm inserting data using execute_values, which takes a sql query. The query is constructed using psycopg2.sql.SQL as recommended in the documentation, but execute_values won't take that object.
Here's the code I have:
import psycopg2 as pg
from psycopg2 import extras
from psycopg2 import sql
config = {
'host' : 'localhost',
'user' : 'username',
'password' : 'password',
'dbname' : 'myDatabase'
}
connection = pg.connect(**config)
cursor = connection.cursor()
tableName = 'myTable'
dataset = [[1,2],[3,4],[5,6]]
queryText = "INSERT INTO {table} (uid,value) VALUES %s"
query = sql.SQL(queryText).format(table=sql.Identifier(tableName))
extras.execute_values(cursor,query,dataset)
The last line gives the following error:
AttributeError: 'Composed' object has no attribute 'encode'
If the query is specified directly as a string, as below, then the execution runs.
query = """INSERT INTO "myTable" (uid,value) VALUES %s"""
It's possible to insert the table name into the query using string format, but apparently that shouldn't be done, even at gunpoint. How can I safely insert a variable table name into the query and use execute_values? I can't find a built-in way to convert the SQL object to a string.
The parameter sql in execute_values(cur, sql, argslist, template=None, page_size=100) is supposed to be a string:
sql – the query to execute. It must contain a single %s placeholder, which will be replaced by a VALUES list. Example: "INSERT INTO mytable (id, f1, f2) VALUES %s".
Use the as_string(context) method:
extras.execute_values(cursor, query.as_string(cursor), dataset)
connection.commit()
As execute_values() expect the sql statement to be a string you can simply user:
queryText = "INSERT INTO {table} (uid,value) VALUES %s".format(table=sql.Identifier(tableName)
Related
I have a following query:
cursor = connection.cursor()
query = """
SELECT *
FROM `my_database`.table_a
"""
result = cursor.execute(query)
which works as expected. But I need to change my_database in cursor.execute. I try:
cursor = connection.cursor()
query = """
SELECT *
FROM %s.table_a
"""
result = cursor.execute(query, ("my_database",))
which gives an error pymysql.err.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ''my_database'.table_a at line 2")
how can I insert database name in cursor.execute please?
It is not possible to bind a database name (or any other database object name) using a placeholder in a prepared statement. This would be, among other problems, a security risk. However, you might be able to use an f-string here instead:
cursor = connection.cursor()
db_name = "my_database"
query = f"""
SELECT *
FROM {db_name}.table_a
"""
result = cursor.execute(query)
It should also be mentioned that the above is only SQL injection safe if you are certain that the database name is not coming from outside your own application.
As described here https://peps.python.org/pep-0249/#paramstyle, in MySql it should be possible to bind parameters using the keyword syntax, like this: email=:email. This is different from using an unnamed placehoder syntax, like email=%s.
But this code doesn't work:
import pymysql
con = pymysql.connect(host='localhost', user='root', password=pw, database=db_name, port=4306)
stmt = "INSERT INTO `test_table` (`email`, `password`) VALUES (:email, :password)"
with con.cursor() as cursor:
# Create a new record
cursor.execute(stmt, {'email': "FOO", 'password': "BAR"})
con.commit()
Not even adding
pymysql.paramstyle = 'named'
at the top.
The error is
(1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ':email, password=:password)' at line 1")
but unfortunately, I'm not able to find such documentation (This page doesn't document anything.. https://dev.mysql.com/doc/refman/8.0/en/sql-prepared-statements.html)
What is the right syntax?
Thanks!
https://pymysql.readthedocs.io/en/latest/modules/cursors.html says:
If args is a list or tuple, %s can be used as a placeholder in the query. If args is a dict, %(name)s can be used as a placeholder in the query.
Even though the :name placeholder format is in the PEP you reference, the pymysql package does not seem to implement that format.
This should work:
stmt = "INSERT INTO `test_table` (`email`, `password`) VALUES (%(email)s, %(password)s)"
with con.cursor() as cursor:
# Create a new record
cursor.execute(stmt, {'email': "FOO", 'password': "BAR"})
I am using cx_Oracle module to connect to oracle database. In the script i use two variables schema_name and table_name. The below query works fine
cur1.execute("select owner,table_name from dba_tables where owner ='schema_name'")
But i need to query the num of rows of a table, where i need to qualify the table_name with the schema_name and so the query should be
SELECT count(*) FROM "schema_name"."table_name"
This does not work when using in the code, i have tried to put it in triple quotes, single quotes and other options but it does not format the query as expected and hence errors out with table does not exist.
Any guidance is appreciated.
A prepared statement containing placeholders with variables of the form ...{}.{}".format(sc,tb) might be used
sc='myschema'
tb='mytable'
cur1.execute("SELECT COUNT(*) FROM {}.{}".format(sc,tb))
print(cur1.fetchone()[0])
In this particular case, you could also try setting Connection.current_schema, see the cx_Oracle API doc
For example, if you create table in your own schema:
SQL> show user
USER is "CJ"
SQL> create table ffff (mycol number);
Table created.
SQL> insert into ffff values (1);
1 row created.
SQL> commit;
Commit complete.
Then run Python code that connects as a different user:
import cx_Oracle
import os
import sys, os
if sys.platform.startswith("darwin"):
cx_Oracle.init_oracle_client(lib_dir=os.environ.get("HOME")+"/Downloads/instantclient_19_8")
username = "system"
password = "oracle"
connect_string = "localhost/orclpdb1"
connection = cx_Oracle.connect(username, password, connect_string)
connection.current_schema = 'CJ';
with connection.cursor() as cursor:
sql = """select * from ffff"""
for r in cursor.execute(sql):
print(r)
sql = """select sys_context('USERENV','CURRENT_USER') from dual"""
for r in cursor.execute(sql):
print(r)
the output will be:
(1,)
('SYSTEM',)
The last query shows that it is not the user that is being changed, but just the first query is automatically changed from 'ffff' to 'CJ.ffff'.
I'm trying to create a database with the name a user will provide. As far as I know the correct way is to use the second argument of execute().
So I did as follows:
import psycopg2
conn = psycopg2.connect(host="...", dbname="...",
user="...", password="...", port='...')
cursor = conn.cursor()
query = ''' CREATE DATABASE %s ;'''
name = 'stackoverflow_example_db'
conn.autocommit = True
cursor.execute(query, (name,))
cursor.close()
conn.close()
And I got this error:
psycopg2.errors.SyntaxError: syntax error at or near "'stackoverflow_example_db'"
LINE 1: CREATE DATABASE 'stackoverflow_example_db' ;
I need to do this statement avoiding SQL injection, so using the second argument is a must.
You can't pass values as second argument of execute(), if the statement is a CREATE DATABASE one.
As pointed out by unutbu one way to approach this is using the psycopg2.sql submodule and use identifiers to build the statement avoiding SQL injection.
The code:
import psycopg2
from psycopg2 import sql
conn = psycopg2.connect(host="...", dbname="...",
user="...", password="...", port='...')
cursor = conn.cursor()
query = ''' CREATE DATABASE {} ;'''
name = 'stackoverflow_example_db'
conn.autocommit = True
cursor.execute(sql.SQL(query).format(
sql.Identifier(name)))
cursor.close()
conn.close()
Other aditional observations:
format() do not work with %s, use {} instead
Autocommit mode is a must for this statement to work
The specified connection user needs creation privileges
From the PostgreSQL documentation if you do a INSERT without specifying a schema it should be a public schema.
conn = psycopg2.connect(dbname = 'orion',
host = 'localhost',
port = 5432,
user = 'earthling',
password = 'mysupersecretpassword')
sql = conn.cursor()
def INSERT(table, info, text):
date = datetime.date.today()
query = "INSERT INTO %s (info, text, date) " \
"VALUES (%s, %s, %s)" %(table, info, text, date)
sql.execute(query)
INSERT("main", "www.capecod.edu", "test")
For some reason I'm seeing the following error?
psycopg2.ProgrammingError: schema "www" does not exist
You are using string interpolation to create the query. This is what psycopg2 executes:
INSERT INTO main (info, text, date)
VALUES (www.capecod.edu, test, 2015-09-12)
If it's not obvious what's wrong here, it's that none of the values are quoted. Here is the properly quoted version:
INSERT INTO main (info, text, date)
VALUES ('www.capecod.edu', 'test', '2015-09-12')
The error is caused by the unquoted www.capecod.edu. Due to the dots, it's being interpreted as schema.table.column.
The "right" way to do this is with a parameterized query.
query = "INSERT INTO main (info, text, date) VALUES (%s, %s, %s)"
params = (info, text, date)
sql.execute(query, params)
psycopg2 will figure out what should be quoted and how. This is a safer option than simply interpolating the string yourself, which often leaves you open to SQL injection attack.
http://initd.org/psycopg/articles/2012/10/01/prepared-statements-psycopg/
Unfortunately, you can't just toss identifiers such as the table name in as a parameter, because then they are quoted as string values, which is bad SQL syntax. I found an answer (python adds "E" to string) that points to psycopg2.extensions.AsIs as a way to pass identifiers such as table names safely as parameters. I wasn't able to make this work in my testing, though.
If you go the AsIs route, you should be cautious about checking the table names are valid, if they somehow come from user input. Something like
valid_tables = ["main", "foo", "bar", "baz"]
if table not in valid_tables:
return False