I'm working on a bit of python code to run a query against a redshift (postgres) SQL database, and I'm running into an issue where I can't strip off the surrounding single quotes from a variable I'm passing to the query. I'm trying to drop a number of tables from a list. This is the basics of my code:
def func(table_list):
drop_query = 'drop table if exists %s' #loaded from file
table_name = table_list[0] #table_name = 'my_db.my_table'
con=psycopg2.connect(dbname=DB, host=HOST, port=PORT, user=USER, password=PASS)
cur=con.cursor()
cur.execute(drop_query, (table_name, )) #this line is giving me trouble
#cleanup statements for the connection
table_list = ['my_db.my_table']
when func() gets called, I am given the following error:
syntax error at or near "'my_db.my_table'"
LINE 1: drop table if exists 'my_db.my_table...
^
Is there a way I can remove the surrounding single quotes from my list item?
for the time being, I've done it (what think is) the wrong way and used string concatenation, but know this is basically begging for SQL-injection.
This is not how psycopg2 works. You are using a string operator %s to replace with a string. The reason for this is to tokenize your string safely to avoid SQL injection, psycopg2 handles the rest.
You need to modify the query before it gets to the execute statement.
drop_query = 'drop table if exists {}'.format(table_name)
I warn you however, do not allow these table names to be create by outside sources, or you risk SQL injection.
However a new version of PSYCOPG2 kind of allows something similar
http://initd.org/psycopg/docs/sql.html#module-psycopg2.sql
from psycopg2 import sql
cur.execute(
sql.SQL("insert into {} values (%s, %s)").format(sql.Identifier('my_table')),[10, 20]
)
Related
I would like to write a python script to create new MariaDB databases.
The database name is a user input. I tried to use arguments for creating the database:
#!/usr/bin/python3
import mysql.connector
mariadb_host = '127.0.0.1'
mariadb_port = 3306
mariadb_user = 'root'
mariadb_password = 'password'
mariadb_connection = mysql.connector.connect(
host=mariadb_host,
port=mariadb_port,
user=mariadb_user,
passwd=mariadb_password,
use_pure=True
)
query = 'CREATE DATABASE %(db_name)s;'
args = {'db_name': 'test-db'}
result = None
cursor = mariadb_connection.cursor()
cursor.execute(query, args)
print(cursor.statement)
result = cursor.fetchall()
cursor.close()
The following error appears: mysql.connector.errors.ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ''test-db'' at line 1
It seems, that the command cursor.execute appends ' around the database name, which results in an invalid sql query.
How could I get around this problem and create safely new database from user input?
Parameter substitution notation - %(name)s or just %s is for interpolating values into an SQL statement.
RDBMSs have different quoting rules for values and identifiers like database, table or column names. For example, a string value will be surrounded by single quotes to tell the RDBMS that is is a character value, but single-quoting an identifier is a syntax error; the RDBMS will require that identifiers are quoted using some other character (for example backticks, double-quotes, square brackets, depending on the RDBMS).
If you want to interpolate identifiers using Python you have to use string formatting techniques. For example, using an f-string
db_name = 'test-db'
query = f'CREATE DATABASE `{db_name}`;'
Note that it is best to quote dynamic identifier names with backticks to handle names which contain special characters.
As always with dynamic SQL generation, you should be aware of the risk of SQL injection when handling data from an untrusted source.
I'm trying to transfer a user input from a python code to a table in postgresql
What I want to do is place an input() in this code and make it's value go to the comment (#) in the code.
conn = psycopg2.connect(
host="localhost",
database="Twitterzuil",
user="postgres",
password="")
cur = conn.cursor()
cur.execute("INSERT INTO Bericht2 (name) VALUES (#THIS IS WHERE I WANT THE INPUT TO GO)");
conn.commit()
I have no idea how, I'm really a beginner in all this so any help is appreciated
I believe what you are asking about is called string interpolation. Using f-style format, this might look like
new_name = "'bob'" # need single quotes for SQL strings
sql = f"INSERT INTO Bericht2 (name) VALUES ({new_name})" # => sql == "INSERT INTO Bericht2 (name) VALUES ('bob')"
cur.execute(sql)
Note the f at the start of the string, when you do this expressions inside {} pairs get replaced with their python values (tutorial). There are also string formatting approaches involving % substitution and the .format method on strings.
If you are doing anything beyond the basics you should look into using the SQLAlchemy package; here's the link to their insert api. Using SQLAlchemy will help reduce the risks that can come with manually constructing SQL queries.
Example from "Inserting Rows with SQLAlchemy"
from sqlalchemy import insert
stmt = insert(user_table).values(name='spongebob', fullname="Spongebob Squarepants")
with engine.connect() as conn:
result = conn.execute(stmt)
conn.commit()
I am trying to write a simple Python script to bulk add movie titles into a local database, using the MySQLdb (mysqlclient) package. I am reading the titles from a TSV file. But when go to sanitize the inputs using MySQLdb::escape_string(), I get the character b before my string. I believe this means that SQL is interpreting it as a bit value, but when I go to execute my query I get the following error:
You have an error in your SQL syntax; check the manual that
corresponds to your MariaDB server version for the right syntax to use
near 'b'Bowery to Bagdad',1955)' at line 1"
The select statement in question:
INSERT INTO movies (imdb_id, title, release_year) VALUES ('tt0044388',b'Bowery to Bagdad',1955)
def TSV_to_SQL(file_to_open):
from MySQLdb import _mysql
db=_mysql.connect(host='localhost', user='root', passwd='', db='tutorialdb', charset='utf8')
q = """SELECT * FROM user_id"""
# MySQLdb.escape_string()
# db.query(q)
# results = db.use_result()
# print(results.fetch_row(maxrows=0, how=1))
print("starting?")
with open(file_to_open, encoding="utf8") as file:
tsv = csv.reader(file, delimiter="\t")
count = 0
for line in tsv:
if count == 10:
break
# print(MySQLdb.escape_string(line[1]))
statement = "INSERT INTO movies (imdb_id, title, release_year) VALUES ('{imdb_id}',{title},{year})\n".format(
imdb_id=line[0], title=MySQLdb.escape_string(line[1]), year=line[2])
# db.query(statement)
print(statement)
count = count + 1
I know a simple solution would be to just remove the character b from the start of the string, but I was wondering if there was a more proper way, or if I missed something in documentation.
The 'b' infront of the string represents that the string is binary encoded rather than a literal string.
If you use .encode() you will be able to get what you want.
How to convert 'binary string' to normal string in Python3?
It's more common to let the connector perform the escaping automatically, by inserting placeholders in the SQL statement and passing a sequence (conventionally a tuple) of values as the second argument to cursor.execute.
conn = MySQLdb.connect(host='localhost', user='root', passwd='', db='tutorialdb', charset='utf8')
cursor = conn.cursor()
statement = """INSERT INTO movies (imdb_id, title, release_year) VALUES (%s, %s, %s)"""
cursor.execute(statement, (line[0], line[1], line[2]))
conn.commit()
The resulting code is more portable - apart from the connection it will work with all DB-API connectors*. Dropping down to low-level functions like _mysql.connect and escape_string is unusual in Python code (though you are perfectly free to code like this if you want, of course).
* Some connection packages may use a different placeholder instead of %s, but %s seems to be the favoured placeholder for MySQL connector packages.
I'm trying to fill a PostgreSQL table (psycopg2, Python) with the filenames I have in a specific folder. I have created a function that should do the trick, but I get the error:
not all arguments converted during string formatting,
when I run my function. I did a test run and called the function in the following way:
insert_file_names_into_database(["filename1_without_extension", "filename2_without_extension"]),
and I had no problems and the INSERT worked fine. If I did the following:
insert_file_names_into_database(["filename1.extension", "filename2.extension"]),
Then I get the error above. So the problem seems to be the "." character (e.g. image.jpg) which causes the SQL INSERT to fail. I tried to consult the Psycopg2 docs about this, but I found no examples relating to this specific case.
How should I edit the piece of code so I can get to work even with "." characters in the filenames?
def insert_file_names_into_database(file_name_list):
""" insert multiple filenames into the table """
sql = "INSERT INTO mytable(filename) VALUES(%s)"
conn = None
try:
# read database configuration
# connect to the PostgreSQL database
conn = psycopg2.connect(
host="localhost",
database="mydatabase",
user="myusername",
password="mypassword")
# create a new cursor
cur = conn.cursor()
# execute the INSERT statement
cur.executemany(sql, file_name_list)
# commit the changes to the database
conn.commit()
# close communication with the database
cur.close()
except (Exception, psycopg2.DatabaseError) as error:
print(error)
finally:
if conn is not None:
conn.close()
Solved it myself already. I knew I should be using tuples when working with the INSERT, but my function worked fine with list of strings without the "." characters.
The solution I got working was to convert the list of strings into a list of tuples like so:
tuple_file_name = [tuple((file_name,)) for file_name in file_name_list]
So for example if:
file_name_list = ["filename1.jpg", "filename2.jpg"]
Then giving this as input to my function fails. But by making it a list of tuples:
tuple_file_name = [tuple((file_name,)) for file_name in file_name_list]
print(tuple_file_name)
[('filename1.jpg',), ('filename2.jpg',)]
Then now the function accepts the input tuple_file_name and the filenames are saved into the SQL table.
I'm a postgres newbie and am having some issues querying a text field in postgresql using Python. What is the correct syntax that will allow me to search the content of column "body" from table "jivemessage" out of database "postgres"?
try:
conn = psycopg2.connect("dbname='postgres' user='postgres' host='localhost' password='<password>'")
except:
print "cannot connect"
i = 'test'
cur = conn.cursor()
cur.execute('SELECT * from jivemessage WHERE body LIKE "%'+i+'%"')
Keep getting the following error:
ProgrammingError: column "%test%" does not exist
Thanks for any help.
You are not quoting the query properly. Don't use string concatenation here, use SQL parameters instead:
cur.execute('SELECT * from jivemessage WHERE body LIKE %s', ("%{}%".format(i),))
Here, the %s placeholder signals to the database driver that the first value of the second argument should be placed there when querying.
This leaves the interpolation up to the database driver, giving the database the opportunity to optimize for the query once, even if you were to reuse the same query.
It also prevents SQL injection attacks better than you could yourself, and most of all, guarantees that the correct quoting rules are followed.