I set queries like following . I'd like to replace bucket_name and file_name with variables
This query is executed by psycopg2
query = '''copy table.test
from 's3://%(bucket_name)s/%(file_name)s.txt'
iam_role 'arn:aws:iam::123453215125:role/test'
timeformat as 'auto'
ACCEPTINVCHARS
delimiter '\t';'''
And I get bucket name and file name
bucket_name = get_bucket_name(event)
file_name = get_file_name(event)
After that, I executed query, but it returned errors
cur.execute(query, {'bucket_name':bucket_name,'file_name':file_name})
[ERROR] SyntaxError: syntax error at or near "expired" LINE 2: from 's3://'expired-test-bucket... ^ Traceback (most recent cal
It seems that bucket_name is replaced with single quotation 'expired-test-bucket'
My desired result is expired-test-bucket
How can I fix this?
Parameters in parameterized queries do not work like string interpolation, even if you use pyformat binding style that happens to look like old-school %-based string interpolation. The quotes are there because the database engine is expecting you to use the placeholder to represent an entire parameter, and putting quotes (and potentially various forms of escaping) into the final query text by design, specifically to prevent the security hole that results from interpolating user data directly into a query.
If you need to use user data to build up a value that is used in a query, then do just that, as separate steps: first, use Python string formatting to create the raw underlying value, and then use the SQL engine's functionality to make it safe to put that value into a query.
So, something like:
bucket_name = get_bucket_name(event)
file_name = get_file_name(event)
url = f's3://{bucket_name}/{file_name}.txt'
query = '''copy table.test
from %s
iam_role 'arn:aws:iam::123453215125:role/test'
timeformat as 'auto'
ACCEPTINVCHARS
delimiter '\t';'''
cur.execute(query, (url,))
Related
I'm working on a bit of python code to run a query against a redshift (postgres) SQL database, and I'm running into an issue where I can't strip off the surrounding single quotes from a variable I'm passing to the query. I'm trying to drop a number of tables from a list. This is the basics of my code:
def func(table_list):
drop_query = 'drop table if exists %s' #loaded from file
table_name = table_list[0] #table_name = 'my_db.my_table'
con=psycopg2.connect(dbname=DB, host=HOST, port=PORT, user=USER, password=PASS)
cur=con.cursor()
cur.execute(drop_query, (table_name, )) #this line is giving me trouble
#cleanup statements for the connection
table_list = ['my_db.my_table']
when func() gets called, I am given the following error:
syntax error at or near "'my_db.my_table'"
LINE 1: drop table if exists 'my_db.my_table...
^
Is there a way I can remove the surrounding single quotes from my list item?
for the time being, I've done it (what think is) the wrong way and used string concatenation, but know this is basically begging for SQL-injection.
This is not how psycopg2 works. You are using a string operator %s to replace with a string. The reason for this is to tokenize your string safely to avoid SQL injection, psycopg2 handles the rest.
You need to modify the query before it gets to the execute statement.
drop_query = 'drop table if exists {}'.format(table_name)
I warn you however, do not allow these table names to be create by outside sources, or you risk SQL injection.
However a new version of PSYCOPG2 kind of allows something similar
http://initd.org/psycopg/docs/sql.html#module-psycopg2.sql
from psycopg2 import sql
cur.execute(
sql.SQL("insert into {} values (%s, %s)").format(sql.Identifier('my_table')),[10, 20]
)
I always failled to insert data to Mysql database from my telegram bot, and always run Exception. Only tanggald allways failed to insert. I thing format of date insert query is wrong. How to write correct format?
tanggald column detail : Data Type = DATE
This is piece of code:
def process_lanjut(message):
try:
chat_id = message.chat.id
qlanjut = message.text
user = user_dict[chat_id]
user.qlanjut=qlanjut
d = datetime.datetime.now().date()
next_monday = next_weekday(d, 4)
user.next_monday = next_monday
print(user.next_monday)
with con.cursor() as cursor:
sql = "INSERT INTO diagnosa(sex, tanggald) VALUES('" + user.sex + "','" +next_monday+ "')"
cursor.execute(sql)
con.commit()
con.close()
msg = bot.send_message(chat_id, 'thanks')
bot.register_next_step_handler(msg, send_end)
except Exception as e:
bot.reply_to(message,'oops lanjut')
On command line output : 2018-04-20 (data that should be INSERT to tanggald)
That's not the proper way to insert data into table. Although your way may work, it is not safe and it lacks data escaping (', ", etc.):
The SQL representation of many data types is often different from their Python string representation. The typical example is with single quotes in strings: in SQL single quotes are used as string literal delimiters, so the ones appearing inside the string itself must be escaped, whereas in Python single quotes can be left unescaped if the string is delimited by double quotes.
Because of the difference, sometime subtle, between the data types representations, a naïve approach to query strings composition, such as using Python strings concatenation, is a recipe for terrible problems.
If the variables containing the data to send to the database come from an untrusted source (such as a form published on a web site) an attacker could easily craft a malformed string, either gaining access to unauthorized data or performing destructive operations on the database. This form of attack is called SQL injection and is known to be one of the most widespread forms of attack to database servers. Before continuing, please print this page as a memo and hang it onto your desk.
So in your case the INSERT statement should look like this:
with con.cursor() as cursor:
sql = 'INSERT INTO diagnosa (sex, tanggald) VALUES (%s, %s)'
data = (user.sex, next_monday)
cursor.execute(sql, data)
Further reading:
MySQL docs and example
Similar article for PostgreSQL (contains a better explanation of potential problems)
I am aware that queries in Python can be parameterized using either ? or %s in execute query here or here
However I have some long query that would use some constant variable defined at the beginning of the query
Set #my_const = 'xyz';
select #my_const;
-- Query that use #my_const 40 times
select ... coalesce(field1, #my_const), case(.. then #my_const)...
I would like to do the least modif possible to the query from Mysql. So that instead of modifying the query to
pd.read_sql(select ... coalesce(field1, %s), case(.. then %s)... , [my_const, my_const, my_const, ..]
,I could write something along the line of the initial query. Upon trying the following, however, I am getting a TypeError: 'NoneType' object is not iterable
query_str = "Set #null_val = \'\'; "\
" select #null_val"
erpur_df = pd.read_sql(query_str, con = db)
Any idea how to use the original variable defined in Mysql query ?
The reason
query_str = "Set #null_val = \'\'; "\
" select #null_val"
erpur_df = pd.read_sql(query_str, con = db)
throws that exception is because all you are doing is setting null_value to '' and then selecting that '' - what exactly would you have expected that to give you? EDIT read_sql only seems to execute one query at a time, and as the first query returns no rows it results in that exception.
If you split them in to two calls to read_sql then it will in fact return you the value of your #null value in the second call. Due to this behaviour read_sql is clearly not a good way to do this. I strongly suggest you use one of my suggestions below.
Why are you wanting to set the variable in the SQL using '#' anyway?
You could try using the .format style of string formatting.
Like so:
query_str = "select ... coalesce(field1, {c}), case(.. then {c})...".format(c=my_const)
pd.read_sql(query_str)
Just remember that if you do it this way and your my_const is a user input then you will need to sanitize it manually to prevent SQL injection.
Another possibility is using a dict of params like so:
query_str = "select ... coalesce(field1, %(my_const)s, case(.. then %(my_const)s)..."
pd.read_sql(query_str, params={'my_const': const_value})
However this is dependent on which database driver you use.
From the pandas.read_sql docs:
Check your database driver documentation for which of the five syntax
styles, described in PEP 249’s paramstyle, is supported. Eg. for
psycopg2, uses %(name)s so use params={‘name’ : ‘value’}
Insert query for string having \" character in it in mysql db-
How to write insert query for string such as:
This is the string i want to insert into my table,
reg="item-cell\"(.*?)</span></div>"
cur = db.cursor()
query='INSERT into table_name(col_name) values("%s")'%(reg)
cur.execute(query)
cur.close()
Below is the error:
_mysql_exceptions.ProgrammingError: (1064, 'You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near \'(.*?)</span></div>")\' at line 1')
I know its something related to escape character, but don't know how to make that work.
EDIT: This string reg is variable i.e. I am getting this string from some API and I want to insert it into my database. So inserting escape characters in between the string literal will not suffice my case. I want something that can generalize single quote, double quote or one double quote(eg. reg) all these cases.
I hope I made my point clear.
EDIT: This is how i am getting the value of reg(froma json file)
import urllib, json
import MySQLdb
url = "some_url"
response = urllib.urlopen(url)
data = json.loads(response.read())
for item in data["key1"]["key2"]["key3"]["key4"]:
prop=str(item)
reg=str(data["key1"]["key2"]["key3"]["key4"][prop]["regex"])
The problem is in the part where you convert the json object to string. To properly do this without altering the string you need to use json.dumps
for item in data["key1"]["key2"]["key3"]["key4"]:
prop = json.dumps(item)
reg = json.dumps(data["key1"]["key2"]["key3"]["key4"][prop]["regex"])
I have a Sqlite3 table
paste{
paste_id int,
paste_content text
}
i have to do an update statement, where text can possibly contain single ' quotes as well as "" double quotes.
In python i wrote
UPDATE_Statement = "Update paste set paste_content = '%s' where paste_id=id" %(content)
But since the content can contain ' or "" , my execute query is not working properly.
How can i escape this properly ?
Do not use string interpolation. Use SQL parameters instead:
UPDATE_Statement = "Update paste set paste_content = %s where paste_id=%s"
cursor.execute(UPDATE_Statement, content)
and leave escaping (and proper quoting) up to the database adapter instead. This:
Simplifies your code
Quotes different data types correctly
Lets the database reuse query plans for varying data
Prevents SQL injection attacks
See the Passing parameters into raw() in the Django SQL documentation.
If you are using a different database connector (not the connection provided by Django) verify the specific style of parameter placeholders in the documentation. The sqlite3 database adapter for example, uses ? as the placeholder syntax.