Insert query for string having \" character in it in mysql db-
How to write insert query for string such as:
This is the string i want to insert into my table,
reg="item-cell\"(.*?)</span></div>"
cur = db.cursor()
query='INSERT into table_name(col_name) values("%s")'%(reg)
cur.execute(query)
cur.close()
Below is the error:
_mysql_exceptions.ProgrammingError: (1064, 'You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near \'(.*?)</span></div>")\' at line 1')
I know its something related to escape character, but don't know how to make that work.
EDIT: This string reg is variable i.e. I am getting this string from some API and I want to insert it into my database. So inserting escape characters in between the string literal will not suffice my case. I want something that can generalize single quote, double quote or one double quote(eg. reg) all these cases.
I hope I made my point clear.
EDIT: This is how i am getting the value of reg(froma json file)
import urllib, json
import MySQLdb
url = "some_url"
response = urllib.urlopen(url)
data = json.loads(response.read())
for item in data["key1"]["key2"]["key3"]["key4"]:
prop=str(item)
reg=str(data["key1"]["key2"]["key3"]["key4"][prop]["regex"])
The problem is in the part where you convert the json object to string. To properly do this without altering the string you need to use json.dumps
for item in data["key1"]["key2"]["key3"]["key4"]:
prop = json.dumps(item)
reg = json.dumps(data["key1"]["key2"]["key3"]["key4"][prop]["regex"])
Related
I set queries like following . I'd like to replace bucket_name and file_name with variables
This query is executed by psycopg2
query = '''copy table.test
from 's3://%(bucket_name)s/%(file_name)s.txt'
iam_role 'arn:aws:iam::123453215125:role/test'
timeformat as 'auto'
ACCEPTINVCHARS
delimiter '\t';'''
And I get bucket name and file name
bucket_name = get_bucket_name(event)
file_name = get_file_name(event)
After that, I executed query, but it returned errors
cur.execute(query, {'bucket_name':bucket_name,'file_name':file_name})
[ERROR] SyntaxError: syntax error at or near "expired" LINE 2: from 's3://'expired-test-bucket... ^ Traceback (most recent cal
It seems that bucket_name is replaced with single quotation 'expired-test-bucket'
My desired result is expired-test-bucket
How can I fix this?
Parameters in parameterized queries do not work like string interpolation, even if you use pyformat binding style that happens to look like old-school %-based string interpolation. The quotes are there because the database engine is expecting you to use the placeholder to represent an entire parameter, and putting quotes (and potentially various forms of escaping) into the final query text by design, specifically to prevent the security hole that results from interpolating user data directly into a query.
If you need to use user data to build up a value that is used in a query, then do just that, as separate steps: first, use Python string formatting to create the raw underlying value, and then use the SQL engine's functionality to make it safe to put that value into a query.
So, something like:
bucket_name = get_bucket_name(event)
file_name = get_file_name(event)
url = f's3://{bucket_name}/{file_name}.txt'
query = '''copy table.test
from %s
iam_role 'arn:aws:iam::123453215125:role/test'
timeformat as 'auto'
ACCEPTINVCHARS
delimiter '\t';'''
cur.execute(query, (url,))
I always failled to insert data to Mysql database from my telegram bot, and always run Exception. Only tanggald allways failed to insert. I thing format of date insert query is wrong. How to write correct format?
tanggald column detail : Data Type = DATE
This is piece of code:
def process_lanjut(message):
try:
chat_id = message.chat.id
qlanjut = message.text
user = user_dict[chat_id]
user.qlanjut=qlanjut
d = datetime.datetime.now().date()
next_monday = next_weekday(d, 4)
user.next_monday = next_monday
print(user.next_monday)
with con.cursor() as cursor:
sql = "INSERT INTO diagnosa(sex, tanggald) VALUES('" + user.sex + "','" +next_monday+ "')"
cursor.execute(sql)
con.commit()
con.close()
msg = bot.send_message(chat_id, 'thanks')
bot.register_next_step_handler(msg, send_end)
except Exception as e:
bot.reply_to(message,'oops lanjut')
On command line output : 2018-04-20 (data that should be INSERT to tanggald)
That's not the proper way to insert data into table. Although your way may work, it is not safe and it lacks data escaping (', ", etc.):
The SQL representation of many data types is often different from their Python string representation. The typical example is with single quotes in strings: in SQL single quotes are used as string literal delimiters, so the ones appearing inside the string itself must be escaped, whereas in Python single quotes can be left unescaped if the string is delimited by double quotes.
Because of the difference, sometime subtle, between the data types representations, a naïve approach to query strings composition, such as using Python strings concatenation, is a recipe for terrible problems.
If the variables containing the data to send to the database come from an untrusted source (such as a form published on a web site) an attacker could easily craft a malformed string, either gaining access to unauthorized data or performing destructive operations on the database. This form of attack is called SQL injection and is known to be one of the most widespread forms of attack to database servers. Before continuing, please print this page as a memo and hang it onto your desk.
So in your case the INSERT statement should look like this:
with con.cursor() as cursor:
sql = 'INSERT INTO diagnosa (sex, tanggald) VALUES (%s, %s)'
data = (user.sex, next_monday)
cursor.execute(sql, data)
Further reading:
MySQL docs and example
Similar article for PostgreSQL (contains a better explanation of potential problems)
I'm using PostgreSQL v9.4.12 and I'm trying to update a jsonb column. I want to update the whole json object and not a specific key of the object.
I'm using a Python dict to store my object and before using it I'm using json.dumps() to transform it to a json formatted String.
However, a value of the json is having a single quote ' that throws an psycopg2.ProgrammingError: syntax error while trying to update.
So far, I've tried:
"UPDATE table "
"SET jsonb_column='{} ".format(json.dumps(new_data)) + ""
"WHERE id='12345'"
Note that new_data is my dict and jsonb_column is the name of the column holding the json data.
The error I'm getting:
psycopg2.ProgrammingError: syntax error at or near "s"
LINE 1: ...code": "BR3", "short_description": "This property's price
is...
^
I was assuming that json.dumps() escapes the single quote but doesn't seem that to be the case. Is there any solution to overcome this error?
Thanks in advance.
json is very fine with single quote, eg:
t=# select $${"short_description": "This property's price is..."}$$::jsonb;
jsonb
------------------------------------------------------
{"short_description": "This property's price is..."}
(1 row)
so I assume you could try using dollar sign quotes, to avoid statement structuring exception with single quotes
The practice of string concatenation is not a good practice.
Better use the way documented in PsyCoPg2 docs.
cur.execute("UPDATE table SET jsonb_column = %s WHERE id = %s", [json, id])
I am trying to insert a string to a SQL DB starting with 0x but it keeps failing on the insert. The characters that come after 0x are random characters that range from A-Z, a-z and 0-9 with no set length. I tried to get around it by adding a letter in front of the string and update it afterwards but it does not work. I am using
SQL statement I am trying to mimic
insert into [TestDB].[dbo].[S3_Files] ([Key],[IsLatest],[LastModified],[MarkedForDelete],[VersionID]) values ('pmtg-dox/CCM/Trades/Buy/Seller_Provided_-_Raw_Data/C''Ds_v2/NID3153422.pdf','1','2015-10-11','Yes', '0xih91kjhdaoi23ojsdpf')
Python Code
import pymssql as mssql
...
cursor.execute("insert into [TestDB].[dbo].[S3_Files] ([Key],[IsLatest],[LastModified],[MarkedForDelete],[VersionID]) values (%s,%s,%s,%s,%s)",(deleteitems['Key'],deleteitems['IsLatest'],deleteitems['LastModified'],MarkedforDelete, deleteitems['VersionId']))
conn_db.commit()
pymssql.ProgrammingError: (102, "Incorrect syntax near
'qb_QWQDrabGr7FTBREfhCLMZLw4ztx'.DB-Lib error message 20018, severity
15: General SQL Server error: Check messages from the SQL Server")
Is there a way to make Python, pymssql\mysql force insert the string? Is there a string manipulation technique that I am not using? I have tried pypyodbc but no luck.
Edit: My current patch is to alter the string and add a flag to the row so I remember that the string starts with 0x
This is the solution that I came up with.
Since running the insert command with the appropriate values worked, I created a stored procedure in SQL to handle my request
USE [TestDB]
GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
ALTER PROCEDURE [dbo].[stp_S3Files]
#Key_ varchar(4000),
#IsLatest_ varchar(200),
#Size_ varchar(200),
#LastModified_ varchar(200),
#MarkedForDelete_ varchar(200),
#VersionID_ varchar(200)
AS
insert into [TestDB].[dbo].[S3_Files] ([Key],[IsLatest],[Size(Bytes)],[LastModified],[MarkedForDelete],[VersionID]) values (#Key_, #IsLatest_, #Size_, #LastModified_, #MarkedForDelete_, #VersionID_)
Then I call it through Python
modkey = deleteitems['Key'].replace("'", "''")
cursor.execute("""exec TestDB.dbo.stp_S3Files
#Key_ = '%s'
,#IsLatest_ = %s
,#Size_ = '%s'
,#LastModified_ = '%s'
,#MarkedForDelete_ = '%s'
,#VersionID_ = '%s' """ %(modkey, deleteitems['IsLatest'],deleteitems['Size'],deleteitems['LastModified'],MarkedforDelete,deleteitems['VersionId']))
conn_db.commit()
Note: the string replace is to handle path names with ' to escape the character. I hope this helps someone who has the same issue down the road.
So here is my problem: I am trying to select a specific value from a table
comparing it with a unicode string. The value is also unicode. I am using
mysql.connector. The server settings are all utf8 oriented. When I run
following query - I get an empty list. When I run it without 'WHERE Title like '%s'' part, I get a full set of values, and they properly displayed in the
output. The same query works in the command line on the server. The value is
there for sure. What is it that I am missing?
conn = sql.connect(host='xxxxxxx', user='xxx', password='xxx', database='db', charset="utf8")
cur = conn.cursor()
townQuery = (u"""SELECT * FROM Towns WHERE Title like '%s' """)
tqd = (u"%" +u"Серов"+u"%")
cur.execute(townQuery, tqd)
for i in cur:
print i
When you use the 2-argument form of cur.execute (thus passing the arguments, tqd, to the parametrized sql, townQuery), the DB adaptor will quote the arguments for you. Therefore, remove the single quotes from around the %s in townQuery:
townQuery = u"""SELECT * FROM Towns WHERE Title like %s"""
tqd = [u"%Серов%"]
cur.execute(townQuery, tqd)
Also note that the second argument, tqd, must be a sequence such as a list or tuple. The square brackets around u"%Серов%" makes [u"%Серов%"] a list. Parentheses around u"%Серов%" do NOT make (u"%Серов%") a tuple because Python evaluates the quantity in parentheses to a unicode. To make it a tuple, add a comma before the closing parenthesis: (u"%Серов%",).