I am trying to insert a string to a SQL DB starting with 0x but it keeps failing on the insert. The characters that come after 0x are random characters that range from A-Z, a-z and 0-9 with no set length. I tried to get around it by adding a letter in front of the string and update it afterwards but it does not work. I am using
SQL statement I am trying to mimic
insert into [TestDB].[dbo].[S3_Files] ([Key],[IsLatest],[LastModified],[MarkedForDelete],[VersionID]) values ('pmtg-dox/CCM/Trades/Buy/Seller_Provided_-_Raw_Data/C''Ds_v2/NID3153422.pdf','1','2015-10-11','Yes', '0xih91kjhdaoi23ojsdpf')
Python Code
import pymssql as mssql
...
cursor.execute("insert into [TestDB].[dbo].[S3_Files] ([Key],[IsLatest],[LastModified],[MarkedForDelete],[VersionID]) values (%s,%s,%s,%s,%s)",(deleteitems['Key'],deleteitems['IsLatest'],deleteitems['LastModified'],MarkedforDelete, deleteitems['VersionId']))
conn_db.commit()
pymssql.ProgrammingError: (102, "Incorrect syntax near
'qb_QWQDrabGr7FTBREfhCLMZLw4ztx'.DB-Lib error message 20018, severity
15: General SQL Server error: Check messages from the SQL Server")
Is there a way to make Python, pymssql\mysql force insert the string? Is there a string manipulation technique that I am not using? I have tried pypyodbc but no luck.
Edit: My current patch is to alter the string and add a flag to the row so I remember that the string starts with 0x
This is the solution that I came up with.
Since running the insert command with the appropriate values worked, I created a stored procedure in SQL to handle my request
USE [TestDB]
GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
ALTER PROCEDURE [dbo].[stp_S3Files]
#Key_ varchar(4000),
#IsLatest_ varchar(200),
#Size_ varchar(200),
#LastModified_ varchar(200),
#MarkedForDelete_ varchar(200),
#VersionID_ varchar(200)
AS
insert into [TestDB].[dbo].[S3_Files] ([Key],[IsLatest],[Size(Bytes)],[LastModified],[MarkedForDelete],[VersionID]) values (#Key_, #IsLatest_, #Size_, #LastModified_, #MarkedForDelete_, #VersionID_)
Then I call it through Python
modkey = deleteitems['Key'].replace("'", "''")
cursor.execute("""exec TestDB.dbo.stp_S3Files
#Key_ = '%s'
,#IsLatest_ = %s
,#Size_ = '%s'
,#LastModified_ = '%s'
,#MarkedForDelete_ = '%s'
,#VersionID_ = '%s' """ %(modkey, deleteitems['IsLatest'],deleteitems['Size'],deleteitems['LastModified'],MarkedforDelete,deleteitems['VersionId']))
conn_db.commit()
Note: the string replace is to handle path names with ' to escape the character. I hope this helps someone who has the same issue down the road.
Related
connection to postgresql database has been connected successfully.but while executing below query i am getting some kind of error which looks like :
column "e" of relation "analysis_result" does not exist
LINE 1: INSERT INTO analysis_result(user_id,E,A,C,N,O,total) VALUES ...
cursor = connection.cursor()
print("inside execution ")
cursor.execute("INSERT INTO analysis_result(user_id,E,A,C,N,O,total) VALUES (%s,%s,%s,%s,%s,%s,%s)",Result_lst)
i understand what is the error .i have to put each of E A C N O in quotes but while quoting them my sql query becomes invalid .Please give me a solution i am scraching my head for quite sometime now.
and Result_lst=[1,20,14,14,38,8]. Result_lst will be dyanamic values in integer forms.
If the column name contains upper case characters, you must surround it with double quotes, otherwise PostgreSQL will fold it to lower case (note the error message).
But you cannot simply use double quotes because that's what you used for the Python string.
Either use single quotes with the Python string:
cursor.execute('INSERT INTO analysis_result(user_id,"E","A","C","N","O",total) ...', Result_lst)
or escape the double quotes:
cursor.execute("INSERT INTO analysis_result(user_id,\"E\",\"A\",\"C\",\"N\",\"O\",total) ...", Result_lst)
I'm working on a bit of python code to run a query against a redshift (postgres) SQL database, and I'm running into an issue where I can't strip off the surrounding single quotes from a variable I'm passing to the query. I'm trying to drop a number of tables from a list. This is the basics of my code:
def func(table_list):
drop_query = 'drop table if exists %s' #loaded from file
table_name = table_list[0] #table_name = 'my_db.my_table'
con=psycopg2.connect(dbname=DB, host=HOST, port=PORT, user=USER, password=PASS)
cur=con.cursor()
cur.execute(drop_query, (table_name, )) #this line is giving me trouble
#cleanup statements for the connection
table_list = ['my_db.my_table']
when func() gets called, I am given the following error:
syntax error at or near "'my_db.my_table'"
LINE 1: drop table if exists 'my_db.my_table...
^
Is there a way I can remove the surrounding single quotes from my list item?
for the time being, I've done it (what think is) the wrong way and used string concatenation, but know this is basically begging for SQL-injection.
This is not how psycopg2 works. You are using a string operator %s to replace with a string. The reason for this is to tokenize your string safely to avoid SQL injection, psycopg2 handles the rest.
You need to modify the query before it gets to the execute statement.
drop_query = 'drop table if exists {}'.format(table_name)
I warn you however, do not allow these table names to be create by outside sources, or you risk SQL injection.
However a new version of PSYCOPG2 kind of allows something similar
http://initd.org/psycopg/docs/sql.html#module-psycopg2.sql
from psycopg2 import sql
cur.execute(
sql.SQL("insert into {} values (%s, %s)").format(sql.Identifier('my_table')),[10, 20]
)
MySQLdb is a module of python to communicate with mysql database. The escape_string is a method provided by MySQLdb to escape some characters in sql. For example, sql like 'Update table Set col = "My"s"' will cause a error. So escape_string will help us to add a '\' before the " in My"s.
However, in multibyte encoding like gbk, which use more than 2 bytes to store a chinese word, the escape_string only search the character to be escaped one character by one, which will cause some special characters to be escaped incorrectly. for example, the Chinese character ' 昞', whose bytes are '\x95\x5c', if the sql is 'update table set col = "昞"', then the MySQLdb.escape_string(sql) will get the result: update table set col = "昞\", which is wrong and cannot be executed correctly.
So is there anyone who ever came over such a problem.
P.S I googled the problem and found there is a method mysqli_set_charset in php which can solve such case, So, I wonder whether there is a such one in python.
This problem is most likely cause because the default character set for your connection is latin1 instead of unicode. There are a couple different things you can try. From this post,
conn = mysql.connect(host='127.0.0.1',
user='user',
passwd='passwd',
db='db',
charset='utf8',
use_unicode=True)
then you run your query like this
cursor.execute('INSERT INTO mytable VALUES (null, %s)',
('\x95\x5c',))
Appearently a similar problem was solved by running the following query first
SET NAMES 'gbk
I'm working on a project that requires me to programmatically create MySQL users from a django app. I can create the users just fine:
from django.db import connection, transaction
cursor = connection.cursor()
cursor.execute("CREATE USER %s#'%'", 'username')
cursor.execute("SET PASSWORD FOR %s#'%' = PASSWORD(%s)", ('username', 'pass'))
That works perfectly. The problem is when I try to grant permissions. The database name is also determined programmatically:
cursor.execute("GRANT SELECT ON %s.* TO %s#'%'", ('dbname', 'username'))
This results in a mysql error because when it does the string substitution, it places single quotes around the database name, which is syntactically incorrect:
DatabaseError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ''dbname'.* to 'username'#'%'' at line 1")
How do I prevent the single quotes from being added around the %s for database name? I know that I could simply do the string substitution in Python and fix this, but that could potentially cause a SQL injection vulnerability.
Sometimes placeholders won't work (as you've found out), so you'll have to use string concatenation. Be careful - validate the string, make sure it's only composed of the characters you expect (don't just look for characters you don't expect), and you should be OK. Also get another developer to check your code, and comment it to make sure no-one else thinks you ought to be using placeholders.
Im using python to access a MySQL database and im getting a unknown column in field due to quotes not being around the variable.
code below:
cur = x.cnx.cursor()
cur.execute('insert into tempPDBcode (PDBcode) values (%s);' % (s))
rows = cur.fetchall()
How do i manually insert double or single quotes around the value of s?
I've trying using str() and manually concatenating quotes around s but it still doesn't work.
The sql statement works fine iv double and triple check my sql query.
You shouldn't use Python's string functions to build the SQL statement. You run the risk of leaving an SQL injection vulnerability. You should do this instead:
cur.execute('insert into tempPDBcode (PDBcode) values (%s);', s)
Note the comma.
Python will do this for you automatically, if you use the database API:
cur = x.cnx.cursor()
cur.execute('insert into tempPDBcode (PDBcode) values (%s)',s)
Using the DB API means that python will figure out whether to use quotes or not, and also means that you don't have to worry about SQL-injection attacks, in case your s variable happens to contain, say,
value'); drop database; '
If this were purely a string-handling question, the answer would be tojust put them in the string:
cur.execute('insert into tempPDBcode (PDBcode) values ("%s");' % (s))
That's the classic use case for why Python supports both kinds of quotes.
However as other answers & comments have pointed out, there are SQL-specific concerns that are relevant in this case.