I am using python 2.7 and pymssql 1.9.908.
In .net to query the database I would do something like this:
using (SqlCommand com = new SqlCommand("select * from Customer where CustomerId = #CustomerId", connection))
{
com.Parameters.AddWithValue("#CustomerID", CustomerID);
//Do something with the command
}
I am trying to figure out what the equivalent is for python and more particularly pymssql. I realize that I could just do string formatting, however that doesn't seem handle escaping properly like a parameter does (I could be wrong on that).
How do I do this in python?
After creating a connection object db:
cursor = db.execute('SELECT * FROM Customer WHERE CustomerID = %s', [customer_id])
then use any of the fetch... methods of the resulting cursor object.
Don't be fooled by the %s part: this is NOT string formatting, it's parameter substitution (different DB API modules use different syntax for parameter substitution -- pymssql just happens to use the unfortunate %s!-).
Related
I have an existing solution in Java to extract data from an Oracle database. I do it like so:
String tmp = "begin ? := pkgioexportora.request(?); end;";
String xml = "<ttc.export.public.data.search><query><popid>1</popid> <moduleid>3</moduleid><only_changes>0</only_changes></query></ttc.export.public.data.search>";
CallableStatement callableStmt = oracle.prepareCall(tmp);
// Register the type of the out param - an Oracle specific type
callableStmt.registerOutParameter(1, OracleTypes.NUMBER);
callableStmt.setString(2, xml);
This construct returns a job id which I'm later required to use in the WHERE clause of a SELECT query.
I've tried using just a cursor and input the complete statement, without the CallableStatement stuff, but no luck.
cursor = con.cursor()
cursor.execute("begin 2 := pkgioexportora.request(xml_stuff_here); end;";)
callproc seems to give me similar error.
I've tried searching for solutions, or similar things done, but yet to come up with any examples. Is it possible to do such a thing with cx_Oracle or am I stuck with my Java code for doing this?
You can use cursor.callfunc() in cx_Oracle. As in the following:
result = cursor.callfunc("pkgioexportora.request", cx_Oracle.NUMBER, [xml_stuff])
You could also do it this way, but its more complex:
var = cursor.var(cx_Oracle.NUMBER)
cursor.execute("begin :1 := pkgioexportora.request(:2); end;", [var, xml_stuff])
result = var.getvalue()
I am aware that queries in Python can be parameterized using either ? or %s in execute query here or here
However I have some long query that would use some constant variable defined at the beginning of the query
Set #my_const = 'xyz';
select #my_const;
-- Query that use #my_const 40 times
select ... coalesce(field1, #my_const), case(.. then #my_const)...
I would like to do the least modif possible to the query from Mysql. So that instead of modifying the query to
pd.read_sql(select ... coalesce(field1, %s), case(.. then %s)... , [my_const, my_const, my_const, ..]
,I could write something along the line of the initial query. Upon trying the following, however, I am getting a TypeError: 'NoneType' object is not iterable
query_str = "Set #null_val = \'\'; "\
" select #null_val"
erpur_df = pd.read_sql(query_str, con = db)
Any idea how to use the original variable defined in Mysql query ?
The reason
query_str = "Set #null_val = \'\'; "\
" select #null_val"
erpur_df = pd.read_sql(query_str, con = db)
throws that exception is because all you are doing is setting null_value to '' and then selecting that '' - what exactly would you have expected that to give you? EDIT read_sql only seems to execute one query at a time, and as the first query returns no rows it results in that exception.
If you split them in to two calls to read_sql then it will in fact return you the value of your #null value in the second call. Due to this behaviour read_sql is clearly not a good way to do this. I strongly suggest you use one of my suggestions below.
Why are you wanting to set the variable in the SQL using '#' anyway?
You could try using the .format style of string formatting.
Like so:
query_str = "select ... coalesce(field1, {c}), case(.. then {c})...".format(c=my_const)
pd.read_sql(query_str)
Just remember that if you do it this way and your my_const is a user input then you will need to sanitize it manually to prevent SQL injection.
Another possibility is using a dict of params like so:
query_str = "select ... coalesce(field1, %(my_const)s, case(.. then %(my_const)s)..."
pd.read_sql(query_str, params={'my_const': const_value})
However this is dependent on which database driver you use.
From the pandas.read_sql docs:
Check your database driver documentation for which of the five syntax
styles, described in PEP 249’s paramstyle, is supported. Eg. for
psycopg2, uses %(name)s so use params={‘name’ : ‘value’}
I am working in python with a MySQL database. I have a table that uses the MySQL geometry extension, so I need to call the GeomFromText MySQL function during an update statement, something like this:
UPDATE myTable SET Location=GeomFromText('Point(39.0 55.0)') where id=1;
UPDATE myTable SET Location=GeomFromText('Point(39.0 55.0)') where id=2;
Originally, I was using the low-level MySQLdb library. I am switching to using the SQLAlchemy core library (I cannot use the SQLAlchemy ORM for speed and other reasons).
If I were using the lower-level MySQLdb library directly, I would do something like this:
import MySQLdb as mysql
commandTemplate = "UPDATE myTable SET Location=GeomFromText(%s) where id=%s"
connection = mysql.connect(host="myhost",user="user",passwd="password",db="my_schema")
cursor = connection.cursor(mysql.cursors.DictCursor)
data = [
("Point(39.0 55.0)",1),
("Point(39.0 55.0)",2),
]
cursor.executemany(commandTemplate,data)
How do I get the equivalent functionality with SQLAlchemy core?
Without the GeomFromText, I think it would look something like this (thanks to this answer):
from sqlalchemy.sql.expression import bindparam
updateCommand = myTable.update().where(id=bindparam("idToChange"))
data = [
{'idToChange':1,'Location':"Point(39.0 55.0)"},
{'idToChange':2,'Location':"Point(39.0 55.0)"},
]
connection.execute(updateCommand,data)
I can't just textually replace "Point(39.0 55.0)" with "GeomFromText('Point(39.0 55.0)')", or I get:
Cannot get geometry object from data you send to the GEOMETRY field
The easiest way I have found so far involves the use of text (i.e. constructing TextClause objects), which lets you enter SQL syntax (almost) literally.
My example would work something like this:
from sqlalchemy.sql.expression import bindparam
from sqlalchemy import text
updateCommand = myTable.update().where(id=bindparam("idToChange"))
valuesDict = {'idToChange':':idToChange',
'Location':text("GeomFromText(:_location)")
}
updateCommand = updateCommand.values(**valuesDict)
data = [
{'idToChange':1,'_location':"Point(39.0 55.0)"},
{'idToChange':2,'_location':"Point(39.0 55.0)"},
]
#see the MySQL command as it will be executed (except for data)
print(connection.compile(bind=connection))
#actually execute the statement
connection.execute(updateCommand,data)
The key points:
calling updateCommand.values replaces the VALUES part of the SQL clause. Only the columns that you give as kwargs to this call will actually be put into the final UPDATE statement
the values of the keyword arguments to updateCommand.values can either be a literal set of data (if you are only updating one row), or it can be a string giving the names of keys in the data dictionary that will eventually be passed with the command to the connection.execute method. The format to use is ColumnName=":dictionaryKeyName".
the values of the keyword arguments can also be the result of a text clause, which can itself contain field names in the same ":dictionaryKeyName" format.
I am using mysql.connector python library with python 2.7
I have a unicode string which may or may not contain single and double quotes.
Here are the things I tried for my escape function:
def escape(string):
#string.MySQL.escape_string()
#string = string.decode('string_escape')
#string = string.encode('unicode-escape').replace("'", "''")
#string = string.encode('unicode-escape').replace('"', '\"')
#string = string.encode('unicode-escape').replace("'", u"\u2019")
#string = string.encode('unicode-escape').replace('''"''', u"\u201D")
#string = string.encode('unicode-escape').replace('''''', u"\u201D")
return string
Nothing seems to have worked. I tried using this function but still gives mysql syntax error.
I need something within mysql.connector library which escapes the single and double quotes without breaking the unicode as well as mysql query.
Here is an example of SQL query I am using:
"""SELECT * FROM messages WHERE msg_id = '{msg_id}'""".format(**db_dict)
Let me know if any more details needed
EDIT: Example SQL query updated
MySQLdb officially declares to use the format paramstyle, but it also supports the pyformat style*, so if you want to use parameters from a dict, you can use:
db_dict = {'msg_id': "1'2'3", ...}
cursor.execute("SELECT * FROM messages WHERE msg_id = %(msg_id)s", db_dict)
Using string manipulation to create sql queries only leads to sql injection vulnerabilities, so you should never do it.
*... most db connectors that use python string formatting behind the screen do the same, they specify one of format or pyformat as paramstyle but actually support both. The dbapi2 doesnt't allow to specify two values here, but it doesn't forbid to support multiple parmstyles either. If you write code that potentially uses an unknowon dbapi2 connector it's enough that you can query a supported paramstyle, being able to know all would be nice but it's not necessary.
cursor.execute('SELECT * FROM messages WHERE msg_id = %s', (db_dict['msg_id'],)) is what you want to run here. Standard string escapes aren't supported by python's database interface, and, per #bobince's comment, are a security hole to boot.
I have a python code, in which I make SQL requests in a database. I would like to be able to switch between a postgresql (using module psycopg2) database and a sqlite one (using module sqlite3), without need of adapting my code. This means, I would like to have in my code some fixed SQL request strings, and I want to switch between the engine, only changing the definition of the database connector object, using one of those:
my_db = psycopg2.connect(...)
my_db = sqlite3.connect(...)
For the moment, I don't see any possibilty since:
Everyone knows that one should NOT use string concatenation to pass arguments to a SQL request, but rather use placeholders (from psycopg2 docu :never, NEVER use Python string concatenation ... to pass variables to a SQL query string. Not even at gunpoint. )
The synthax for placeholders are different is the 2 APIs psycopg2 and sqlite3. Even for NON-named placeholders. Psycopg uses "%" and sqlite3 uses "?":
my_cursor.execute("SELECT * FROM table WHERE id= ?", (my_id,)) # for SQLITE3
my_cursor.execute("SELECT * FROM table WHERE id= %", (my_id,)) # for PSYCOPG2
One could in principle use the SQL built-in placeholder synthax ("?"
for postgresql), but this would mean precisely preparing a SQL-string with python string concatenation, and so on... that is forbidden by 1.
I'm lacking ideas...