We Keep Coding

Convert Java to Python using cx_Oracle SQL/XML query

I have an existing solution in Java to extract data from an Oracle database. I do it like so:
String tmp = "begin ? := pkgioexportora.request(?); end;";
String xml = "<ttc.export.public.data.search><query><popid>1</popid> <moduleid>3</moduleid><only_changes>0</only_changes></query></ttc.export.public.data.search>";
CallableStatement callableStmt = oracle.prepareCall(tmp);
// Register the type of the out param - an Oracle specific type
callableStmt.registerOutParameter(1, OracleTypes.NUMBER);
callableStmt.setString(2, xml);
This construct returns a job id which I'm later required to use in the WHERE clause of a SELECT query.
I've tried using just a cursor and input the complete statement, without the CallableStatement stuff, but no luck.
cursor = con.cursor()
cursor.execute("begin 2 := pkgioexportora.request(xml_stuff_here); end;";)
callproc seems to give me similar error.
I've tried searching for solutions, or similar things done, but yet to come up with any examples. Is it possible to do such a thing with cx_Oracle or am I stuck with my Java code for doing this?

You can use cursor.callfunc() in cx_Oracle. As in the following:
result = cursor.callfunc("pkgioexportora.request", cx_Oracle.NUMBER, [xml_stuff])
You could also do it this way, but its more complex:
var = cursor.var(cx_Oracle.NUMBER)
cursor.execute("begin :1 := pkgioexportora.request(:2); end;", [var, xml_stuff])
result = var.getvalue()

MySql read_sql python query with variable #

I am aware that queries in Python can be parameterized using either ? or %s in execute query here or here
However I have some long query that would use some constant variable defined at the beginning of the query
Set #my_const = 'xyz';
select #my_const;
-- Query that use #my_const 40 times
select ... coalesce(field1, #my_const), case(.. then #my_const)...
I would like to do the least modif possible to the query from Mysql. So that instead of modifying the query to
pd.read_sql(select ... coalesce(field1, %s), case(.. then %s)... , [my_const, my_const, my_const, ..]
,I could write something along the line of the initial query. Upon trying the following, however, I am getting a TypeError: 'NoneType' object is not iterable
query_str = "Set #null_val = \'\'; "\
" select #null_val"
erpur_df = pd.read_sql(query_str, con = db)
Any idea how to use the original variable defined in Mysql query ?

The reason
query_str = "Set #null_val = \'\'; "\
" select #null_val"
erpur_df = pd.read_sql(query_str, con = db)
throws that exception is because all you are doing is setting null_value to '' and then selecting that '' - what exactly would you have expected that to give you? EDIT read_sql only seems to execute one query at a time, and as the first query returns no rows it results in that exception.
If you split them in to two calls to read_sql then it will in fact return you the value of your #null value in the second call. Due to this behaviour read_sql is clearly not a good way to do this. I strongly suggest you use one of my suggestions below.
Why are you wanting to set the variable in the SQL using '#' anyway?
You could try using the .format style of string formatting.
Like so:
query_str = "select ... coalesce(field1, {c}), case(.. then {c})...".format(c=my_const)
pd.read_sql(query_str)
Just remember that if you do it this way and your my_const is a user input then you will need to sanitize it manually to prevent SQL injection.
Another possibility is using a dict of params like so:
query_str = "select ... coalesce(field1, %(my_const)s, case(.. then %(my_const)s)..."
pd.read_sql(query_str, params={'my_const': const_value})
However this is dependent on which database driver you use.
From the pandas.read_sql docs:
Check your database driver documentation for which of the five syntax
styles, described in PEP 249’s paramstyle, is supported. Eg. for
psycopg2, uses %(name)s so use params={‘name’ : ‘value’}

Python: Calling queries from a file with variables

I have a few queries in my python script which also use variables as parameters to retrieve data.
'''SELECT * FROM order_projections_daily WHERE days >= ''' + each + '''AND days < ''' + next_monday
How can I store queries like this in a separate file and call it directly from there rather than cramming them in the code?
I have tried storing the queries in a file and calling them as a string but it doesn't work with variables. it works with:
'''SELECT * FROM order_projections_daily'''
This is a very simple query, I am using much more complicated queries in the real case.

Use parameterised strings—
'''SELECT * FROM order_projections_daily WHERE days >= %(start)s AND days < %(end)s'''
Later, when executing the query, build a params dict like this:
params = {'start': ..., 'end': ...}
These params should then be passed to the DBC driver function which will take care of inserting the parameters:
cursor.execute(query, params)
Note: Do not inject format strings into your query, you're liable to SQL injection that way.

Use python string formatting.
In your separate file save query as below e.g:
query = "select * from my_persons where name = {name} and age = {age}"
In python file, format the query this way:
sql = query.format(name="jack", age=27)
You can save all your queries in a separate python file as string values and import all of them into your code anywhere. In my eg I am assuming query is saved in a separate python file.
Formatting your query
query='''SELECT * FROM order_projections_daily WHERE days >={each} AND days < {next_monday}
Format it as:
sql = query.format(each=each, next_monday=next_monday)
It's a good practice to use format method or % or maybe even join rather than using string concatenation as string concatenation creates intermediate string objects.
Ok so formatting is a bad idea then, have a look at this. Name binding in oracle https://stackoverflow.com/a/33882805/2196290

In Python entering variables(dates) from raw_input into postgreSQL SELECT...WHERE

I would not call myself a newbie, but I am not terribly conversant with programming. Any help would be appreciated. I have this project that is almost done. Figured out lots of stuff, but this issue has me at a loss.
Is there a simple way to insert an acceptable date value in a postgresql query from:
start_date = raw_input('Start date: ')
end_date = raw_input('End date: ')
I want the variables above to work in the following.
WHERE (gltx.post_date > start_date AND gltx.post_date < end_date )
'YYYY-MM-DD' format works in the SELECT Query of the postgresql database through python triple quoted cursor.execute.
The postgresql column(post.date) is date format.
here is the header for the python script.
#!/usr/bin/python
import psycopg2 as dbapi2
import psycopg2.extras
import sys
import csv
For now I have been altering the query for different periods of time.
Also is there an easy way format the date returned as YYYYMMDD. Perhaps a filter that replaced dashes or hyphens with nothing. I could use that for phone numbers also.

If you are going to execute this SELECT inside a Python script, you should not be placing strings straight into your database query - else you run the risk of SQL injections. See the psycopg2 docs - the problem with query parameters.
Instead you need to use placeholders and place all your string arguments into an iterable (usually a tuple) which is passed as the second argument to cursor.execute(). Again see the docs -passing parameters to sql queries.
So you would create a cursor object, and call the execute() method passing the query string as the first argument and a tuple containing the two dates as the second. Eg
query = "SELECT to_char(gltx.post_date, 'YYYYMMDD') FROM gltx WHERE (gltx.post_date > %s AND gltx.post_date < %s)"
args = (start_date, end_date)
cursor.execute(query, args)
To format the date in Python space, you can use the strftime() method on a date object. You should probably be working with datetime objects not strings anyway, if you want to do anything more than print the output.
You also probably want to validate that the date entered into the raw_input() is a valid date too.

Use the cursor.execute method's parameter substitution
import psycopg2
query = """
select to_char(gltx.post_date, 'YYYYMMDD') as post_date
from gltx
where gltx.post_date > %s AND gltx.post_date < %s
;"""
start_date = '2014-02-17'
end_date = '2014-03-04'
conn = psycopg2.connect("dbname=cpn")
cur = conn.cursor()
cur.execute(query, (start_date, end_date))
rs = cur.fetchall()
conn.close()
print rs

How do I use SQL parameters with python?

I am using python 2.7 and pymssql 1.9.908.
In .net to query the database I would do something like this:
using (SqlCommand com = new SqlCommand("select * from Customer where CustomerId = #CustomerId", connection))
{
com.Parameters.AddWithValue("#CustomerID", CustomerID);
//Do something with the command
}
I am trying to figure out what the equivalent is for python and more particularly pymssql. I realize that I could just do string formatting, however that doesn't seem handle escaping properly like a parameter does (I could be wrong on that).
How do I do this in python?

After creating a connection object db:
cursor = db.execute('SELECT * FROM Customer WHERE CustomerID = %s', [customer_id])
then use any of the fetch... methods of the resulting cursor object.
Don't be fooled by the %s part: this is NOT string formatting, it's parameter substitution (different DB API modules use different syntax for parameter substitution -- pymssql just happens to use the unfortunate %s!-).