Imagine this is my query :
query = '''
SELECT *
FROM table
WHERE id = {{myid}}'''
params = {'myid':3}
j= JinJaSql(param_style='pyformat'}
myquery, bind_params = j.prepare_query(query,params)
when I print bind_params I would get
{'myid_1':3}
why my parameter name was changed to myid_1 while I named it myid. Is there anything wrong with my code? How can I fix it?
According to the readme on the JinjaSql github page, if using the "pyformat" or "named" param style, the bound parameters returned by prepare_query are guaranteed to be unique. I suspect that this is why myid gets changed to myid_1.
When calling read_sql you should make sure to use the myquery and bind_params returned by prepare_query and not the query and params used in the prepare_query call, i.e.
# ...
# your code above
# ...
myquery, bind_params = j.prepare_query(query, params)
result = pd.read_sql(myquery, conn, params=bind_params)
Note that the format of your params for pd.read_sql depends on the SQL dialect you're using (docs). If you're using postgres for example, dictionary params are the way to go.
Hope this helps!
Related
Here is some custom code I wrote that I think might be problematic for this particular use case.
class SQLServerConnection:
def __init__(self, database):
...
self.connection_string = \
"DRIVER=" + str(self.driver) + ";" + \
"SERVER=" + str(self.server) + ";" + \
"DATABASE=" + str(self.database) + ";" + \
"Trusted_Connection=yes;"
self.engine = sqlalchemy.create_engine(
sqlalchemy.engine.URL.create(
"mssql+pyodbc", \
query={'odbc_connect': self.connection_string}
)
)
# Runs a command and returns in plain text (python list for multiple rows)
# Can be a select, alter table, anything like that
def execute(self, command, params=False):
# Make a connection object with the server
with self.engine.connect() as conn:
# Can send some parameters along with a plain text query...
# could be single dict or list of dict
# Doc: https://docs.sqlalchemy.org/en/14/tutorial/dbapi_transactions.html#sending-multiple-parameters
if params:
output = conn.execute(sqlalchemy.text(command,params))
else:
output = conn.execute(sqlalchemy.text(command))
# Tell SQL server to save your changes (assuming that is applicable, is not with select)
# Doc: https://docs.sqlalchemy.org/en/14/tutorial/dbapi_transactions.html#committing-changes
try:
conn.commit()
except Exception as e:
#pass
warn("Could not commit changes...\n" + str(e))
# Try to consolidate select statement result into single object to return
try:
output = output.all()
except:
pass
return output
If I try:
cnxn = SQLServerConnection(database='MyDatabase')
cnxn.execute("SELECT * INTO [dbo].[MyTable_newdata] FROM [dbo].[MyTable] ")
or
cnxn.execute("SELECT TOP 0 * INTO [dbo].[MyTable_newdata] FROM [dbo].[MyTable] ")
Python returns this object without error, <sqlalchemy.engine.cursor.LegacyCursorResult at 0x2b793d71880>, but upon looking in MS SQL Server, the new table was not generated. I am not warned about the commit step failing with the SELECT TOP 0 way; I am warned ('Connection' object has no attribute 'commit') in the above way.
CREATE TABLE, ALTER TABLE, or SELECT (etc) appears to work fine, but SELECT * INTO seems to not be working, and I'm not sure how to troubleshoot further. Copy-pasting the query into SQL Server and running appears to work fine.
As noted in the introduction to the 1.4 tutorial here:
A Note on the Future
This tutorial describes a new API that’s released in SQLAlchemy 1.4 known as 2.0 style. The purpose of the 2.0-style API is to provide forwards compatibility with SQLAlchemy 2.0, which is planned as the next generation of SQLAlchemy.
In order to provide the full 2.0 API, a new flag called future will be used, which will be seen as the tutorial describes the Engine and Session objects. These flags fully enable 2.0-compatibility mode and allow the code in the tutorial to proceed fully. When using the future flag with the create_engine() function, the object returned is a subclass of sqlalchemy.engine.Engine described as sqlalchemy.future.Engine. This tutorial will be referring to sqlalchemy.future.Engine.
That is, it is assumed that the engine is created with
engine = create_engine(connection_url, future=True)
You are getting the "'Connection' object has no attribute 'commit'" error because you are creating an old-style Engine object.
You can avoid the error by adding future=True to your create_engine() call:
self.engine = sqlalchemy.create_engine(
sqlalchemy.engine.URL.create(
"mssql+pyodbc",
query={'odbc_connect': self.connection_string}
),
future=True
)
Use this recipe instead:
#!python
from sqlalchemy.sql import Select
from sqlalchemy.ext.compiler import compiles
class SelectInto(Select):
def __init__(self, columns, into, *arg, **kw):
super(SelectInto, self).__init__(columns, *arg, **kw)
self.into = into
#compiles(SelectInto)
def s_into(element, compiler, **kw):
text = compiler.visit_select(element)
text = text.replace('FROM',
'INTO TEMPORARY TABLE %s FROM' %
element.into)
return text
if __name__ == '__main__':
from sqlalchemy.sql import table, column
marker = table('marker',
column('x1'),
column('x2'),
column('x3')
)
print SelectInto([marker.c.x1, marker.c.x2], "tmp_markers").\
where(marker.c.x3==5).\
where(marker.c.x1.in_([1, 5]))
This needs some tweaking, hence it will replace all subquery selects as select INTOs, but test it for now, if it worked it would be better than raw text statments.
Have you tried this from this answer by #Michael Berkowski:
INSERT INTO assets_copy
SELECT * FROM assets;
The answer states that MySQL documentation states that SELECT * INTO isn't supported.
I have an oracle procedure that is supposed to return a chunk of json.
The procedure has 4 paramaters
input is an ID value
output json CLOB
output some message in json format CLOB
output if success or failure varchar2
so in my code I have done the following just to test if I can successfully call and return it
ConnectionString = 'someconnection.connection'
con = cx_Oracle.connect(ConnectionString)
cur = con.cursor()
ID = '51858645'
json_out = cur.var(cx_Oracle.CLOB)
message = cur.var(cx_Oracle.CLOB)
status = cur.var(cx_Oracle.STRING)
oracle_return = cur.callproc('project.getsomejson',[ID,json_out,message,status])
However, it fails and returns
PLS-00306: wrong number or types of arguments in call to 'getsomejson'
ORA-06550: line 1, column 7:
PL/SQL: Statement ignored
This is the procedure definition
procedure callOracle(json_in clob,json_out out clob,message out clob,status out varchar2)
This is calling an oracle 12c db
I'm really not familiar at all with calling procedures in python. Typically I callfunc with the stored type and just get a return
The procedure is expecting
CLOB, CLOB, CLOB, VARCHAR2
but you are passing
VARCHAR2, CLOB, CLOB, VARCHAR2
The name you gave (callOracle) also doesn't match what you are calling in Python (project.getsomejson). Perhaps verify that you have the right procedure signature? Assuming it is correct, though, you'll need to change the first one to be a CLOB as well or change the stored procedure to accept VARCHAR2. Something like this should do it:
json_in = conn.createlob(cx_Oracle.DB_TYPE_CLOB)
json_in.write("51858645")
json_out_var = cur.var(cx_Oracle.DB_TYPE_CLOB)
message_var = cur.var(cx_Oracle.DB_TYPE_CLOB)
status_var = cur.var(str)
cur.callproc(json_in, json_out, message, status)
I am writing a generic function to query a database where I want to use *args to get values. Here's the function
def get_latest_value(col_name, *args):
query = "select * from latest_value where {column} in (%s)"
query = query.format(**dict(column=col_name))
results = pd.read_sql_query(query, engine, params = [args])
return(results)
However, when I run this, I get this error:
ProgrammingError: (MySQLdb._exceptions.ProgrammingError) not all arguments converted during bytes formatting
[SQL: select * from latest_value where amfi_code in (%s)]
[parameters: (100370, 100371)]
(Background on this error at: http://sqlalche.me/e/f405)
I even tried converting the *args to a list before passing it, but that didn't help either.
Not able to figure out how to pass the values in *args to the IN clause. Any help is appreciated.
Also, is using string concatenation for column name this ways safe from sql injection attacks?
Update:
I changed the code as follows:
def get_latest_value(col_name, *args):
query = "select * from latest_value where {column} in (%s)"
query = query.format(**dict(column=col_name))
conditions = ", ".join(map(str, args))
results = pd.read_sql_query(query, engine, params = [conditions])
return(results)
I don't get an error now, but I only get the results for the first value in args.
I could not figure out exactly what, but I figured that sqlalchemy has a problem dealing with this scenario. It can be solved by using psycopg2 instead.
I wrote the code as follows
conn = psycopg2.connect(user='', host='', password='', database='')
query = "select * from latest_value where {column} = any(%s)"
query = query.format(**dict(column=col_name))
results = pd.read_sql_query(query, conn, params=(args,))
Note the change in the query. Instead of using in (%s), I have used =any(%s) as seen in the psycopg2 documentation.
Also note that for the above to work, args needs to be a Python list, not a tuple.
Using python, I'm trying to access text from a database (FileMaker Pro) using a pyodbc connection. When I use a foreach loop, for some reason the text prints. But, when I just straight away print it then I think it prints the location or something. The following code explains it better:
import pyodbc
connectString = "DSN=FMODBC32;UID=...;PWD=..."
connection = pyodbc.connect(connectString)
cursor = connection.cursor()
param = cursor.execute("select Parameters from Info")
## Parameters = fieldName in the database, Info = tableName in database
print(param)
## This prints: <pyodbc.Cursor object at 0x08930C20> (Is this the location?)
for info in param:
print(info)
## This prints the actual text I need, not the location
I want to just do print(param) and not a whole foreach loop to get the text I need. Any advice?
print(param)
only prints you the param object.
You should add
cursor = connection.cursor()
cursor.execute("select Parameters from Info")
param = cursor.fetchall()
and print(param) should be ok.
My code:
import yql
y = yql.Public()
query = 'SELECT * FROM yahoo.finance.option_contracts WHERE symbol="SPY"'
y.execute(query)
Result:
yql.YQLError: No definition found for Table yahoo.finance.option_contracts
I know that the table exists because I can test the query at http://developer.yahoo.com/yql/console/ and it works. What am I missing?
Update: I posted the url to the console but not the query I tried in the console. The query is now attached.
http://goo.gl/mNXwC
Since the yahoo.finance.option_contracts table is a Community Open Data Table you will want to include it as part of the environment for the query. The easiest way to do that is to load up the environment file for all community tables; just like clicking "Show Community Tables" in the YQL console.
One would normally do that by specifying an env=... parameter in the YQL query URL, or (as you have done) with a use clause in the query itself.
The Python library that you are using lets you pass in the environment file as an argument to execute().
import yql
y = yql.Public()
query = 'SELECT * FROM yahoo.finance.option_contracts WHERE symbol="SPY"'
y.execute(query, env="store://datatables.org/alltableswithkeys")
Here's an example of extending yql.Public to be able to define the default environment on instantiation.
class MyYql(yql.Public):
def __init__(self, api_key=None, shared_secret=None, httplib2_inst=None, env=None):
super(MyYql, self).__init__(api_key, shared_secret, httplib2_inst)
self.env = env if env else None
def execute(self, query, params=None, **kwargs):
kwargs["env"] = kwargs.get("env", self.env)
return super(MyYql, self).execute(query, params, **kwargs);
It can be used like:
y = MyYql(env="store://datatables.org/alltableswithkeys")
query = 'SELECT * FROM yahoo.finance.option_contracts WHERE symbol="SPY"'
r = y.execute(query)
You can still override the env in an individual call to y.execute() if you need to.
Amending query to the following is what works.
query = 'use "http://www.datatables.org/yahoo/finance/yahoo.finance.option_contracts.xml" as foo; SELECT * FROM foo WHERE symbol="SPY"'
More elegant solutions might exist. Please share if such do. Thanks.