I am attempting to run a SQL query on an oracle database like so:
import cx_Oracle as cx
import pandas as pd
un = "my username"
pw = "my password"
db = "database name"
lookup = "1232DX%"
myconn = cx.connect(un, pw, db)
cursor = myconn.cursor()
qry = """SELECT *
FROM tableX
WHERE tableX.code LIKE '1232DX%'"""
qry.df = pd.read_sql(qry, con = myconn)
myconn.close()
My issue is that it is redundant to define lookup before the query and use the value in the query itself. I would like to just be able to type
WHERE tableX.code LIKE lookup
and have the value 1232DX% substituted into my query.
I imagine there is a straightforward way to do this in Python, but I am hardly an expert so I thought I would ask someone here. All suggestions are welcome. If there is a better way to do this than what I have shown please include it. Thank you in advance.
You use the same syntax as when passing parameters to cursor.execute().
qry = """SELECT *
FROM tableX
WHERE tableX.code LIKE :pattern"""
qry.df = pd.read_sql(qry, con = myconn, params={":pattern": lookup})
Related
I have a Python function to read from an SQL table into a pandas DataFrame:
def project_cable_collector(dbase, table, project):
engine = create_engine(dbase)
df = pd.read_sql('SELECT * from table WHERE project_id = project', engine)
return (df)
However it returns sqlalchemy.exc.ProgrammingError:
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.SyntaxError) syntax error at or near "table"
LINE 1: SELECT * from table WHERE project_id = project
I tried editing quotation marks to see if that's a fix, but it fails.
Any ideas?
An exact fix to your current problem might be to use an f-string:
def project_cable_collector(dbase, table, project):
engine = create_engine(dbase)
sql = f"SELECT * FROM {table} WHERE project_id = {project}"
df = pd.read_sql(sql, engine)
return (df)
However, note that it is highly undesirable to build a SQL query string this way using concatenation and substitution. The reason is that your function invites something called SQL injection, which means that someone could pass in a malicious SQL code fragment into the function and try to get your Python script to execute it. Instead, you should read about using prepared statements.
Further to Tim's answer, you'll need to use an f-string to insert the table name into the SQL text, but you should use a parameter to specify the column value:
from sqlalchemy import text
# …
def project_cable_collector(dbase, table, project):
engine = create_engine(dbase)
sql = f"SELECT * FROM {table} WHERE project_id = :project_id"
df = pd.read_sql_query(text(sql), engine, params=dict(project_id=project))
return df
Note also that read_sql_query() is preferable to read_sql().
Currently i'm executing stored procedure that way:
engine = sqlalchemy.create_engine(self.getSql_conn_url())
query = "exec sp_getVariablesList #City = '{0}', #Station='{1}'".format(City, Station)
self.Variables = pd.read_sql_query(query, engine)
but at How set ARITHABORT ON at sqlalchemy was correctly noticed that that make that open to SQL injection. I tried different ways but without success. So how should I pass parameters to the MSSQL stored procedure to eliminate the risk of SQL injection? That can be with sqlalchemy or any other way.
Write your SQL command text using the "named" paramstyle, wrap it in a SQLAlchemy text() object, and pass the parameter values as a dict:
import pandas as pd
import sqlalchemy as sa
connection_uri = "mssql+pyodbc://#mssqlLocal64"
engine = sa.create_engine(connection_uri)
# SQL command text using "named" paramstyle
sql = """
SET NOCOUNT ON;
SET ARITHABORT ON;
EXEC dbo.breakfast #name = :name_param, #food = :food_param;
"""
# parameter values
param_values = {"name_param": "Gord", "food_param": "bacon"}
# execute query wrapped in SQLAlchemy text() object
df = pd.read_sql_query(sa.text(sql), engine, params=param_values)
print(df)
"""
column1
0 Gord likes bacon for breakfast.
"""
I am accessing oracle database and trying to update it using python. Below is my code :
import cx_Oracle
import pandas as pd
import datetime
import numpy
import math
conn = cx_Oracle.connect(conn_str)
c = conn.cursor()
def update_output_table(customer_id_list,column_name,column_vlaue_list) :
num_rows_to_add = len(customer_id_list)
conn = cx_Oracle.connect(conn_str)
c = conn.cursor()
for i in range(0,num_rows_to_add,1) :
c.execute("""UPDATE output SET """+column_name+""" = %s WHERE customer_id = %s""" %(column_vlaue_list[i],customer_id_list[i]))
total_transaction_df = pd.read_sql("""select distinct b.customer_id,count(a.transaction_id) as total_transaction from transaction_fact a,customer_dim b where a.customer_id = b.CUSTOMER_ID group by b.CUSTOMER_ID""",conn)
# Update this details to the output table
update_output_table(list(total_transaction_df['CUSTOMER_ID']),'TOTAL_TRANSACTION',list(total_transaction_df['TOTAL_TRANSACTION']))
conn.close()
My program is getting executed completely but I don't see my database table getting updated. Can someone suggest where I am going wrong?
Note : I am a newbie.Sorry for asking silly doubts. Thanks in advance.
You're missing conn.commit() before conn.close():
Here you will find some info why you need it explicitely. Without commit your code is doing update then when closing connection all non-commited changes are rolled back so you see no changes in DB.
You can also set cx_Oracle.Connection.autocommit = 1 but this is not recommended way as you're loosing control over transactions.
I have about 40 MS Access Databases and have some troubles if need to create or transfer one of MS Access Query (like object) from one db to other dbs.
So I tried to solve this problem with pyodbc but.. as I saw pyodbc doesn't support to create new, permanent MS Access Query (object).
I can connect to db, create or delete tables/rows but can't to create and save new query.
import pyodbc
odbc_driver = r"{Microsoft Access Driver (*.mdb, *.accdb)}"
db_test1 = r'''..\Test #1.accdb'''
db_test2 = r'''..\Test #2.accdb'''
db_test3 = r'''..\Test #3.accdb'''
db_test4 = r'''..\Test #4.accdb'''
db_test_objects = [db_test1, db_test2, db_test3, db_test4]
odbc_conn_str = "Driver=%s;DBQ=%s;" % (odbc_driver, db_file)
print (odbc_conn_str)
conn = pyodbc.connect(odbc_conn_str)
odbc_cursor = conn.cursor()
NewQuery = "CREATE TABLE TestTable(symbol varchar(15), leverage double)"
odbc_cursor.execute(NewQuery)
conn.commit()
conn.close()
SO, How to create and save MS Access Query like objects from python?
I tried to search info in Google, but the answers were related with Run SQL code.
On VBA this code looks like:
Public Sub CreateQueryDefX()
Dim base(1 To 4) As String
base(1) = "..\Test #1.accdb"
base(2) = "..\Test #2.accdb"
base(3) = "..\Test #3.accdb"
base(4) = "..\Test #4.accdb"
For i = LBound(base) To UBound(base)
CurrentBase = base(i)
Set dbo = OpenDatabase(CurrentBase)
With dbo
Set QueryNew = .CreateQueryDef("TestQuery", _
"SELECT * FROM TestTable")
RefreshDatabaseWindow
.Close
End With
Next i
RefreshDatabaseWindow
End Sub
Sorry for my English, it's not my native :)
By the way, I know how to solve this by VBA, but I'm interested in solve this by python.
Thank you.
You can use a CREATE VIEW statement to create a saved Select Query in Access. The pyodbc equivalent to your VBA example would be
crsr = conn.cursor()
sql = """\
CREATE VIEW TestQuery AS
SELECT * FROM TestTable
"""
crsr.execute(sql)
To delete that saved query you could simply execute a DROP VIEW statement.
For more information on DDL in Access see
Data Definition Language
Consider the Python equivalent of the VBA running exactly what VBA uses: a COM interface to the Access Object library. With Python's win32com third-party module, you can call the CreateQueryDef method. Do note: this COM interfacing can be applied in other languages such as PHP and R!
Below uses a try/except/finally block to ensure the Access application process closes regardless of error or success of code (similar to VBA's On Error handling):
import win32com.client
# OPEN ACCESS APP AND DATABASE
dbases = ["..\Test #1.accdb", "..\Test #2.accdb", "..\Test #3.accdb", "..\Test #4.accdb"]
try:
oApp = win32com.client.Dispatch("Access.Application")
# CREATE QUERYDEF
for db in dbases:
oApp.OpenCurrentDatabase(db)
currentdb = oApp.CurrentDb()
currentdb.CreateQueryDef("TestQuery", "SELECT * FROM TestTable")
currentdb = None
oApp.DoCmd.CloseDatabase
except Exception as e:
print(e)
finally:
currentdb = None
oApp.Quit
oApp = None
Also, if you need to run DML statements via pyodbc and not a COM interface, consider distributed queries as Access can query other databases directly in SQL. Below should work in Python (be sure to escape the backslash):
SELECT t.* FROM [C:\Path\To\Other\Database.accdb].TestTable t
I`m quite new to python and would like to copy a table from one mdb to another mdb using pyodbc. There seems to be a problem with the paths if a Foldername starts with a digit. I googled for an hour now and couldn't find a solution:
DBfile = r"W:\path\1020 Folder\MDB1.mdb"
conn = pyodbc.connect('DRIVER={Microsoft Access Driver (*.mdb)};DBQ='+DBfile1)
cursor = conn.cursor()
sql = """SELECT Table1.* INTO test FROM [W:\path\A 1020 Folder\MB2.mdb].Table1;"""
sql1 = """SELECT Table1.* INTO test FROM [W:\path\1020 Folder\MB2.mdb].Table1;"""
cursor.execute(sql) #WORKING
cursor.execute(sql1) #NOT WORKING
conn.commit()
Thanks alot, Achim
You must be very careful when you want to use backshlash \ in strings. You can escape those using \\:
sql1 = """SELECT Table1.* INTO test FROM [W:\\path\\1020 Folder\\MB2.mdb].Table1;"""
You can also use raw string just like you did it with DBfile