We Keep Coding

Using Python variables for table name and column value in an SQL query

I have a Python function to read from an SQL table into a pandas DataFrame:
def project_cable_collector(dbase, table, project):
engine = create_engine(dbase)
df = pd.read_sql('SELECT * from table WHERE project_id = project', engine)
return (df)
However it returns sqlalchemy.exc.ProgrammingError:
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.SyntaxError) syntax error at or near "table"
LINE 1: SELECT * from table WHERE project_id = project
I tried editing quotation marks to see if that's a fix, but it fails.
Any ideas?

An exact fix to your current problem might be to use an f-string:
def project_cable_collector(dbase, table, project):
engine = create_engine(dbase)
sql = f"SELECT * FROM {table} WHERE project_id = {project}"
df = pd.read_sql(sql, engine)
return (df)
However, note that it is highly undesirable to build a SQL query string this way using concatenation and substitution. The reason is that your function invites something called SQL injection, which means that someone could pass in a malicious SQL code fragment into the function and try to get your Python script to execute it. Instead, you should read about using prepared statements.

Further to Tim's answer, you'll need to use an f-string to insert the table name into the SQL text, but you should use a parameter to specify the column value:
from sqlalchemy import text
# …
def project_cable_collector(dbase, table, project):
engine = create_engine(dbase)
sql = f"SELECT * FROM {table} WHERE project_id = :project_id"
df = pd.read_sql_query(text(sql), engine, params=dict(project_id=project))
return df
Note also that read_sql_query() is preferable to read_sql().

Python - use string literals in Oracle SQL Query

I am attempting to run a SQL query on an oracle database like so:
import cx_Oracle as cx
import pandas as pd
un = "my username"
pw = "my password"
db = "database name"
lookup = "1232DX%"
myconn = cx.connect(un, pw, db)
cursor = myconn.cursor()
qry = """SELECT *
FROM tableX
WHERE tableX.code LIKE '1232DX%'"""
qry.df = pd.read_sql(qry, con = myconn)
myconn.close()
My issue is that it is redundant to define lookup before the query and use the value in the query itself. I would like to just be able to type
WHERE tableX.code LIKE lookup
and have the value 1232DX% substituted into my query.
I imagine there is a straightforward way to do this in Python, but I am hardly an expert so I thought I would ask someone here. All suggestions are welcome. If there is a better way to do this than what I have shown please include it. Thank you in advance.

You use the same syntax as when passing parameters to cursor.execute().
qry = """SELECT *
FROM tableX
WHERE tableX.code LIKE :pattern"""
qry.df = pd.read_sql(qry, con = myconn, params={":pattern": lookup})

ORA-00942 when importing data from Oracle Database to Pandas

I try to import data from a oracle database to a pandas dataframe.
right Now i am using:
import cx_Oracle
import pandas as pd
db_connection_string = '.../A1#server:port/servername'
con = cx_Oracle.connect(db_connection_string)
query = """SELECT*
FROM Salesdata"""
df = pd.read_sql(query, con=con)
and get the following error: DatabaseError: ORA-00942: Table or view doesn't exist
When I run a query to get the list of all tables:
cur = con.cursor()
cur.execute("SELECT table_name FROM dba_tables")
for row in cur:
print (row)
The output looks like this:
('A$',)
('A$BD',)
('Salesdata',)
What I am doing wrong? I used this question to start.
If I use the comment to print(query)I get:
SELECT*
FROM Salesdata

Getting ORA-00942 when running SELECT can have 2 possible causes:
The table does not exist: here you should make sure the table name is prefixed by the table owner (schema name) as in select * from owner_name.table_name. This is generally needed if the current Oracle user connected is not the table owner.
You don't have SELECT privileges on the table. This is also generally needed if the current Oracle user connected is not the table owner.
You need to check both.

Running python script to get SQL Statement for Google BigQuery

Trying to run a script that contains a SQL query:
import example_script
example_script.df.describe()
example_script.df.info()
q1 = '''
example_script.df['specific_column'])
'''
job_config = bigquery.QueryJobConfig()
query_job = client.query(q1, job_config= job_config)
q = query_job.to_dataframe()
Issues I'm having are when I import it, how do I get that specific column name used as a text? Then it will run the query from GBQ but instead, it's stuck in pandas formatting that Google doesn't want to read. Are there other options?

Search Sqlite Database - All Tables and Columns

Is there a library or open source utility available to search all the tables and columns of an Sqlite database? The only input would be the name of the sqlite DB file.
I am trying to write a forensics tool and want to search sqlite files for a specific string.

Just dump the db and search it.
% sqlite3 file_name .dump | grep 'my_search_string'
You could instead pipe through less, and then use / to search:
% sqlite3 file_name .dump | less

You could use "SELECT name FROM sqlite_master WHERE type='table'"
to find out the names of the tables in the database. From there it is easy to SELECT all rows of each table.
For example:
import sqlite3
import os
filename = ...
with sqlite3.connect(filename) as conn:
conn.row_factory = sqlite3.Row
cursor = conn.cursor()
cursor.execute("SELECT name FROM sqlite_master WHERE type='table'")
for tablerow in cursor.fetchall():
table = tablerow[0]
cursor.execute("SELECT * FROM {t}".format(t = table))
for row in cursor:
for field in row.keys():
print(table, field, row[field])

I know this is late to the party, but I had a similar issue but since it was inside of a docker image I had no access to python, so I solved it like so:
for X in $(sqlite3 database.db .tables) ; do sqlite3 database.db "SELECT * FROM $X;" | grep >/dev/null 'STRING I WANT' && echo $X; done
This will iterate through all tables in a database file and perform a select all operation which I then grep for the string. If it finds the string, it prints the table, and from there I can simply use sqlite3 to find out how it was used.
Figured it might be helpful to other who cannot use python.

#MrWorf's answer didn't work for my sqlite file (an .exb file from Evernote) but this similar method worked:
Open the file with DB Browser for SQLite sqlitebrowser mynotes.exb
File / Export to SQL file (will create mynotes.exb.sql)
grep 'STRING I WANT" mynotes.exb.sql