I'm trying to execute some raw SQL against a temp table through the web2py DAL, but my results are all returning None.
Here's the full function:
def test():
db_test = DAL('mysql://root:root#localhost/test')
sql = """CREATE TEMPORARY TABLE tmp LIKE people;
INSERT INTO tmp SELECT * FROM people;
INSERT INTO tmp SELECT * FROM people;
SELECT * FROM tmp;"""
results = db_test.executesql(sql)
Obviously the SQL is a simplification, but running the same SQL in a SQL pane returns the correct results. What do I need to do to get the DAL working with this?
You cannot execute multiple statements in one executesql call I suspect; web2py uses the DBAPI 2.0 .execute() call for sending these to the backend database and that usually supports only single statements:
db_test = DAL('mysql://root:root#localhost/test')
sqlddl = """CREATE TEMPORARY TABLE tmp LIKE people;
INSERT INTO tmp SELECT * FROM people;
INSERT INTO tmp SELECT * FROM people;"""
for statement in sqlddl.split(';'):
db_test.executesql(statement.strip())
sqlselect = "SELECT * FROM tmp;"
results = db_test.executesql(sqlselect)
Related
I have a Python function to read from an SQL table into a pandas DataFrame:
def project_cable_collector(dbase, table, project):
engine = create_engine(dbase)
df = pd.read_sql('SELECT * from table WHERE project_id = project', engine)
return (df)
However it returns sqlalchemy.exc.ProgrammingError:
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.SyntaxError) syntax error at or near "table"
LINE 1: SELECT * from table WHERE project_id = project
I tried editing quotation marks to see if that's a fix, but it fails.
Any ideas?
An exact fix to your current problem might be to use an f-string:
def project_cable_collector(dbase, table, project):
engine = create_engine(dbase)
sql = f"SELECT * FROM {table} WHERE project_id = {project}"
df = pd.read_sql(sql, engine)
return (df)
However, note that it is highly undesirable to build a SQL query string this way using concatenation and substitution. The reason is that your function invites something called SQL injection, which means that someone could pass in a malicious SQL code fragment into the function and try to get your Python script to execute it. Instead, you should read about using prepared statements.
Further to Tim's answer, you'll need to use an f-string to insert the table name into the SQL text, but you should use a parameter to specify the column value:
from sqlalchemy import text
# …
def project_cable_collector(dbase, table, project):
engine = create_engine(dbase)
sql = f"SELECT * FROM {table} WHERE project_id = :project_id"
df = pd.read_sql_query(text(sql), engine, params=dict(project_id=project))
return df
Note also that read_sql_query() is preferable to read_sql().
I like to keep one file with just my f strings that represents SQL queries and each query needs a db name. In my python program, i will populate the db variable as I am reading out db names.
E.g.
in queries.py, i have
query_1 = f"select * from {db} limit 10;"
in main.py, i use
from quries import quries
# read out db information into dbs
for db in dbs:
# execute the query_1 for each db
How can I achieve the logic in the loop?
Don't use an f-string, but .format()
# queries
query_1 = "select * from {db} limit 10;"
from queries import query_1
...
for db in dbs:
query = query_1.format(db=db)
...
I am trying to connect to a Postgres DB and execute a simple select query. But I am getting errors. Below is the sample program
import psycopg2 as ps
param_dic = ps.connect(
host="localhost",
database="Test_DB",
user="username",
password="password"
)
cur = param_dic.cursor()
cur.execute("select * from <schema_name>.'employee_tbl'") # I am
rows = cur.fetchall()
for r in rows:
print(r)
I get below error:
psycopg2.errors.SyntaxError: syntax error at or near "'tbl_name'"
LINE 1: select * from <schema_name>.'tbl_name'
If I use
cur.execute("select * from <schema_name>.'Employee_TBL'")
Still, I get the same error.
Note: when I check my pgAdmin, I see below as the table name
<schema_name>.Employee_TBL
So, what is the right way to execute SQL queries on a Postgres table?
replace <schema_name>.'Employee_TBL'" with true value :
dbo.Employee_TBL or another schema name in your database
Maybe it's just a typo, use double quotes for the table if you need to.
Single quotes indicate a string literal.
import psycopg2
conn = psycopg2.connect(...)
cur = conn.cursor()
cur.execute('SELECT * FROM public.servers')
print(len(cur.fetchall()))
cur.execute('SELECT * FROM PUBLIC."servers"')
print(len(cur.fetchall()))
Out:
6
6
Try removing quotes around ‘emloyee_tbl’
The overriding issue is that you created a table name with quoted mixed case. This means that from then on you need to quote the name to be able to use it:
create table "Mixed_Case" (id int, fld_1 varchar);
select * from mixed_case;
ERROR: relation "mixed_case" does not exist
LINE 1: select * from mixed_case;
select * from "Mixed_Case";
id | fld_1
----+-------
(0 rows)
--NOTE the double quotes.
For more information see 4.1.1. Identifiers and Key Words
For the psycopg2 case:
import psycopg2
con = psycopg2.connect("dbname='test' host='localhost' user='aklaver'")
cur = con.cursor()
# Single quoted
cur.execute("select * from public.'Mixed_Case'")
SyntaxError: syntax error at or near "'Mixed_Case'"
LINE 1: select * from public.'Mixed_Case'
# Identifiers need double quotes
# Changing to double quotes
cur.execute('select * from public."Mixed_Case"')
# No error.
# You can also use Postgres functions
cur.execute("select * from quote_ident('public.Mixed_Case')")
For more on Postgres functions and methods see string functions
Also in psycopg2 there is the sql module that allows you to work with identifiers:
from psycopg2 import sql
sql_str = sql.SQL("select * from public.{}").format(sql.Identifier('Mixed_Case'))
print(sql_str.as_string(con))
select * from public."Mixed_Case"
cur.execute(sql_str)
I have a SQLite db with three relational tables. I'm trying to return the max record from a log table along with related columns from the other tables based on the ID relationships.
I created the query in DB Browser and verified it returns the expected record however, when I use the exact same query statement in my python code it never steps into the 'for' loop.
SQL statement in python -
def GetLastLogEntry():
readings = ()
conn = sqlite3.connect(dbName)
conn.row_factory = sqlite3.Row
cursor = conn.cursor()
cursor.execute("SELECT f.FoodCategory, f.FoodName, gs.FoodWeight,
gsl.GrillDateTime, gsl.CurrentGrillTemp, gsl.TargetGrillTemp,
gsl.CurrentFoodTemp, gsl.TargetFoodTemp, gsl.CurrentOutsideTemp,
gsl.CurrentOutsideHumidity FROM Food as f, GrillSession as gs,
GrillSessionLog as gsl WHERE f.FoodId = gs.FoodId AND
gs.GrillSessionID = gsl.GrillSessionID AND gsl.GrillSessionLogID =
(SELECT MAX(GrillSessionLog.GrillSessionLogID) FROM
GrillSessionLog, GrillSession WHERE GrillSessionLog.GrillSessionID
= GrillSession.GrillSessionID AND GrillSession.ActiveSession =
1)")
for row in cursor:
print("In for loop")
readings = readings + (row['FoodCategory'], row['FoodName'])
print("Food Cat = " + row['FoodCategory'])
cursor.close()
return readings
The query in DB Browser returns only one row which is what I'm trying to have happen in the python code.
Just discovered the issue....
Using DB Browser, I updated a record I'm using for testing but failed to "write" the change to the database table. As a result, every time I was executing my python code against the table it was executing the query with the original record values because my change wasn't yet committed via DB Browser.
Huge brain fart on that one.... Hopefully it will be a lesson learned for someone else in the future.
I'm trying to do a simple select statement on a table that's part of the "dam_vector" schema. The error I get is:
psycopg2.ProgrammingError: relation
"dam_vector.parcels_full" does not
exist LINE 1: SELECT * FROM
"dam_vector.parcels_full"
I can't figure this out and know I'm missing something obvious. Any help you can provide would be great.
Here's the code I'm using. db is a connection string that successfully connects to the database.
cur = db.cursor()
query = 'SELECT * FROM "dam_vector.parcels_full"'
cur.execute(query)
results = cur.fetchall()
and when that failed and after I did some research on Google I tried this. Same error.
cur.execute("SET search_path TO dam_vector,public")
db.commit()
cur = db.cursor()
query = 'SELECT * FROM "parcels_full"'
cur.execute(query)
results = cur.fetchall()
Double quotes makes whatever is in them an identifier, so query
SELECT * FROM "dam_vector.parcels_full";
hits table dam_vector.parcels_full (period interpreted as part of table name) from schama public (or anything in search path).
As Adam said, you don't need quotes with names without some special characters.
Try:
SELECT * FROM dam_vector.parcels_full;
If you really want to use a double quotes, go for:
SELECT * FROM "dam_vector"."parcels_full";
You shouldn't need the quotes around dam_vector.parcels_full.
Does the output of the following show that a parcels_full table is indeed present?
cur.execute("""SELECT tablename
FROM pg_tables
WHERE tablename NOT LIKE ALL (ARRAY['pg_%','sql_%']);""")
cur.fetchall()