I am trying to go through each query in a SQL file and execute it in my python script using psycopg2. Each query has an id which I replace before executing.
The first query in the sql file is the following:
select * from subscriber where org_id = '1111111111';
I get the old id and replace it with the new id that I am looking for
id_regex = re.compile("\d{10,}")
m = id_regex.search(q)
old_id = m.group(0)
new_q = q.replace(old_id, new_id)
I then execute the queries on the following manner
for index, cmd in enumerate(cmds):
# ... (other stuff here)
elif cmd != '\n':
new_cmd = p_helper.replace_id(org_id, cmd)
logger.debug("Running Command:\n" + new_cmd)
try:
if not test_run:
db_cursor.execute(new_cmd)
except psycopg2.Error as e:
logger.error(e.pgerror)
else:
pass
# DO NOTHING
When I run my program I get the following error:
ERROR: syntax error at or near "select"
LINE 1: select * from subscriber where org_id = '9999999999';
^
Every query after the first doesn't run
ERROR: current transaction is aborted, commands ignored until end of transaction block
I ran the select query manually in psql and it worked perfectly so I don't think the problem is the syntax of the statement. I think it has something to do with the formatting of queries that psycopg2 takes. I'm not sure exactly what to change, I have looked at other SO posts and could not figure out what I needed to change. It'd be great if someone could help me figure this out. Thanks!
Versions
python: 2.7.6
psycopg2: 2.4.5
Related
I am unable to understand why there are two queries being executed. First we are executing the prepared statement and we are using the build cypher function. The code can be found here
https://github.com/apache/age/blob/master/drivers/python/age/age.py
def execCypher(conn:ext.connection, graphName:str, cypherStmt:str, cols:list=None, params:tuple=None) -> ext.cursor :
if conn == None or conn.closed:
raise _EXCEPTION_NoConnection
cursor = conn.cursor()
#clean up the string for modification
cypherStmt = cypherStmt.replace("\n", "")
cypherStmt = cypherStmt.replace("\t", "")
cypher = str(cursor.mogrify(cypherStmt, params))
cypher = cypher[2:len(cypher)-1]
preparedStmt = "SELECT * FROM age_prepare_cypher({graphName},{cypherStmt})"
cursor = conn.cursor()
try:
cursor.execute(sql.SQL(preparedStmt).format(graphName=sql.Literal(graphName),cypherStmt=sql.Literal(cypher)))
except SyntaxError as cause:
conn.rollback()
raise cause
except Exception as cause:
conn.rollback()
raise SqlExecutionError("Execution ERR[" + str(cause) +"](" + preparedStmt +")", cause)
stmt = buildCypher(graphName, cypher, cols)
cursor = conn.cursor()
try:
cursor.execute(stmt)
return cursor
except SyntaxError as cause:
conn.rollback()
raise cause
except Exception as cause:
conn.rollback()
raise SqlExecutionError("Execution ERR[" + str(cause) +"](" + stmt +")", cause)
Both statements perform the same operation.
The difference is that preparedStmt and buildCypher function use different form of cypher queries as shown in code. (cypherStmt & cypher) And their code for building the query is a bit different.
I can't tell you why it's done this way but I'll show you why it's different. Also apologies but I'm not used to Python or C.
The preparedStatement is calling a custom postgres function age_prepare_cypher in this file here apache/age/src/backend/utils/adt/age_session_info.c, which calls set_session_info(graph_name_str, cypher_statement_str);.
And the set_session_info in this file here apache/age/src/backend/utils/adt/age_session_info.c just sets it to a global variable session_info_cypher_statement.
So your graph name and query are being set in the session.
There's another function that gets your graph name and query back out of the session, and that is the convert_cypher_to_subquery. It only gets them out if is_session_info_prepared() is true, and only if graph_name and query_str provided to it are NULL.
Seems strange right? But now let's look at this bit of the python buildCypher function code:
stmtArr = []
stmtArr.append("SELECT * from cypher(NULL,NULL) as (")
stmtArr.append(','.join(columnExp))
stmtArr.append(");")
return "".join(stmtArr)
It's taking your query and saying your graph name and query string are NULL.
So we can conclude that the prepare statement is storing those values in session memory, and then when you execute your statement after using buildCypher, it's getting them out of memory and completing the statement again.
I can't explain exactly why or how it does it, but I can see a chunk of test sql in the project that is doing the same sort of thing here:
-- should return true and execute cypher command
SELECT * FROM age_prepare_cypher('analyze', 'MATCH (u) RETURN (u)');
SELECT * FROM cypher(NULL, NULL) AS (result agtype);
So tl;dr, executing the prepareStatement is storing it in session memory, and executing the normal statement after running it through buildCypher is grabbing what was just stored in the session.
I have a script, in which part of it calls to Snowflake on a query. That query is located in an .sql folder and two params are passed through to it in the script. For example:
#query.sql
select *
from {some_table}
where
date == '{hit_date}'
and
id in ({id_list}) or ({id_list}) is null)
And the piece of script acts like:
def run_query(hit_date, id_list):
conn = snowflake.connector.connect(**snowflake_creds)
cursor = conn.cursor()
with open(query.sql) as f:
query = f.read()
cursor.execute(query.format(hit_date, id_list))
The problem comes when the input for 'id_list' is 'None'. I am thrown this error:
snowflake.connector.errors.ProgrammingError: 000904 (42000): 019b2591-04da-7b13-0000-89bd25e0043a: SQL compilation error: error line 20 at position 42
invalid identifier 'NONE'
From my understanding Python's 'None' will automatically convert to SQL's 'Null'. Is this not the case? What should the input for 'id_list' be if 'None/Null'?
I was unable to find a clear answer for this specific error. However I was able to work around this by replacing the condition with an impossible value when no list is given.
#query.sql
select *
from {some_table}
where
date == '{hit_date}'
and
(id in ({id_list}) or impossible_value in ({id_list}))
I am trying to setup a python script to get some data and store it into a SQLite database. However when I am running the script a .fuse_hidden file is created.
On windows no .fuse_hidden file is observed but on ubuntu it generates at each call. The .fuse_hidden file seems to contain some form of sql query with input and tables.
I can delete the files without error during runtime but they are not deleted automatically. I make sure to end my connection to the db when I am finished with the query.
lsof give no information.
I am out of ideas on what to try next to get the files removed automatically. Any suggestions?
Testing
In order to confirm that it is nothing wrong with the code I made a simple script
(Assume there is an empty error.db)
import sqlite3
conn = sqlite3.connect("error.db")
cur = conn.cursor()
create_query = """
CREATE TABLE Errors (
name TEXT
);"""
try:
cur.execute(create_query)
except:
pass
cur.execute("INSERT INTO Errors (name) VALUES(?)", ["Test2"])
conn.commit()
cur.close()
conn.close()
I am running a rather complec update MS Access query from Python:
qry = '''
UPDATE H500_ODFlows INNER JOIN H500_UPDATE ON
(H500_ODFlows.Product = H500_UPDATE.Product)
AND (H500_ODFlows.Dest = H500_UPDATE.DestCode)
AND (H500_ODFlows.Orig = H500_UPDATE.OrigCode)
SET H500_ODFlows.Pieces = [H500_UPDATE].[Pieces],
H500_ODFlows.Weight = [H500_UPDATE].[Weight],
H500_ODFlows.Cons = [H500_UPDATE].[Pieces],
H500_ODFlows.DeadWeight = [H500_UPDATE].[DeadWeight],
H500_ODFlows.DoNotRead = [H500_UPDATE].DoNotRead,
H500_ODFlows.[_OrigCountryCode] = [H500_UPDATE].[_OrigCountryCode],
H500_ODFlows.[_DestCountryCode] = [H500_UPDATE].[_DestCountryCode]
'''
try:
crsr.execute(lb.cleanqry(qry))
cnxn.commit()
print('Updating was successful.')
except Exception as err:
print('Updating failed. See the error.' + str(err))
but get the following error:
('HY000', '[HY000] [Microsoft][ODBC Microsoft Access Driver] File
sharing lock count exceeded. Increase MaxLocksPerFile registry entry.
(-1033) (SQLExecDirectW)')
I followed the instructions to increase "MaxLocksPerFile" but it is not helping. Moreover, the query runs in MS Access quite OK but not through Python. Any advice?
Try running the query with autocommit on. That way, the database won't need to keep all those locks open, but can just commit everything as the query runs.
qry = '''
UPDATE H500_ODFlows INNER JOIN H500_UPDATE ON
(H500_ODFlows.Product = H500_UPDATE.Product)
AND (H500_ODFlows.Dest = H500_UPDATE.DestCode)
AND (H500_ODFlows.Orig = H500_UPDATE.OrigCode)
SET H500_ODFlows.Pieces = [H500_UPDATE].[Pieces],
H500_ODFlows.Weight = [H500_UPDATE].[Weight],
H500_ODFlows.Cons = [H500_UPDATE].[Pieces],
H500_ODFlows.DeadWeight = [H500_UPDATE].[DeadWeight],
H500_ODFlows.DoNotRead = [H500_UPDATE].DoNotRead,
H500_ODFlows.[_OrigCountryCode] = [H500_UPDATE].[_OrigCountryCode],
H500_ODFlows.[_DestCountryCode] = [H500_UPDATE].[_DestCountryCode]
'''
try:
cnxn.autocommit = True
crsr.execute(lb.cleanqry(qry))
print('Updating was successful.')
except Exception as err:
print('Updating failed. See the error.' + str(err))
Since you note: The query runs in MS Access quite OK but not through Python. One possible reason for this is Access stored queries are more efficient than application layer called queries since the engine saves and caches best execution plan. In the application layer (Python, VBA, etc.) when processing a string SQL statement, the Jet/ACE engine does not have time to plan the best execution.
Therefore, consider the following:
Add any needed indexes to JOIN variables of respective tables.
Save your UPDATE query as a stored query inside the database. Here, the saving process checks syntax, calculates and optimizes plan, and cache stats.
Run Compact & Repair in database to refresh stats.
Then, run query in Python as a stored proc with CALL command:
# SET AUTOCOMMIT PREFERENCE IN CONNECTION
cnxn = pyodbc.connect(..., autocommit=True)
...
crsr.execute("{CALL myUpdateQuery}")
My simple test code is listed below. I created the table already and can query it using the SQLite Manager add-in on Firefox so I know the table and data exist. When I run the query in python (and using the python shell) I get the no such table error
def TroyTest(self, acctno):
conn = sqlite3.connect('TroyData.db')
curs = conn.cursor()
v1 = curs.execute('''
SELECT acctvalue
FROM balancedata
WHERE acctno = ? ''', acctno)
print v1
conn.close()
When you pass SQLite a non-existing path, it'll happily open a new database for you, instead of telling you that the file did not exist before. When you do that, it'll be empty and you'll instead get a "No such table" error.
You are using a relative path to the database, meaning it'll try to open the database in the current directory, and that is probably not where you think it is..
The remedy is to use an absolute path instead:
conn = sqlite3.connect('/full/path/to/TroyData.db')
You need to loop over the cursor to see results:
curs.execute('''
SELECT acctvalue
FROM balancedata
WHERE acctno = ? ''', acctno)
for row in curs:
print row[0]
or call fetchone():
print curs.fetchone() # prints whole row tuple
The problem is the SQL statment. you must specify the db name and after the table name...
'''SELECT * FROM db_name.table_name WHERE acctno = ? '''