I have a script, in which part of it calls to Snowflake on a query. That query is located in an .sql folder and two params are passed through to it in the script. For example:
#query.sql
select *
from {some_table}
where
date == '{hit_date}'
and
id in ({id_list}) or ({id_list}) is null)
And the piece of script acts like:
def run_query(hit_date, id_list):
conn = snowflake.connector.connect(**snowflake_creds)
cursor = conn.cursor()
with open(query.sql) as f:
query = f.read()
cursor.execute(query.format(hit_date, id_list))
The problem comes when the input for 'id_list' is 'None'. I am thrown this error:
snowflake.connector.errors.ProgrammingError: 000904 (42000): 019b2591-04da-7b13-0000-89bd25e0043a: SQL compilation error: error line 20 at position 42
invalid identifier 'NONE'
From my understanding Python's 'None' will automatically convert to SQL's 'Null'. Is this not the case? What should the input for 'id_list' be if 'None/Null'?
I was unable to find a clear answer for this specific error. However I was able to work around this by replacing the condition with an impossible value when no list is given.
#query.sql
select *
from {some_table}
where
date == '{hit_date}'
and
(id in ({id_list}) or impossible_value in ({id_list}))
Related
I am unable to understand why there are two queries being executed. First we are executing the prepared statement and we are using the build cypher function. The code can be found here
https://github.com/apache/age/blob/master/drivers/python/age/age.py
def execCypher(conn:ext.connection, graphName:str, cypherStmt:str, cols:list=None, params:tuple=None) -> ext.cursor :
if conn == None or conn.closed:
raise _EXCEPTION_NoConnection
cursor = conn.cursor()
#clean up the string for modification
cypherStmt = cypherStmt.replace("\n", "")
cypherStmt = cypherStmt.replace("\t", "")
cypher = str(cursor.mogrify(cypherStmt, params))
cypher = cypher[2:len(cypher)-1]
preparedStmt = "SELECT * FROM age_prepare_cypher({graphName},{cypherStmt})"
cursor = conn.cursor()
try:
cursor.execute(sql.SQL(preparedStmt).format(graphName=sql.Literal(graphName),cypherStmt=sql.Literal(cypher)))
except SyntaxError as cause:
conn.rollback()
raise cause
except Exception as cause:
conn.rollback()
raise SqlExecutionError("Execution ERR[" + str(cause) +"](" + preparedStmt +")", cause)
stmt = buildCypher(graphName, cypher, cols)
cursor = conn.cursor()
try:
cursor.execute(stmt)
return cursor
except SyntaxError as cause:
conn.rollback()
raise cause
except Exception as cause:
conn.rollback()
raise SqlExecutionError("Execution ERR[" + str(cause) +"](" + stmt +")", cause)
Both statements perform the same operation.
The difference is that preparedStmt and buildCypher function use different form of cypher queries as shown in code. (cypherStmt & cypher) And their code for building the query is a bit different.
I can't tell you why it's done this way but I'll show you why it's different. Also apologies but I'm not used to Python or C.
The preparedStatement is calling a custom postgres function age_prepare_cypher in this file here apache/age/src/backend/utils/adt/age_session_info.c, which calls set_session_info(graph_name_str, cypher_statement_str);.
And the set_session_info in this file here apache/age/src/backend/utils/adt/age_session_info.c just sets it to a global variable session_info_cypher_statement.
So your graph name and query are being set in the session.
There's another function that gets your graph name and query back out of the session, and that is the convert_cypher_to_subquery. It only gets them out if is_session_info_prepared() is true, and only if graph_name and query_str provided to it are NULL.
Seems strange right? But now let's look at this bit of the python buildCypher function code:
stmtArr = []
stmtArr.append("SELECT * from cypher(NULL,NULL) as (")
stmtArr.append(','.join(columnExp))
stmtArr.append(");")
return "".join(stmtArr)
It's taking your query and saying your graph name and query string are NULL.
So we can conclude that the prepare statement is storing those values in session memory, and then when you execute your statement after using buildCypher, it's getting them out of memory and completing the statement again.
I can't explain exactly why or how it does it, but I can see a chunk of test sql in the project that is doing the same sort of thing here:
-- should return true and execute cypher command
SELECT * FROM age_prepare_cypher('analyze', 'MATCH (u) RETURN (u)');
SELECT * FROM cypher(NULL, NULL) AS (result agtype);
So tl;dr, executing the prepareStatement is storing it in session memory, and executing the normal statement after running it through buildCypher is grabbing what was just stored in the session.
I want to create a new table if I query a table that ends up being non existent. I thought to do this via catching the exception generated, but I can unable to do that despite specifying the error that pops up in my except statement.
Each of the try blocks I have below lead the two exception at the end.
import pyodbc as py
qry = \
"""
SELECT *
FROM NON_EXISTENT_TABLE
"""
cmd = \
"""
CREATE TABLE SOME_TABLE(VAR1 VARCHAR(10),
VAR2 VARCHAR(50), VAR3 VARCHAR(5), VAR4 VARCHAR(6));
"""
try:
df = pd.read_sql_query(qry, py.connect('DSN=SOMEDSN; Trusted_Connection = Yes'))
except py.ProgrammingError:
py.connect('DSN=SOMEDSN; Trusted_Connection=Yes').cursor().execute(cmd)
try:
df = pd.read_sql_query(qry, py.connect('DSN=SOMEDSN; Trusted_Connection = Yes'))
except py.DatabaseError:
py.connect('DSN=SOMEDSN; Trusted_Connection=Yes').cursor().execute(cmd)
This is the error I get
ProgrammingError: ('42S02', '[42S02] [SAP AG][LIBODBCHDB DLL][HDBODBC] Base table or view not found;259 invalid table name: Could not find table/view NON_EXISTENT_TABLE in schema SOMESCHEMA: line 3 col 15 (at pos 25) (259) (SQLExecDirectW)')
The above exception was the direct cause of the following exception:
DatabaseError: Execution failed on sql '
SELECT *
FROM SOMESCHEMA.NON_EXISTENT_TABLE
': ('42S02', '[42S02] [SAP AG][LIBODBCHDB DLL][HDBODBC] Base table or view not found;259 invalid table name: Could not find table/view NON_EXISTENT_TABLE in schema SOMESCHEMA: line 3 col 15 (at pos 25) (259) (SQLExecDirectW)')
I am not able to provide an answer on the specific Python issue. However, it may be helpful for you to know, that SAP offers an advanced Python client for SAP HANA, which makes life a lot easier especially when working with Pandas dataframes.
With the hana-ml client, you can use function has_table to check if a table exists. Depending on the outcome you may use function create_table to create a new table. Finally you can use the HANA Dataframe to query data and collect the result as a Pandas dataframe.
I have an oracle procedure that is supposed to return a chunk of json.
The procedure has 4 paramaters
input is an ID value
output json CLOB
output some message in json format CLOB
output if success or failure varchar2
so in my code I have done the following just to test if I can successfully call and return it
ConnectionString = 'someconnection.connection'
con = cx_Oracle.connect(ConnectionString)
cur = con.cursor()
ID = '51858645'
json_out = cur.var(cx_Oracle.CLOB)
message = cur.var(cx_Oracle.CLOB)
status = cur.var(cx_Oracle.STRING)
oracle_return = cur.callproc('project.getsomejson',[ID,json_out,message,status])
However, it fails and returns
PLS-00306: wrong number or types of arguments in call to 'getsomejson'
ORA-06550: line 1, column 7:
PL/SQL: Statement ignored
This is the procedure definition
procedure callOracle(json_in clob,json_out out clob,message out clob,status out varchar2)
This is calling an oracle 12c db
I'm really not familiar at all with calling procedures in python. Typically I callfunc with the stored type and just get a return
The procedure is expecting
CLOB, CLOB, CLOB, VARCHAR2
but you are passing
VARCHAR2, CLOB, CLOB, VARCHAR2
The name you gave (callOracle) also doesn't match what you are calling in Python (project.getsomejson). Perhaps verify that you have the right procedure signature? Assuming it is correct, though, you'll need to change the first one to be a CLOB as well or change the stored procedure to accept VARCHAR2. Something like this should do it:
json_in = conn.createlob(cx_Oracle.DB_TYPE_CLOB)
json_in.write("51858645")
json_out_var = cur.var(cx_Oracle.DB_TYPE_CLOB)
message_var = cur.var(cx_Oracle.DB_TYPE_CLOB)
status_var = cur.var(str)
cur.callproc(json_in, json_out, message, status)
I am trying to go through each query in a SQL file and execute it in my python script using psycopg2. Each query has an id which I replace before executing.
The first query in the sql file is the following:
select * from subscriber where org_id = '1111111111';
I get the old id and replace it with the new id that I am looking for
id_regex = re.compile("\d{10,}")
m = id_regex.search(q)
old_id = m.group(0)
new_q = q.replace(old_id, new_id)
I then execute the queries on the following manner
for index, cmd in enumerate(cmds):
# ... (other stuff here)
elif cmd != '\n':
new_cmd = p_helper.replace_id(org_id, cmd)
logger.debug("Running Command:\n" + new_cmd)
try:
if not test_run:
db_cursor.execute(new_cmd)
except psycopg2.Error as e:
logger.error(e.pgerror)
else:
pass
# DO NOTHING
When I run my program I get the following error:
ERROR: syntax error at or near "select"
LINE 1: select * from subscriber where org_id = '9999999999';
^
Every query after the first doesn't run
ERROR: current transaction is aborted, commands ignored until end of transaction block
I ran the select query manually in psql and it worked perfectly so I don't think the problem is the syntax of the statement. I think it has something to do with the formatting of queries that psycopg2 takes. I'm not sure exactly what to change, I have looked at other SO posts and could not figure out what I needed to change. It'd be great if someone could help me figure this out. Thanks!
Versions
python: 2.7.6
psycopg2: 2.4.5
My simple test code is listed below. I created the table already and can query it using the SQLite Manager add-in on Firefox so I know the table and data exist. When I run the query in python (and using the python shell) I get the no such table error
def TroyTest(self, acctno):
conn = sqlite3.connect('TroyData.db')
curs = conn.cursor()
v1 = curs.execute('''
SELECT acctvalue
FROM balancedata
WHERE acctno = ? ''', acctno)
print v1
conn.close()
When you pass SQLite a non-existing path, it'll happily open a new database for you, instead of telling you that the file did not exist before. When you do that, it'll be empty and you'll instead get a "No such table" error.
You are using a relative path to the database, meaning it'll try to open the database in the current directory, and that is probably not where you think it is..
The remedy is to use an absolute path instead:
conn = sqlite3.connect('/full/path/to/TroyData.db')
You need to loop over the cursor to see results:
curs.execute('''
SELECT acctvalue
FROM balancedata
WHERE acctno = ? ''', acctno)
for row in curs:
print row[0]
or call fetchone():
print curs.fetchone() # prints whole row tuple
The problem is the SQL statment. you must specify the db name and after the table name...
'''SELECT * FROM db_name.table_name WHERE acctno = ? '''