We Keep Coding

What is the difference between building cypher and executing the prepared statement in the following code

I am unable to understand why there are two queries being executed. First we are executing the prepared statement and we are using the build cypher function. The code can be found here
https://github.com/apache/age/blob/master/drivers/python/age/age.py
def execCypher(conn:ext.connection, graphName:str, cypherStmt:str, cols:list=None, params:tuple=None) -> ext.cursor :
if conn == None or conn.closed:
raise _EXCEPTION_NoConnection
cursor = conn.cursor()
#clean up the string for modification
cypherStmt = cypherStmt.replace("\n", "")
cypherStmt = cypherStmt.replace("\t", "")
cypher = str(cursor.mogrify(cypherStmt, params))
cypher = cypher[2:len(cypher)-1]
preparedStmt = "SELECT * FROM age_prepare_cypher({graphName},{cypherStmt})"
cursor = conn.cursor()
try:
cursor.execute(sql.SQL(preparedStmt).format(graphName=sql.Literal(graphName),cypherStmt=sql.Literal(cypher)))
except SyntaxError as cause:
conn.rollback()
raise cause
except Exception as cause:
conn.rollback()
raise SqlExecutionError("Execution ERR[" + str(cause) +"](" + preparedStmt +")", cause)
stmt = buildCypher(graphName, cypher, cols)
cursor = conn.cursor()
try:
cursor.execute(stmt)
return cursor
except SyntaxError as cause:
conn.rollback()
raise cause
except Exception as cause:
conn.rollback()
raise SqlExecutionError("Execution ERR[" + str(cause) +"](" + stmt +")", cause)

Both statements perform the same operation.
The difference is that preparedStmt and buildCypher function use different form of cypher queries as shown in code. (cypherStmt & cypher) And their code for building the query is a bit different.

I can't tell you why it's done this way but I'll show you why it's different. Also apologies but I'm not used to Python or C.
The preparedStatement is calling a custom postgres function age_prepare_cypher in this file here apache/age/src/backend/utils/adt/age_session_info.c, which calls set_session_info(graph_name_str, cypher_statement_str);.
And the set_session_info in this file here apache/age/src/backend/utils/adt/age_session_info.c just sets it to a global variable session_info_cypher_statement.
So your graph name and query are being set in the session.
There's another function that gets your graph name and query back out of the session, and that is the convert_cypher_to_subquery. It only gets them out if is_session_info_prepared() is true, and only if graph_name and query_str provided to it are NULL.
Seems strange right? But now let's look at this bit of the python buildCypher function code:
stmtArr = []
stmtArr.append("SELECT * from cypher(NULL,NULL) as (")
stmtArr.append(','.join(columnExp))
stmtArr.append(");")
return "".join(stmtArr)
It's taking your query and saying your graph name and query string are NULL.
So we can conclude that the prepare statement is storing those values in session memory, and then when you execute your statement after using buildCypher, it's getting them out of memory and completing the statement again.
I can't explain exactly why or how it does it, but I can see a chunk of test sql in the project that is doing the same sort of thing here:
-- should return true and execute cypher command
SELECT * FROM age_prepare_cypher('analyze', 'MATCH (u) RETURN (u)');
SELECT * FROM cypher(NULL, NULL) AS (result agtype);
So tl;dr, executing the prepareStatement is storing it in session memory, and executing the normal statement after running it through buildCypher is grabbing what was just stored in the session.

Are spatial queries possible using Python and cx_Oracle?

I am trying to execute a spatial query on an Oracle spatial table via python using the cx_Oracle package.
I can make generic queries successfully, but when I try a spatial query it results in errors.
This is what I have tried:
import cx_Oracle
...
lon = -120.494352
lat = 36.585289
# open a connection to oracle
con = cx_Oracle.connect('myuser/mypass#spatialdb')
# create a cursor
cur = con.cursor()
# Create and populate Oracle objects
typeObj = con.gettype("MDSYS.SDO_GEOMETRY")
elementInfoTypeObj = con.gettype("MDSYS.SDO_ELEM_INFO_ARRAY")
ordinateTypeObj = con.gettype("MDSYS.SDO_ORDINATE_ARRAY")
obj = typeObj.newobject()
obj.SDO_GTYPE = 2001
obj.SDO_SRID = 8307
obj.SDO_ELEM_INFO = elementInfoTypeObj.newobject()
obj.SDO_ELEM_INFO.extend([1, 1, 1])
obj.SDO_ORDINATES = ordinateTypeObj.newobject()
obj.SDO_ORDINATES.extend([lon, lat])
print("Created object", obj)
# set up a distance-calculating sql statement
sql = "select id into :id from spatialtbl s where sdo_nn(s.geometry, :obj, 'sdo_num_res=1', 1) = 'TRUE'"
try:
# execute the distance sql
cur.execute(sql, id=id, obj=obj)
print(f'The id is {id.getvalue()}')
except cx_Oracle.Error as error:
print(error)
which results in the error:
ORA-01036: illegal variable name/number
Can anyone tell me what I may be doing wrong code-wise or if spatial queries are even possible using Python and cx_Oracle? The cx_Oracle documentation doesn't specifically address this as far as I can tell/find.

There is a brief mention in the documentation:
Binding Spatial Datatypes
Here are two examples from the cx_Oracle source code repository:
InsertGeometry.py
SpatialToGeoPandas.py
Here's a presentation from the recent Oracle Conference:
Analyzing Location-based Patterns with Python and Oracle Database
The download links are there but may not be obvious pdf and zip.
In your example, you probably need to do at least id = cursor.var(int), see Bind Direction so cx_Oracle knows what to do with the value you are getting from the DB.

I think the "select into" was the problem (reserved for pl/sql?).
By doing the following I was able to obtain the answer:
# set up a distance-calculating sql statement
sql = """select id from spatialtbl s where sdo_nn(s.geometry, :ob, 'sdo_num_res=1', 1) = 'TRUE'"""
try:
# execute the distance sql
cur.execute(sql, ob=obj)
id = cur.fetchone()
print(f'The id is {id}')
except cx_Oracle.Error as error:
print(error)

How do I prevent an SQL database from being locked when using UPDATE?

I have a program that uses a couple sql databases to store data. I have a class that manages the various sql functions, such as getting a value, an entire table or just updating a value. All of the processes work fine until I run a function that uses UPDATE. I execute an UPDATE command and try to commit the change and the database is always locked. Every function I have in my custom sql class has
cursor.close
database.close
So there shouldn't be any issue with the database connection still being open. Am I missing something in this syntax that is not connecting to the database correctly? I used the extra print statements in an attempt to find out where the problem is occurring, so those can be ignored.
import sqlite3 as db
import os
databaseName = "site"
class MassDb:
def __init__(self,databaseName):
super(MassDb, self).__init__()
print("Current Directory: ",os.getcwd())
self.databaseName = databaseName
def updateValue(self, location, metric, input_value):
print("OPEN CONNECTION UPDATE - running updateValue: ",location, metric, input_value)
if self.databaseName == "site":
try:
siteConn = db.connect("site_data.db")
siteCursor = siteConn.cursor()
siteCursor.execute("UPDATE sites SET " + metric + " = ? WHERE LOCATI ON = ?", (input_value, location))
siteConn.commit()
except:
print("UPDATE FAILED")
finally:
siteCursor.close
siteConn.close
elif self.databaseName == "comp":
try:
compConn = db.connect("comp_data.db")
compCursor = compConn.cursor()
compCursor.execute("UPDATE competitors SET " + metric + " = ? WHERE NAME = ?", (input_value, location))
compConn.commit()
except:
print("UPDATE FAILED")
finally:
compCursor.close
compConn.close
print("CLOSED CONNECTION UPDATE - Update Connection Closed")
else:
print("Update Error")
MassDb("site").updateValue("Location", "CURRENT_SCORE", "100")

As #roganjosh commented, my problem was that I wasn't properly closing the database. If
commit()
is used, there's no need to close the database. However,
cursor.close()
and
conn.close()
need to be written as such. Leaving off the parentheses would be as though an attribute is being referenced, rather than a method. In order to execute the close method, the () must be present. Seems obvious now, but I wasn't aware at the time. Hopefully this can help someone else if they too run across this.
Additionally, using a context manager works and eliminates the need to use close()
with conn:
#do stuff here
commit()

How to create permanent MS Access Query by Python 3.5.1?

I have about 40 MS Access Databases and have some troubles if need to create or transfer one of MS Access Query (like object) from one db to other dbs.
So I tried to solve this problem with pyodbc but.. as I saw pyodbc doesn't support to create new, permanent MS Access Query (object).
I can connect to db, create or delete tables/rows but can't to create and save new query.
import pyodbc
odbc_driver = r"{Microsoft Access Driver (*.mdb, *.accdb)}"
db_test1 = r'''..\Test #1.accdb'''
db_test2 = r'''..\Test #2.accdb'''
db_test3 = r'''..\Test #3.accdb'''
db_test4 = r'''..\Test #4.accdb'''
db_test_objects = [db_test1, db_test2, db_test3, db_test4]
odbc_conn_str = "Driver=%s;DBQ=%s;" % (odbc_driver, db_file)
print (odbc_conn_str)
conn = pyodbc.connect(odbc_conn_str)
odbc_cursor = conn.cursor()
NewQuery = "CREATE TABLE TestTable(symbol varchar(15), leverage double)"
odbc_cursor.execute(NewQuery)
conn.commit()
conn.close()
SO, How to create and save MS Access Query like objects from python?
I tried to search info in Google, but the answers were related with Run SQL code.
On VBA this code looks like:
Public Sub CreateQueryDefX()
Dim base(1 To 4) As String
base(1) = "..\Test #1.accdb"
base(2) = "..\Test #2.accdb"
base(3) = "..\Test #3.accdb"
base(4) = "..\Test #4.accdb"
For i = LBound(base) To UBound(base)
CurrentBase = base(i)
Set dbo = OpenDatabase(CurrentBase)
With dbo
Set QueryNew = .CreateQueryDef("TestQuery", _
"SELECT * FROM TestTable")
RefreshDatabaseWindow
.Close
End With
Next i
RefreshDatabaseWindow
End Sub
Sorry for my English, it's not my native :)
By the way, I know how to solve this by VBA, but I'm interested in solve this by python.
Thank you.

You can use a CREATE VIEW statement to create a saved Select Query in Access. The pyodbc equivalent to your VBA example would be
crsr = conn.cursor()
sql = """\
CREATE VIEW TestQuery AS
SELECT * FROM TestTable
"""
crsr.execute(sql)
To delete that saved query you could simply execute a DROP VIEW statement.
For more information on DDL in Access see
Data Definition Language

Consider the Python equivalent of the VBA running exactly what VBA uses: a COM interface to the Access Object library. With Python's win32com third-party module, you can call the CreateQueryDef method. Do note: this COM interfacing can be applied in other languages such as PHP and R!
Below uses a try/except/finally block to ensure the Access application process closes regardless of error or success of code (similar to VBA's On Error handling):
import win32com.client
# OPEN ACCESS APP AND DATABASE
dbases = ["..\Test #1.accdb", "..\Test #2.accdb", "..\Test #3.accdb", "..\Test #4.accdb"]
try:
oApp = win32com.client.Dispatch("Access.Application")
# CREATE QUERYDEF
for db in dbases:
oApp.OpenCurrentDatabase(db)
currentdb = oApp.CurrentDb()
currentdb.CreateQueryDef("TestQuery", "SELECT * FROM TestTable")
currentdb = None
oApp.DoCmd.CloseDatabase
except Exception as e:
print(e)
finally:
currentdb = None
oApp.Quit
oApp = None
Also, if you need to run DML statements via pyodbc and not a COM interface, consider distributed queries as Access can query other databases directly in SQL. Below should work in Python (be sure to escape the backslash):
SELECT t.* FROM [C:\Path\To\Other\Database.accdb].TestTable t

mysql command out of sync when executing insert from celery

I am running in to the dreaded MySQL Commands out of Sync when using a custom DB library and celery.
The library is as follows:
import pymysql
import pymysql.cursors
from furl import furl
from flask import current_app
class LegacyDB:
"""Db
Legacy Database connectivity library
"""
def __init__(self,app):
with app.app_context():
self.rc = current_app.config['RAVEN']
self.logger = current_app.logger
self.data = {}
# setup Mysql
try:
uri = furl(current_app.config['DBCX'])
self.dbcx = pymysql.connect(
host=uri.host,
user=uri.username,
passwd=uri.password,
db=str(uri.path.segments[0]),
port=int(uri.port),
cursorclass=pymysql.cursors.DictCursor
)
except:
self.rc.captureException()
def query(self, sql, params = None, TTL=36):
# INPUT 1 : SQL query
# INPUT 2 : Parameters
# INPUT 3 : Time To Live
# OUTPUT : Array of result
# check that we're still connected to the
# database before we fire off the query
try:
db_cursor = self.dbcx.cursor()
if params:
self.logger.debug("%s : %s" % (sql, params))
db_cursor.execute(sql,params)
self.dbcx.commit()
else:
self.logger.debug("%s" % sql)
db_cursor.execute(sql)
self.data = db_cursor.fetchall()
if self.data == None:
self.data = {}
db_cursor.close()
except Exception as ex:
if ex[0] == "2006":
db_cursor.close()
self.connect()
db_cursor = self.dbcx.cursor()
if params:
db_cursor.execute(sql,params)
self.dbcx.commit()
else:
db_cursor.execute(sql)
self.data = db_cursor.fetchall()
db_cursor.close()
else:
self.rc.captureException()
return self.data
The purpose of the library is to work alongside SQLAlchemy whilst I migrate a legacy database schema from a C++-based system to a Python based system.
All configuration is done via a Flask application and the app.config['DBCX'] value reads the same as a SQLAlchemy String ("mysql://user:pass#host:port/dbname") allowing me to easily switch over in future.
I have a number of tasks that run "INSERT" statements via celery, all of which utilise this library. As you can imagine, the main reason for running Celery is so that I can increase throughput on this application, however I seem to be hitting an issue with the threading in my library or the application as after a while (around 500 processed messages) I see the following in the logs:
Stacktrace (most recent call last):
File "legacy/legacydb.py", line 49, in query
self.dbcx.commit()
File "pymysql/connections.py", line 662, in commit
self._read_ok_packet()
File "pymysql/connections.py", line 643, in _read_ok_packet
raise OperationalError(2014, "Command Out of Sync")
I'm obviously doing something wrong to hit this error, however it doesn't seem to matter whether MySQL has autocommit enabled/disabled or where I place my connection.commit() call.
If I leave out the connection.commit() then I don't get anything inserted into the database.
I've recently moved from mysqldb to pymysql and the occurrences appear to be lower, however given that these are simple "insert" commands and not a complicated select (there aren't even any foreign key constraints on this database!) I'm struggling to work out where the issue is.
As things stand at present, I am unable to use executemany as I cannot prepare the statements in advance (I am pulling data from a "firehose" message queue and storing it locally for later processing).

First of all, make sure that the celery thingamajig uses its own connection(s) since
>>> pymysql.threadsafety
1
Which means: "threads may share the module but not connections".

Is the init called once, or per-worker? If only once, you need to move the initialisation.
How about lazily initialising the connection in a thread-local variable the first time query is called?