We Keep Coding

SQLAlchemy 1.4 tutorial code "'Connection' object has no attribute 'commit'" error or does not commit changes

Here is some custom code I wrote that I think might be problematic for this particular use case.
class SQLServerConnection:
def __init__(self, database):
...
self.connection_string = \
"DRIVER=" + str(self.driver) + ";" + \
"SERVER=" + str(self.server) + ";" + \
"DATABASE=" + str(self.database) + ";" + \
"Trusted_Connection=yes;"
self.engine = sqlalchemy.create_engine(
sqlalchemy.engine.URL.create(
"mssql+pyodbc", \
query={'odbc_connect': self.connection_string}
)
)
# Runs a command and returns in plain text (python list for multiple rows)
# Can be a select, alter table, anything like that
def execute(self, command, params=False):
# Make a connection object with the server
with self.engine.connect() as conn:
# Can send some parameters along with a plain text query...
# could be single dict or list of dict
# Doc: https://docs.sqlalchemy.org/en/14/tutorial/dbapi_transactions.html#sending-multiple-parameters
if params:
output = conn.execute(sqlalchemy.text(command,params))
else:
output = conn.execute(sqlalchemy.text(command))
# Tell SQL server to save your changes (assuming that is applicable, is not with select)
# Doc: https://docs.sqlalchemy.org/en/14/tutorial/dbapi_transactions.html#committing-changes
try:
conn.commit()
except Exception as e:
#pass
warn("Could not commit changes...\n" + str(e))
# Try to consolidate select statement result into single object to return
try:
output = output.all()
except:
pass
return output
If I try:
cnxn = SQLServerConnection(database='MyDatabase')
cnxn.execute("SELECT * INTO [dbo].[MyTable_newdata] FROM [dbo].[MyTable] ")
or
cnxn.execute("SELECT TOP 0 * INTO [dbo].[MyTable_newdata] FROM [dbo].[MyTable] ")
Python returns this object without error, <sqlalchemy.engine.cursor.LegacyCursorResult at 0x2b793d71880>, but upon looking in MS SQL Server, the new table was not generated. I am not warned about the commit step failing with the SELECT TOP 0 way; I am warned ('Connection' object has no attribute 'commit') in the above way.
CREATE TABLE, ALTER TABLE, or SELECT (etc) appears to work fine, but SELECT * INTO seems to not be working, and I'm not sure how to troubleshoot further. Copy-pasting the query into SQL Server and running appears to work fine.

As noted in the introduction to the 1.4 tutorial here:
A Note on the Future
This tutorial describes a new API that’s released in SQLAlchemy 1.4 known as 2.0 style. The purpose of the 2.0-style API is to provide forwards compatibility with SQLAlchemy 2.0, which is planned as the next generation of SQLAlchemy.
In order to provide the full 2.0 API, a new flag called future will be used, which will be seen as the tutorial describes the Engine and Session objects. These flags fully enable 2.0-compatibility mode and allow the code in the tutorial to proceed fully. When using the future flag with the create_engine() function, the object returned is a subclass of sqlalchemy.engine.Engine described as sqlalchemy.future.Engine. This tutorial will be referring to sqlalchemy.future.Engine.
That is, it is assumed that the engine is created with
engine = create_engine(connection_url, future=True)
You are getting the "'Connection' object has no attribute 'commit'" error because you are creating an old-style Engine object.
You can avoid the error by adding future=True to your create_engine() call:
self.engine = sqlalchemy.create_engine(
sqlalchemy.engine.URL.create(
"mssql+pyodbc",
query={'odbc_connect': self.connection_string}
),
future=True
)

Use this recipe instead:
#!python
from sqlalchemy.sql import Select
from sqlalchemy.ext.compiler import compiles
class SelectInto(Select):
def __init__(self, columns, into, *arg, **kw):
super(SelectInto, self).__init__(columns, *arg, **kw)
self.into = into
#compiles(SelectInto)
def s_into(element, compiler, **kw):
text = compiler.visit_select(element)
text = text.replace('FROM',
'INTO TEMPORARY TABLE %s FROM' %
element.into)
return text
if __name__ == '__main__':
from sqlalchemy.sql import table, column
marker = table('marker',
column('x1'),
column('x2'),
column('x3')
)
print SelectInto([marker.c.x1, marker.c.x2], "tmp_markers").\
where(marker.c.x3==5).\
where(marker.c.x1.in_([1, 5]))
This needs some tweaking, hence it will replace all subquery selects as select INTOs, but test it for now, if it worked it would be better than raw text statments.

Have you tried this from this answer by #Michael Berkowski:
INSERT INTO assets_copy
SELECT * FROM assets;
The answer states that MySQL documentation states that SELECT * INTO isn't supported.

Is there a proper way to handle cursors returned from a postgresql function in psycopg?

I'm trying to make friends with postgresql (14.0 build 1914 64-bit on windows), psycopg2 (2.9.1 installed using pip) and python 3.8.10 on windows.
I have created a postgresql function in a database that returns a cursor, somthing like below
CREATE get_rows
...
RETURNS refcursor
...
DECLARE
res1 refcursor;
BEGIN
OPEN res1 FOR
SELECT some_field, and_another_field FROM some_table;
RETURN res1;
END
The function can be run from pgAdmin4 Quert tool
SELECT get_rows();
and will then return a cursor like "<unnamed portal 1>"
Still within query tool in pgAdmin4 I can issue:
BEGIN;
SELECT get_rows();
FETCH ALL IN "<unnamed portal 2>"; -- Adjust counter number
And this will get me the rows returned by the cursor.
Now I want to replicate this using psycopg instead of pgAdmin4
I have the below code
conn = psycopg2.connect("dbname='" + db_name + "' "\
"user='" + db_user + "' " +\
"host='" + db_host + "' "+\
"password='" + db_passwd + "'")
cursor = conn.cursor()
cursor.callproc('get_rows')
print("cursor.description: ", end = '')
print(cursor.description)
for record in cursor:
print("record: ", end = '')
print (record)
The above code only gives the cursor string name (as returned by the postgresql function 'get_rows') in the single record of the cursor created by psycopg.
How can I get a cursor-class object from psycopg that provides access the cursor returned by 'get_rows'?
https://www.psycopg.org/docs/cursor.html says cursor.name is read-only and I dont see an obvious way to connect the cursor from 'get_rows' with a psycopg cursor-instance

The cursor link you show refers to the Python DB API cursor not the Postgres one. There is an example of how to do what you want here Server side cursor in section:
Note It is also possible to use a named cursor to consume a cursor created in some other way than using the DECLARE executed by execute(). For example, you may have a PL/pgSQL function returning a cursor:
CREATE FUNCTION reffunc(refcursor) RETURNS refcursor AS $$
BEGIN
OPEN $1 FOR SELECT col FROM test;
RETURN $1;
END;
$$ LANGUAGE plpgsql;
You can read the cursor content by calling the function with a regular, non-named, Psycopg cursor:
cur1 = conn.cursor()
cur1.callproc('reffunc', ['curname'])
and then use a named cursor in the same transaction to “steal the cursor”:
cur2 = conn.cursor('curname')
for record in cur2: # or cur2.fetchone, fetchmany...
# do something with record
pass
UPDATE
Be sure and close the named cursor(cur2) to release the server side cursor. So:
cur2.close()

Are spatial queries possible using Python and cx_Oracle?

I am trying to execute a spatial query on an Oracle spatial table via python using the cx_Oracle package.
I can make generic queries successfully, but when I try a spatial query it results in errors.
This is what I have tried:
import cx_Oracle
...
lon = -120.494352
lat = 36.585289
# open a connection to oracle
con = cx_Oracle.connect('myuser/mypass#spatialdb')
# create a cursor
cur = con.cursor()
# Create and populate Oracle objects
typeObj = con.gettype("MDSYS.SDO_GEOMETRY")
elementInfoTypeObj = con.gettype("MDSYS.SDO_ELEM_INFO_ARRAY")
ordinateTypeObj = con.gettype("MDSYS.SDO_ORDINATE_ARRAY")
obj = typeObj.newobject()
obj.SDO_GTYPE = 2001
obj.SDO_SRID = 8307
obj.SDO_ELEM_INFO = elementInfoTypeObj.newobject()
obj.SDO_ELEM_INFO.extend([1, 1, 1])
obj.SDO_ORDINATES = ordinateTypeObj.newobject()
obj.SDO_ORDINATES.extend([lon, lat])
print("Created object", obj)
# set up a distance-calculating sql statement
sql = "select id into :id from spatialtbl s where sdo_nn(s.geometry, :obj, 'sdo_num_res=1', 1) = 'TRUE'"
try:
# execute the distance sql
cur.execute(sql, id=id, obj=obj)
print(f'The id is {id.getvalue()}')
except cx_Oracle.Error as error:
print(error)
which results in the error:
ORA-01036: illegal variable name/number
Can anyone tell me what I may be doing wrong code-wise or if spatial queries are even possible using Python and cx_Oracle? The cx_Oracle documentation doesn't specifically address this as far as I can tell/find.

There is a brief mention in the documentation:
Binding Spatial Datatypes
Here are two examples from the cx_Oracle source code repository:
InsertGeometry.py
SpatialToGeoPandas.py
Here's a presentation from the recent Oracle Conference:
Analyzing Location-based Patterns with Python and Oracle Database
The download links are there but may not be obvious pdf and zip.
In your example, you probably need to do at least id = cursor.var(int), see Bind Direction so cx_Oracle knows what to do with the value you are getting from the DB.

I think the "select into" was the problem (reserved for pl/sql?).
By doing the following I was able to obtain the answer:
# set up a distance-calculating sql statement
sql = """select id from spatialtbl s where sdo_nn(s.geometry, :ob, 'sdo_num_res=1', 1) = 'TRUE'"""
try:
# execute the distance sql
cur.execute(sql, ob=obj)
id = cur.fetchone()
print(f'The id is {id}')
except cx_Oracle.Error as error:
print(error)

Error: Increase MaxLocksPerFile registry entry via Python

I am running a rather complec update MS Access query from Python:
qry = '''
UPDATE H500_ODFlows INNER JOIN H500_UPDATE ON
(H500_ODFlows.Product = H500_UPDATE.Product)
AND (H500_ODFlows.Dest = H500_UPDATE.DestCode)
AND (H500_ODFlows.Orig = H500_UPDATE.OrigCode)
SET H500_ODFlows.Pieces = [H500_UPDATE].[Pieces],
H500_ODFlows.Weight = [H500_UPDATE].[Weight],
H500_ODFlows.Cons = [H500_UPDATE].[Pieces],
H500_ODFlows.DeadWeight = [H500_UPDATE].[DeadWeight],
H500_ODFlows.DoNotRead = [H500_UPDATE].DoNotRead,
H500_ODFlows.[_OrigCountryCode] = [H500_UPDATE].[_OrigCountryCode],
H500_ODFlows.[_DestCountryCode] = [H500_UPDATE].[_DestCountryCode]
'''
try:
crsr.execute(lb.cleanqry(qry))
cnxn.commit()
print('Updating was successful.')
except Exception as err:
print('Updating failed. See the error.' + str(err))
but get the following error:
('HY000', '[HY000] [Microsoft][ODBC Microsoft Access Driver] File
sharing lock count exceeded. Increase MaxLocksPerFile registry entry.
(-1033) (SQLExecDirectW)')
I followed the instructions to increase "MaxLocksPerFile" but it is not helping. Moreover, the query runs in MS Access quite OK but not through Python. Any advice?

Try running the query with autocommit on. That way, the database won't need to keep all those locks open, but can just commit everything as the query runs.
qry = '''
UPDATE H500_ODFlows INNER JOIN H500_UPDATE ON
(H500_ODFlows.Product = H500_UPDATE.Product)
AND (H500_ODFlows.Dest = H500_UPDATE.DestCode)
AND (H500_ODFlows.Orig = H500_UPDATE.OrigCode)
SET H500_ODFlows.Pieces = [H500_UPDATE].[Pieces],
H500_ODFlows.Weight = [H500_UPDATE].[Weight],
H500_ODFlows.Cons = [H500_UPDATE].[Pieces],
H500_ODFlows.DeadWeight = [H500_UPDATE].[DeadWeight],
H500_ODFlows.DoNotRead = [H500_UPDATE].DoNotRead,
H500_ODFlows.[_OrigCountryCode] = [H500_UPDATE].[_OrigCountryCode],
H500_ODFlows.[_DestCountryCode] = [H500_UPDATE].[_DestCountryCode]
'''
try:
cnxn.autocommit = True
crsr.execute(lb.cleanqry(qry))
print('Updating was successful.')
except Exception as err:
print('Updating failed. See the error.' + str(err))

Since you note: The query runs in MS Access quite OK but not through Python. One possible reason for this is Access stored queries are more efficient than application layer called queries since the engine saves and caches best execution plan. In the application layer (Python, VBA, etc.) when processing a string SQL statement, the Jet/ACE engine does not have time to plan the best execution.
Therefore, consider the following:
Add any needed indexes to JOIN variables of respective tables.
Save your UPDATE query as a stored query inside the database. Here, the saving process checks syntax, calculates and optimizes plan, and cache stats.
Run Compact & Repair in database to refresh stats.
Then, run query in Python as a stored proc with CALL command:
# SET AUTOCOMMIT PREFERENCE IN CONNECTION
cnxn = pyodbc.connect(..., autocommit=True)
...
crsr.execute("{CALL myUpdateQuery}")

using python 2.7 to query sqlite3 database and getting "sqlite3 operational error no such table"

My simple test code is listed below. I created the table already and can query it using the SQLite Manager add-in on Firefox so I know the table and data exist. When I run the query in python (and using the python shell) I get the no such table error
def TroyTest(self, acctno):
conn = sqlite3.connect('TroyData.db')
curs = conn.cursor()
v1 = curs.execute('''
SELECT acctvalue
FROM balancedata
WHERE acctno = ? ''', acctno)
print v1
conn.close()

When you pass SQLite a non-existing path, it'll happily open a new database for you, instead of telling you that the file did not exist before. When you do that, it'll be empty and you'll instead get a "No such table" error.
You are using a relative path to the database, meaning it'll try to open the database in the current directory, and that is probably not where you think it is..
The remedy is to use an absolute path instead:
conn = sqlite3.connect('/full/path/to/TroyData.db')
You need to loop over the cursor to see results:
curs.execute('''
SELECT acctvalue
FROM balancedata
WHERE acctno = ? ''', acctno)
for row in curs:
print row[0]
or call fetchone():
print curs.fetchone() # prints whole row tuple

The problem is the SQL statment. you must specify the db name and after the table name...
'''SELECT * FROM db_name.table_name WHERE acctno = ? '''