We Keep Coding

What is this ZopeTransactionEvents error with SQLAlchemy while updating a Pyramid application?

I'm updating Pyramid/SQLAlchemy legacy code to Python 3.8 from an app working fine under Python 2.7, and running it locally. All the necessary requirements are pip installed and setup.py runs without error.
On running initialise with my local .ini file, All goes well, the database tables (MariaDB) are all written.
in models.py
from sqlalchemy.orm import (
scoped_session,
sessionmaker,
relationship,
backref,
synonym,
)
from zope.sqlalchemy import ZopeTransactionEvents
#[...]
DBSession = scoped_session(sessionmaker(extension=ZopeTransactionEvents()))
in the main app it fails with 'ZopeTransactionEvents' object has no attribute 'after_commit' at this function, after getting the final input and attempting to add it to the DB at DBSession.add(user):
def do_create_admin_user(self):
from ..models import User
from getpass import getpass
print("Create an administrative user")
fullname = input("full name: ")
username = input("username: ")
if not username:
self.log.info("missing username - aborted")
return
if len(username) > 50:
self.log.info("username too long - aborted")
return
password = getpass("password for {}: ".format(username))
with transaction.manager:
user = User(
username=username,
fullname=fullname,
administrator=True,
password=password
)
DBSession.add(user)
self.log.info("{} created".format(username))
Here are the two key parts of the stack trace:
Traceback (most recent call last):
"[...]sqlalchemy/util/_collections.py", line 1055, in __call__
return self.registry.value
AttributeError: '_thread._local' object has no attribute 'value'
During handling of the above exception, another exception occurred:
[cruft omitted]
"[...]sqlalchemy/orm/deprecated_interfaces.py", line 367, in _adapt_listener
ls_meth = getattr(listener, meth)
AttributeError: 'ZopeTransactionEvents' object has no attribute 'after_commit'
This specific issue halted the process, and despite days of research (and some unproductive hacking) I'm no closer to a solution. This is a legacy project and I'm not previously familiar with Pyramid or SQAlchemy, so finding my way as I go along.
Fixed
In the end, this is what worked i.e. no arguments to sessionmaker()
from zope.sqlalchemy import register
# ...
DBSession = scoped_session(sessionmaker())
register(DBSession)
Now on to the next error.

This is due to a breaking change introduced in zope.sqlalchemy v1.2. See details in the zope.sqlalchemy pypi page
To make things clearer we renamed the ZopeTransactionExtension class to ZopeTransactionEvents. Existing code using the ‘register’ version stays compatible.
To upgrade from 1.1
Your old code like this:
from zope.sqlalchemy import ZopeTransactionExtension
DBSession = scoped_session(sessionmaker(extension=ZopeTransactionExtension(), **options))
becomes:
from zope.sqlalchemy import register
DBSession = scoped_session(sessionmaker(**options))
register(DBSession)
edit
Essentially its a minor breaking change to the zope.sqlalchemy API that was introduced in version 1.2, this is now the way. Rather than registering the extension via keyword argument, you now register the session explicitly with the register callable.
The above is a direct quote from the documentation. It may not be obvious but the use of **options refers to optional keyword arguments. If you are not using any keyword arguments then this would be omitted (i.e. just call sessionmaker() with no arguments).

Why BINARY usage in SQLAlchemy with Python3 cause a TypeError: 'string argument without an encoding'

I read a lot of similar questions but none of the clearly answer my issue.
I'm using sqlalchemy-utils EncryptedType on a mysql table column.
The table creation and the insert is ok, but when I'm trying to do a query a receive:
Traceback (most recent call last):
File "workspace/bin/test.py", line 127, in <module>
result = session.query(Tester).all()
File "workspace\ERP\venv\lib\site-packages\sqlalchemy\orm\query.py", line 3244, in all
return list(self)
File "workspace\venv\lib\site-packages\sqlalchemy\orm\loading.py", line 101, in instances
cursor.close()
File "workspace\venv\lib\site-packages\sqlalchemy\util\langhelpers.py", line 69, in __exit__
exc_value, with_traceback=exc_tb,
File "workspace\venv\lib\site-packages\sqlalchemy\util\compat.py", line 178, in raise_
raise exception
File "workspace\venv\lib\site-packages\sqlalchemy\orm\loading.py", line 81, in instances
rows = [proc(row) for row in fetch]
File "workspace\venv\lib\site-packages\sqlalchemy\orm\loading.py", line 81, in <listcomp>
rows = [proc(row) for row in fetch]
File "workspace\venv\lib\site-packages\sqlalchemy\orm\loading.py", line 642, in _instance
populators,
File "workspace\venv\lib\site-packages\sqlalchemy\orm\loading.py", line 779, in _populate_partial
dict_[key] = getter(row)
File "workspace\venv\lib\site-packages\sqlalchemy\engine\result.py", line 107, in __getitem__
return processor(self._row[index])
File "workspace\venv\lib\site-packages\sqlalchemy\sql\sqltypes.py", line 944, in process
value = bytes(value)
TypeError: string argument without an encoding
I found out that this error occurs only using python 3, not using python 2.
And also that the problem is with the sqlalchemy bynary type, because I get the same error with Binary, Varbinary, and Blob columns.
Since bytes in python3 needs an encoding for strings, I changed the code of sqlalchemy\sql\sqltypes.py on line 944 to value = bytes(value, 'utf-8) and al works well, so my question is:
why I need to change the sqlalchemy code? Is sqlalchemy fully usable with python3? Is it safe to change the code of a package?
Here is a code sample to try:
from sqlalchemy import MetaData, Integer, Column, Table, Binary, create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker, scoped_session
DB_CONFIG = {
'user': 'user_test',
'password': 'PSW_test',
'host': '127.0.0.1',
'database': 'db_test',
}
if __name__ == '__main__':
Base = declarative_base()
engine = create_engine("mysql+mysqlconnector://%(user)s:%(password)s#%(host)s/%(database)s" % DB_CONFIG,
echo=False)
Base.metadata.bind = engine
db_sessionmaker = sessionmaker(bind=engine)
Session = scoped_session(db_sessionmaker)
# create the table
meta = MetaData()
tests = Table(
'test', meta,
Column('id', Integer, primary_key=True),
Column('attr', Binary)
)
meta.create_all(engine)
class Test(Base):
__tablename__ = 'test'
id = Column(Integer, primary_key=True)
attr = Column(Binary)
new_test = Test(attr='try'.encode('utf-8'))
session = Session()
session.add(new_test)
session.commit()
result = session.query(Test).all()
for a in result:
print(a, a.id, a.attr)
Session.remove()

Thanks to the hint provided by Ilja Everilä, I was able to find a solution. Maybe not the best solution, but now is working.
I think that the root cause is that my DB-API automatically converts bytes to str during the query. So I just disabled this behavior by adding a parameter to the create_engine:
engine = create_engine("mysql+mysqlconnector://%(user)s:%(password)s#%(host)s/%(database)s" % DB_CONFIG, connect_args={'use_unicode': False})
The consequence is that if you have a String column it will be returned in queries as bytes not as 'str', and you have to manually decode it.
Surely there is a better solution.

There does seem to be anything wrong with the MySQL connector. Just switch your mysql-connector-python to mysqlclient. I had the same problem and it helped me.

Instead of mysql+mysqlconnector://<user>:<password>#<host>[:<port>]/<dbname> you'll have mysql+mysqldb://<user>:<password>#<host>[:<port>]/<dbname>
The SQLAchemy docs suggests using mysqlclient (fork of MySQL-Python)¶ over MySQL-Connector¶.
The MySQL Connector/Python DBAPI has had many issues since its release, some of which may remain unresolved, and the mysqlconnector dialect is not tested as part of SQLAlchemy’s continuous integration. The recommended MySQL dialects are mysqlclient and PyMySQL.

Does Context/Scoping of a SQLAlchemy Session Require Non-Automatic Object/Attribute Expiration?

The Situation: Simple Class with Basic Attributes
In an application I'm working on, instances of particular class are persisted at the end of their lifecycle, and while they are not subsequently modified, their attributes may need to be read. For example, the end_time of the instance or its ordinal position relative to other instances of the same class (first instance initialized gets value 1, the next has value 2, etc.).
class Foo(object):
def __init__(self, position):
self.start_time = time.time()
self.end_time = None
self.position = position
# ...
def finishFoo(self):
self.end_time = time.time()
self.duration = self.end_time - self.start_time
# ...
The Goal: Persist an Instance using SQLAlchemy
Following what I believe to be a best practice - using a scoped SQLAlchemy Session, as suggested here, by way of contextlib.contextmanager - I save the instance in a newly-created Session which immediately commits. The very next line references the newly persistent instance by mentioning it in a log record, which throws a DetachedInstanceError because the attribute its referencing expired when the Session committed.
class Database(object):
# ...
def scopedSession(self):
session = self.sessionmaker()
try:
yield session
session.commit()
except:
session.rollback()
logger.warn("blah blah blah...")
finally:
session.close()
# ...
def saveMyFoo(self, foo):
with self.scopedSession() as sql_session:
sql_session.add(foo)
logger.info("Foo number {0} finished at {1} has been saved."
"".format(foo.position, foo.end_time))
## Here the DetachedInstanceError is raised
Two Known Possible Solutions: No Expiring or No Scope
I know I can set the expire_on_commit flag to False to circumvent this issue, but I'm concerned this is a questionable practice -- automatic expiration exists for a reason, and I'm hesitant to arbitrarily lump all ORM-tied classes into a non-expiry state without sufficient reason and understanding behind it. Alternatively, I can forget about scoping the Session and just leave the transaction pending until I explicitly commit at a (much) later time.
So my question boils down to this:
Is a scoped/context-managed Session being used appropriately in the case I described?
Is there an alternative way to reference expired attributes that is a better/more preferred approach? (e.g. using a property to wrap the steps of catching expiration/detached exceptions or to create & update a non-ORM-linked attribute that "mirrors" the ORM-linked expired attribute)
Am I misunderstanding or misusing the SQLAlchemy Session and ORM? It seems contradictory to me to use a contextmanager approach when that precludes the ability to subsequently reference any of the persisted attributes, even for a task as simple and broadly applicable as logging.
The Actual Exception Traceback
The example above is simplified to focus on the question at hand, but should it be useful, here's the actual exact traceback produced. The issue arises when str.format() is run in the logger.debug() call, which tries to execute the Set instance's __repr__() method.
Unhandled Error
Traceback (most recent call last):
File "/opt/zenith/env/local/lib/python2.7/site-packages/twisted/python/log.py", line 73, in callWithContext
return context.call({ILogContext: newCtx}, func, *args, **kw)
File "/opt/zenith/env/local/lib/python2.7/site-packages/twisted/python/context.py", line 118, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/opt/zenith/env/local/lib/python2.7/site-packages/twisted/python/context.py", line 81, in callWithContext
return func(*args,**kw)
File "/opt/zenith/env/local/lib/python2.7/site-packages/twisted/internet/posixbase.py", line 614, in _doReadOrWrite
why = selectable.doRead()
--- <exception caught here> ---
File "/opt/zenith/env/local/lib/python2.7/site-packages/twisted/internet/udp.py", line 248, in doRead
self.protocol.datagramReceived(data, addr)
File "/opt/zenith/operations/network.py", line 311, in datagramReceived
self.reactFunction(datagram, (host, port))
File "/opt/zenith/operations/schema_sqlite.py", line 309, in writeDatapoint
logger.debug("Data written: {0}".format(dataz))
File "/opt/zenith/operations/model.py", line 1770, in __repr__
repr_info = "Set: {0}, User: {1}, Reps: {2}".format(self.setNumber, self.user, self.repCount)
File "/opt/zenith/env/local/lib/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 239, in __get__
return self.impl.get(instance_state(instance), dict_)
File "/opt/zenith/env/local/lib/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 589, in get
value = callable_(state, passive)
File "/opt/zenith/env/local/lib/python2.7/site-packages/sqlalchemy/orm/state.py", line 424, in __call__
self.manager.deferred_scalar_loader(self, toload)
File "/opt/zenith/env/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 563, in load_scalar_attributes
(state_str(state)))
sqlalchemy.orm.exc.DetachedInstanceError: Instance <Set at 0x1c96b90> is not bound to a Session; attribute refresh operation cannot proceed

1.
Most likely, yes. It's used correctly insofar as correctly saving data to the database. However, because your transaction only spans the update, you may run into race conditions when updating the same row. Depending on the application, this can be okay.
2.
Not expiring attributes is the right way to do it. The reason the expiration is there by default is because it ensures that even naive code works correctly. If you are careful, it shouldn't be a problem.
3.
It's important to separate the concept of the transaction from the concept of the session. The contextmanager does two things: it maintains the session as well as the transaction. The lifecycle of each ORM instance is limited to the span of each transaction. This is so you can assume the state of the object is the same as the state of the corresponding row in the database. This is why the framework expires attributes when you commit, because it can no longer guarantee the state of the values after the transaction commits. Hence, you can only access the instance's attributes while a transaction is active.
After you commit, any subsequent attribute you access will result in a new transaction being started so that the ORM can once again guarantee the state of the values in the database.
But why do you get an error? This is because your session is gone, so the ORM has no way of starting a transaction. If you do a session.commit() in the middle of your context manager block, you'll notice a new transaction being started if you access one of the attributes.
Well, what if I want to just access the previously-fetched values? Then, you can ask the framework not to expire those attributes.

Editing Python Class in Shell and SQLAlchemy

I'm working on in the terminal on a shell script following this tutorial http://docs.sqlalchemy.org/en/latest/orm/tutorial.html SQLAlchemy tutorial on Declare A Mapping. I needed to type in
>>> from sqlalchemy import Column, Integer, String
>>> class User(Base):
__tablename__ = 'users'
id = Column(Integer, primary_key=True)
name = Column(String)
fullname = Column(String)
password = Column(String)
def __repr__(self):
return "<\User(name='%s', fullname='%s', password='%s')>" % (
self.name, self.fullname, self.password)
Issue is after I typed the password = Column(String) I hit enter twice and the .... changed to >>>. I then retyped everything back in but then an error was thrown because the class already exists... I'm not totally sure how to fix this. How do I open up that class in the shell script and edit it (add in the def repr)
The error thrown is below:
/Users/GaryPeters/TFsqlAlc001/lib/python2.7/site-packages/sqlalchemy/ext/declarative/clsregistry.py:160: SAWarning: This declarative base already contains a class with the same class name and module name as __main__.User, and will be replaced in the string-lookup table.
existing.add_item(cls)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/GaryPeters/TFsqlAlc001/lib/python2.7/site-packages/sqlalchemy/ext/declarative/api.py", line 53, in __init__
_as_declarative(cls, classname, cls.__dict__)
File "/Users/GaryPeters/TFsqlAlc001/lib/python2.7/site-packages/sqlalchemy/ext/declarative/base.py", line 251, in _as_declarative
**table_kw)
File "/Users/GaryPeters/TFsqlAlc001/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 339, in __new__
"existing Table object." % key)
sqlalchemy.exc.InvalidRequestError: Table 'users' is already defined for this MetaData instance. Specify 'extend_existing=True' to redefine options and columns on an existing Table object.

Just close and reopen the shell, and type everything in again, this time making sure to hit enter only once, not twice.
Alternatively, make sure to add the indents whenever you encounter a blank line -- if you hit enter and then hit tab or space the appropriate amount of times so you're indented to the right level, then you should be able to hit enter again without the shell ending your definition and displaying >>> again.
You should also be to redefine the class in the shell, so I'm not quite sure what you mean by "an error was thrown" -- it might be helpful if you were to edit your post to include the specific stack trace.

How to use SQLAlchemy reflection with Sybase? [answer: turns out it's not supported!]

I'm trying to learn more about the .egg concept and overriding methods in Python. Here's the error message I'm receiving:
Traceback (most recent call last):
File "C:/local/work/scripts/plmr/plmr_db.py", line 42, in <module>
insp.reflecttable(reo_daily_table, column_list)
File "build\bdist.win32\egg\sqlalchemy\engine\reflection.py", line 370, in reflecttable
File "build\bdist.win32\egg\sqlalchemy\engine\reflection.py", line 223, in get_columns
File "build\bdist.win32\egg\sqlalchemy\engine\base.py", line 260, in get_columns
NotImplementedError
Here's the specific function from base.py:
def get_columns(self, connection, table_name, schema=None, **kw):
"""Return information about columns in `table_name`.
Given a :class:`.Connection`, a string
`table_name`, and an optional string `schema`, return column
information as a list of dictionaries with these keys:
name
the column's name
type
[sqlalchemy.types#TypeEngine]
nullable
boolean
default
the column's default value
autoincrement
boolean
sequence
a dictionary of the form
{'name' : str, 'start' :int, 'increment': int}
Additional column attributes may be present.
"""
raise NotImplementedError()
So my question is - do I override this function by writing a new method in my main module? Or am I missing a step somewhere along the way with my imports? Or am I just completely off track here?
Any and all help is appreciated :)
edit: adding my code
import sys
from sqlalchemy import create_engine, select, Table, MetaData
from sqlalchemy.engine import reflection
dbPath = 'connection_string'
engine = create_engine(dbPath, echo=True)
connection = engine.connect()
#reflect tables into memory
meta = MetaData()
reo_daily_table = Table('reo_daily',meta)
insp = reflection.Inspector.from_engine(engine)
column_list=[...]
insp.reflecttable(reo_daily_table, column_list)
connection.close()

EDIT:
The Sybase dialect currently lacks the ability to reflect tables.

You have misunderstood completely. You do not need to subclass anything and this problem has nothing to do with eggs and .ini files at all.
You are not supposed to instantiate Inspector this way. If you read
SQLAlchemy docs carefully, you will notice that you are not supposed to use Reflection constructor directly; instead you should write
insp = reflection.Inspector.from_engine(engine)