We Keep Coding

Create CompositeArray of CompositeType without using sqlalchemy_utils

Within my FastAPI, SQLAlchemy, Alembic project, I was using postgresql engine, and now I'm trying to migrate to async with postgresql+asyncpg.
The issue here is that one of my DB schemas has this structure:
class MyTable(...):
__tablename__ = 'my_table'
name = Column(String(255), nullable=False, unique=True)
tridimensional = Column(CompositeArray(
CompositeType(
'tridimensional_type', [
Column('x', Numeric(4, 0), nullable=False, default=0),
Column('y', Numeric(4, 0), nullable=False),
Column('z', Numeric(4, 0), nullable=False),
]
),
),
)
Since this was relying entirely on sqlalchemy_utils.types.pg_composite (Both CompositeArray and CompositeType) and this does not have support to register_composites for postgresql+asyncpg, I was wondering (if possible) how to:
Create my own sqlalchemy.types.UserDefinedType that involves this tridimensional type
Add it to an alembic migration
Create a column using it within an array

So, I managed to figure it out kinda using CompositeType:
I created this two classes:
To access elements from a dict using the dot operator .
class Dotdict(dict):
__getattr__ = dict.get
__setattr__ = dict.__setitem__
__delattr__ = dict.__delitem__
And the inherited class to process it from
class AsyncCompositeType(CompositeType):
# async sessions return asyncpg.Record not the type define in sqlalchemy_utils
# these wrappers returns a dict when the composite type is loaded using an async session
# see https://docs.sqlalchemy.org/en/14/core/custom_types.html#typedecorator-recipes
def result_processor(self, dialect, coltype):
def process(record):
if isinstance(record, asyncpg.Record): # handle async sqlalchemy session
return Dotdict(record)
return record
return process
Finally, your table would only be changed to:
class MyTable(...):
__tablename__ = 'my_table'
name = Column(String(255), nullable=False, unique=True)
tridimensional = Column(CompositeArray(
AsyncCompositeType(
'tridimensional_type', [
Column('x', Numeric(4, 0), nullable=False, default=0),
Column('y', Numeric(4, 0), nullable=False),
Column('z', Numeric(4, 0), nullable=False),
]
),
),
)
Hopefully this helps someone in the future.

I've stumbled upon the same issue and came up with this (probably incomplete and not very thoroughly tested) implementation. It's also based on ideas from sqlalchemy-utils but avoids hard psycopg2 dependency.
Mind that there is no CompositeArray equivalent. It seems that no custom type is actually required for such arrays. The same functionality can be achieved by explicitly declaring dimensions and item type of your ARRAY of composites and SQLAlchemy will properly pass each array entry to result_processor of item's UserDefinedType (and hence to process_result_value of TypeDecorator wrapper).
All in all, the following code demonstrates intended usage (tested on Python 3.11 and SQLAlchemy 1.4):
import asyncio
import dataclasses
import pprint
from contextlib import AsyncExitStack
from decimal import Decimal
import sqlalchemy as sa
import sqlalchemy.ext.asyncio
from sqlalchemy.dialects import postgresql as sa_pg
import composite
if __name__ == "__main__":
async def main():
metadata = sa.MetaData()
demo_composite_type = composite.define_composite(
"demo_composite",
metadata,
("field1", sa.TEXT),
("field2", sa.Numeric),
)
# NOTE: The following class might be omitted if `asyncpg.Record` as a return type is sufficient for your
# needs and there is no need to convert items to some custom (data)class.
class demo_composite_type(sa.types.TypeDecorator):
impl, cache_ok = demo_composite_type, True
python_type = dataclasses.make_dataclass(
"DemoComposite",
[name for name in demo_composite_type.fields],
)
def process_result_value(self, value, dialect):
if value is not None:
return self.python_type(**value)
async with AsyncExitStack() as deffer:
pg_engine = sa.ext.asyncio.create_async_engine(
"postgresql+asyncpg://scott:tiger#localhost/test"
)
deffer.push_async_callback(pg_engine.dispose)
pg_conn = await deffer.enter_async_context(
pg_engine.connect()
)
await pg_conn.run_sync(metadata.create_all)
deffer.push_async_callback(pg_conn.run_sync, metadata.drop_all)
values_stmt = (
sa.sql.Values(
sa.column("column1", sa.TEXT),
sa.column("column2", sa.Numeric),
)
.data([
("1", Decimal(1)),
("2", Decimal(2)),
])
.alias("samples")
)
result = (await pg_conn.execute(
sa.select([
sa.func.array_agg(
sa.cast(
sa.tuple_(
values_stmt.c.column1,
values_stmt.c.column2,
),
demo_composite_type,
),
type_=sa_pg.ARRAY(
demo_composite_type,
dimensions=1,
),
)
.label("array_of_composites")
])
.select_from(
values_stmt
)
)).scalar_one()
pprint.pprint(result)
asyncio.run(main())
Output of the sample above is the following:
[DemoComposite(field1='1', field2=Decimal('1')),
DemoComposite(field1='2', field2=Decimal('2'))]

How to use make_transient() to duplicate an SQLAlchemy mapped object?

I know the question how to duplicate or copy a SQLAlchemy mapped object was asked a lot of times. The answer always depends on the needs or how "duplicate" or "copy" is interpreted.
This is a specialized version of the question because I got the tip to use make_transient() for that.
But I have some problems with that. I don't really know how to handle the primary key (PK) here. In my use cases the PK is always autogenerated by SQLA (or the DB in background). But this doesn't happen with a new duplicated object.
The code is a little bit pseudo.
import sqlalchemy as sa
from sqlalchemy.orm.session import make_transient
_engine = sa.create_engine('postgres://...')
_session = sao.sessionmaker(bind=_engine)()
class MachineData(_Base):
__tablename__ = 'Machine'
_oid = sa.Column('oid', sa.Integer, primary_key=True)
class TUnitData(_Base):
__tablename__ = 'TUnit'
_oid = sa.Column('oid', sa.Integer, primary_key=True)
_machine_fk = sa.Column('machine', sa.Integer, sa.ForeignKey('Machine.oid'))
_machine = sao.relationship("MachineData")
def __str__(self):
return '{}.{}: oid={}(hasIdentity={}) machine={}(fk={})' \
.format(type(self), id(self),
self._oid, has_identity(self),
self._machine, self._machine_fk)
if __name__ == '__main__':
# any query resulting in one persistent object
obj = GetOneMachineDataFromDatabase()
# there is a valid 'oid', has_identity == True
print(obj)
# should i call expunge() first?
# remove the association with any session
# and remove its “identity key”
make_transient(obj)
# 'oid' is still there but has_identity == False
print(obj)
# THIS causes an error because the 'oid' still exsits
# and is not new auto-generated (what should happen in my
# understandings)
_session.add(obj)
_session.commit()

After making a object instance transient you have to remove its object-id. Without an object-id you can add it again to the database which will generate a new object-id for it.
if __name__ == '__main__':
# the persistent object with an identiy in the database
obj = GetOneMachineDataFromDatabase()
# make it transient
make_transient(obj)
# remove the identiy / object-id
obj._oid = None
# adding the object again generates a new identiy / object-id
_session.add(obj)
# this include a flush() and create a new primary key
_session.commit()

SQLAlchemy Error Appending to Relationship

I've been using SQLAlchemy 0.9.2 with Python Version 2.7.3 and have run into a bit of an odd problem that I can't quite seem to explain. Here is my relevant code:
Base = declarative_base()
class Parent(Base):
__tablename__ = 'parents'
__table_args__ = (UniqueConstraint('first_name', 'last_name', name='_name_constraint'),)
id = Column(Integer, primary_key=True)
first_name = Column(String(32), nullable=False)
last_name = Column(String(32), nullable=False)
children = relationship(Child, cascade='all,delete', backref='parent')
## Constructors and other methods ##
class Child(Base):
__tablename__ = 'children'
id = Column(Integer, primary_key=True)
parent_id = Column(Integer, ForeignKey('parents.id'))
foo = Column(String(32), nullable=False)
## Constructors and other methods ##
So a pretty basic set of models. The problem I'm experiencing is that I want to add a child to a parent that is saved to the database. The kicker is that the child is currently related to a parent that is not in the database. Consider the following example:
database_engine = create_engine("mysql://user:password#localhost/db", echo=False)
session = scoped_session(sessionmaker(autoflush=True,autocommit=False))
p1 = Parent("Foo", "Bar") # Create a parent and append a child
c1 = Child("foo")
p1.children.append(c1)
session.add(p1)
session.commit() # This works without a problem
db_parent = session.query(Parent).first()
db_parent.children.append(Child("bar"))
session.commit() # This also works without a problem
p2 = Parent("Foo", "Bar")
c3 = Child("baz")
p2.children.append(c3)
db_parent = session.query(Parent).first()
db_parent.children.append(p2.children[0])
session.commit() # ERROR: This blows up
The error I'm receiving is that I'm breaking an integrity Constraint, namely '_name_constraint'. SQLAlchemy is telling me that is trying to insert a Parent with the same information. My question is, why in the world is it trying to add a secondary parent?
These are the steps I've taken so far and don't have a good answer for:
I've inspected db_parent.children[2] It points to the same memory address as p1 once I have appended it to the list
I've inspected p2.children after the append. Oddly, p2 has no children once I have appended its child to db_parent I think this has something to do with what is going on, I just don't understand why its happening
Any help would be much appreciated, as I simply don't understand what's going on here. If you need me to post more please let me know. Thanks in advance.

Okay, after some more digging I think I have found a solution to my problem, but I don't yet have the answer as to why its happening the way it is, but I think I may have a guess. The solution I discovered was to use session.expunge(p2) before session.commit()
I started exploring SQLAlchemy's Internals, particularly, the instance state. I found that once you add the child to the parent, the original parent's state becomes pending. Here is an example:
from sqlalchemy import inspect
p2 = Parent("Foo", "Bar")
p2_inst = inspect(p2)
c3 = Child("Baz")
c3_inst = inspect(c3)
db_parent = session.query(Parent).first()
db_parent_inst = inspect(db_parent)
print("Pending State before append:")
print("p2_inst : {}".format(p2_inst.pending))
print("c3_inst : {}".format(c3_inst.pending))
print("db_parent_inst : {}".format(db_parent_inst.pending))
db_parent.children.append(p2.children[0])
print("Pending State after append:")
print("p2_inst : {}".format(p2_inst.pending))
print("c3_inst : {}".format(c3_inst.pending))
print("db_parent_inst : {}".format(db_parent_inst.pending))
session.expunge(p2)
print("Pending State after expunge:")
print("p2_inst : {}".format(p2_inst.pending))
print("c3_inst : {}".format(c3_inst.pending))
print("db_parent_inst : {}".format(db_parent_inst.pending))
session.commit()
The result of running this will be:
Pending State before append:
p2_inst : False
c3_inst : False
db_parent_inst : False
Pending State after append:
p2_inst : True
c3_inst : True
db_parent_inst : False
Pending State after expunge:
p2_inst : False
c3_inst : True
db_parent_inst : False
And there you have it. Once I thought about it a bit, I suppose it makes sense. There is no reason for the db_parent to ever enter a "pending" state because, you're not actually doing anything to the record in MySQL. My guess on why p2 becomes pending is due to an order of operations? In order for c3 to become pending, then all of its relationships must exist (to include p2) and so even when you change the child's parent, the session still think that it needs to add the parent.
I'd love for someone more knowledgeable on SQLAlchemy to correct me, but to the best of my knowledge, that's my best explanation :)

How to get sqlalchemy length of a string column

Consider this simple table definition (using SQLAlchemy-0.5.6)
from sqlalchemy import *
db = create_engine('sqlite:///tutorial.db')
db.echo = False # Try changing this to True and see what happens
metadata = MetaData(db)
user = Table('user', metadata,
Column('user_id', Integer, primary_key=True),
Column('name', String(40)),
Column('age', Integer),
Column('password', String),
)
from sqlalchemy.ext.declarative import declarative_base
class User(declarative_base()):
__tablename__ = 'user'
user_id = Column('user_id', Integer, primary_key=True)
name = Column('name', String(40))
I want to know what is the max length of column name e.g. from user table and from User (declarative class)
print user.name.length
print User.name.length
I have tried (User.name.type.length) but it throws exception
Traceback (most recent call last):
File "del.py", line 25, in <module>
print User.name.type.length
File "/usr/lib/python2.5/site-packages/SQLAlchemy-0.5.6-py2.5.egg/sqlalchemy/orm/attributes.py", line 135, in __getattr__
key)
AttributeError: Neither 'InstrumentedAttribute' object nor 'Comparator' object has an attribute 'type'

User.name.property.columns[0].type.length
Note, that SQLAlchemy supports composite properties, that's why columns is a list. It has single item for simple column properties.

This should work (tested on my machine) :
print user.columns.name.type.length

I was getting errors when fields were too big so I wrote a generic function to trim any string down and account for words with spaces. This will leave words intact and trim a string down to insert for you. I included my orm model for reference.
class ProductIdentifierTypes(Base):
__tablename__ = 'prod_id_type'
id = Column(Integer, primary_key=True, autoincrement=True)
name = Column(String(length=20))
description = Column(String(length=100))
def trim_for_insert(field_obj, in_str) -> str:
max_len = field_obj.property.columns[0].type.length
if len(in_str) <= max_len:
return in_str
logger.debug(f'Trimming {field_obj} to {max_len} max length.')
trim_str = in_str[:(max_len-1)]
if ' ' in trim_str[:int(max_len*0.9)]:
return(str.join(' ', trim_str.split(' ')[:-1]))
return trim_str
def foo_bar():
from models.deals import ProductIdentifierTypes, ProductName
_str = "Foo is a 42 year old big brown dog that all the kids call bar."
print(_str)
print(trim_for_insert(ProductIdentifierTypes.name, _str))
_str = "Full circle from the tomb of the womb to the womb of the tomb we come, an ambiguous, enigmatical incursion into a world of solid matter that is soon to melt from us like the substance of a dream."
print(_str)
print(trim_for_insert(ProductIdentifierTypes.description, _str))```

If you have access to the class:
TableClass.column_name.type.length
If you have access to an instance, you access the Class using the __class__ dunder method.
table_instance.__class__.column_name.type.length
So in your case:
# Via Instance
user.__class__.name.type.length
# Via Class
User.name.type.length
My use case is similar to #Gregg Williamson
However, I implemented it differently:
def __setattr__(self, attr, value):
column = self.__class__.type
if length := getattr(column, "length", 0):
value = value[:length]
super().__setattr__(name, value)

SQLAlchemy - MappedCollection problem

I have some problems with setting up the dictionary collection in Python's SQLAlchemy:
I am using declarative definition of tables. I have Item table in 1:N relation with Record table. I set up the relation using the following code:
_Base = declarative_base()
class Record(_Base):
__tablename__ = 'records'
item_id = Column(String(M_ITEM_ID), ForeignKey('items.id'))
id = Column(String(M_RECORD_ID), primary_key=True)
uri = Column(String(M_RECORD_URI))
name = Column(String(M_RECORD_NAME))
class Item(_Base):
__tablename__ = 'items'
id = Column(String(M_ITEM_ID), primary_key=True)
records = relation(Record, collection_class=column_mapped_collection(Record.name), backref='item')
Now I want to work with the Items and Records. Let's create some objects:
i1 = Item(id='id1')
r = Record(id='mujrecord')
And now I want to associate these objects using the following code:
i1.records['source_wav'] = r
but the Record r doesn't have set the name attribute (the foreign key). Is there any solution how to automatically ensure this? (I know that setting the foreign key during the Record creation works, but it doesn't sound good for me).
Many thanks

You want something like this:
from sqlalchemy.orm import validates
class Item(_Base):
[...]
#validates('records')
def validate_record(self, key, record):
assert record.name is not None, "Record fails validation, must have a name"
return record
With this, you get the desired validation:
>>> i1 = Item(id='id1')
>>> r = Record(id='mujrecord')
>>> i1.records['source_wav'] = r
Traceback (most recent call last):
[...]
AssertionError: Record fails validation, must have a name
>>> r.name = 'foo'
>>> i1.records['source_wav'] = r
>>>

I can't comment yet, so I'm just going to write this as a separate answer:
from sqlalchemy.orm import validates
class Item(_Base):
[...]
#validates('records')
def validate_record(self, key, record):
record.name=key
return record
This is basically a copy of Gunnlaugur's answer but abusing the validates decorator to do something more useful than exploding.

You have:
backref='item'
Is this a typo for
backref='name'
?