We Keep Coding

SQLAlchemy nested model creation one-liner

I'm looking to create a new object from q2, which fails because the Question class is expecting options to be a dictionary of Options, and it's receiving a dict of dicts instead.
So, unpacking obviously fails with a nested model.
What is the best approach to handle this? Is there something that's equivalent to the elegance of the **dict for a nested model?
main.py
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
import models.base
from models.question import Question
from models.option import Option
engine = create_engine('sqlite:///:memory:')
models.base.Base.metadata.create_all(engine, checkfirst=True)
Session = sessionmaker(bind=engine)
session = Session()
def create_question(q):
# The following hard coding works:
# q = Question(text='test text',
# frequency='test frequency',
# options=[Option(text='test option')]
# )
question = Question(**q)
session.add(question)
session.commit()
q1 = {
'text': 'test text',
'frequency': 'test frequency'
}
q2 = {
'text': 'test text',
'frequency': 'test frequency',
'options': [
{'text': 'test option 123'},
]
}
create_question(q1)
# create_question(q2) FAILS
base.py
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
question.py
from sqlalchemy import *
from sqlalchemy.orm import relationship
from .base import Base
class Question(Base):
__tablename__ = 'questions'
id = Column(Integer, primary_key=True)
text = Column(String(120), nullable=False)
frequency = Column(String(20), nullable=False)
active = Column(Boolean(), default=True, nullable=False)
options = relationship('Option', back_populates='question')
def __repr__(self):
return "<Question(id={0}, text={1}, frequency={2}, active={3})>".format(self.id, self.text, self.frequency, self.active)
option.py
from sqlalchemy import *
from sqlalchemy.orm import relationship
from .base import Base
class Option(Base):
__tablename__ = 'options'
id = Column(Integer, primary_key=True)
question_id = Column(Integer, ForeignKey('questions.id'))
text = Column(String(20), nullable=False)
question = relationship('Question', back_populates='options')
def __repr__(self):
return "<Option(id={0}, question_id={1}, text={2})>".format(self.id, self.question_id, self.text)

I liked the answer provided by #Abdou, but wanted to see if I couldn't make it a bit more generic.
I ended up coming up with the following, which should handle any nested model.
from sqlalchemy import event, inspect
#event.listens_for(Question, 'init')
#event.listens_for(Option, 'init')
def received_init(target, args, kwargs):
for rel in inspect(target.__class__).relationships:
rel_cls = rel.mapper.class_
if rel.key in kwargs:
kwargs[rel.key] = [rel_cls(**c) for c in kwargs[rel.key]]
Listens for the init event of any specified models, checks for relationships that match the kwargs passed in, and then converts those to the matching class of the relationship.
If anyone knows how to set this up so it can work on all models instead of specifying them, I would appreciate it.

Given that you need to create an Option object every time there is an options key in the dictionary passed to the create_question function, you should use dictionary comprehension to create your options before passing the result to the Question instantiator. I would rewrite the function as follows:
def create_question(q):
# The following hard coding works:
# q = Question(text='test text',
# frequency='test frequency',
# options=[Option(text='test option')]
# )
q = dict((k, [Option(**x) for x in v]) if k == 'options' else (k,v) for k,v in q.items())
print(q)
question = Question(**q)
session.add(question)
session.commit()
The dictionary comprehension part basically checks if there is an options key in the given dictionary; and if there is one, then it creates Option objects with the values. Otherwise, it carries on as normal.
The above function generated the following:
# {'text': 'test text', 'frequency': 'test frequency'}
# {'text': 'test text', 'frequency': 'test frequency', 'options': [<Option(id=None, question_id=None, text=test option 123)>]}
I hope this helps.

For SQLAlchemy objects you can simply use Model.__dict__

Building on #Searle's answer, this avoids needing to directly list all models in the decorators, and also provides handling for when uselist=False (e.g. 1:1, many:1 relationships):
from sqlalchemy import event
from sqlalchemy.orm import Mapper
#event.listens_for(Mapper, 'init')
def received_init(target, args, kwargs):
"""Allow initializing nested relationships with dict only"""
for rel in db.inspect(target).mapper.relationships:
if rel.key in kwargs:
if rel.uselist:
kwargs[rel.key] = [rel.mapper.class_(**c) for c in kwargs[rel.key]]
else:
kwargs[rel.key] = rel.mapper.class_(**kwargs[rel.key])
Possible further improvements:
add handling for if kwargs[rel.key] is a model instance (right now this fails if you pass a model instance for relationships instead of a dict)
allow relationships to be specified as None (right now requires empty lists or dicts)
source: SQLAlchemy "event.listen" for all models

How to use marshmallow to serialize a custom sqlalchemy field?

I just start a simple project called flask_wiki this days and I'm using some flask extensions as the follows:
Flask-SQLAlchemy
Flask-Restful
MarshMallow
Well, I just discovered that the MarshMallow project provides a class called 'ModelSchema', which reads all fields from my SQLAlchemy Model and provide a fully automated (de)serialializer.
In my case, I created a 'GUID' Field which is RDBM agnostic and inserted it on my Sqlalchemy model as follows:
from flask.ext.sqlalchemy import SQLAlchemy
from flask_wiki.backend.custom_fields import GUIDField
class Page(db.Model):
"""
Implements the Page Model.
"""
guid = db.Column(GUIDField, primary_key=True, default=uuid.uuid4)
name = db.Column(db.String, nullable=False)
raw_content = db.Column(db.Text)
rendered_content = db.Column(db.Text)
def __repr__(self):
return self.__str__()
def __str__(self):
return self.name
The GUIDField is implemented is this way:
from sqlalchemy.types import TypeDecorator, CHAR
import uuid
class GUIDField(TypeDecorator):
# Platform independent GUID Implementation that uses little endianess.
impl = CHAR
def load_dialect_impl(self, dialect):
return dialect.type_descriptor(CHAR(32))
def process_bind_param(self, value, dialect):
if value is None:
return value
else:
if isinstance(value, uuid.UUID):
return value.bytes_le
def process_result_value(self, value, dialect):
if value is None:
return value
else:
return uuid.UUID(bytes_le=value)
The code for create test objects (through mixer) is working; all the guids are generated and verified correctly.
With this in mind, a just created the following MarshMallow Serializer Field:
from marshmallow import fields
import uuid
class GUIDSerializationField(fields.Field):
def _serialize(self, value, attr, obj):
if value is None:
return value
else:
if isinstance(value, uuid.UUID):
return str(value)
else:
return None
Finally, I created the SerializerClass:
from flask_wiki.backend.backend import marsh
from flask_wiki.backend.custom_serialization_fields import GUIDSerializationField
from flask_wiki.backend.models import Page
from marshmallow import fields
class PageSchema(marsh.ModelSchema):
class Meta:
model = Page
guid = GUIDSerializationField()
page_schema = PageSchema()
pages_schema = PageSchema(many=True)
I tried to use this last code with and without inserting the guid field, but in all cases the following error occurs:
Traceback (most recent call last):
File "/home/arthas/dev/flask-wiki/flask_wiki/wiki.py", line 3, in <module>
from flask_wiki.backend import backend
File "/home/arthas/dev/flask-wiki/flask_wiki/backend/__init__.py", line 1, in <module>
import flask_wiki.backend.routes
File "/home/arthas/dev/flask-wiki/flask_wiki/backend/routes.py", line 2, in <module>
from flask_wiki.backend.views import PageView
File "/home/arthas/dev/flask-wiki/flask_wiki/backend/views.py", line 3, in <module>
from flask_wiki.backend.serializers import pages_schema, page_schema
File "/home/arthas/dev/flask-wiki/flask_wiki/backend/serializers.py", line 7, in <module>
class PageSchema(marsh.ModelSchema):
File "/home/arthas/env/wiki/lib/python3.5/site-packages/marshmallow/schema.py", line 116, in __new__
dict_cls=dict_cls
File "/home/arthas/env/wiki/lib/python3.5/site-packages/marshmallow_sqlalchemy/schema.py", line 53, in get_declared_fields
declared_fields = mcs.get_fields(converter, opts)
File "/home/arthas/env/wiki/lib/python3.5/site-packages/marshmallow_sqlalchemy/schema.py", line 77, in get_fields
return converter.fields_for_model(opts.model, fields=opts.fields, exclude=opts.exclude)
File "/home/arthas/env/wiki/lib/python3.5/site-packages/marshmallow_sqlalchemy/convert.py", line 75, in fields_for_model
field = self.property2field(prop)
File "/home/arthas/env/wiki/lib/python3.5/site-packages/marshmallow_sqlalchemy/convert.py", line 93, in property2field
field_class = self._get_field_class_for_property(prop)
File "/home/arthas/env/wiki/lib/python3.5/site-packages/marshmallow_sqlalchemy/convert.py", line 151, in _get_field_class_for_property
field_cls = self._get_field_class_for_column(column)
File "/home/arthas/env/wiki/lib/python3.5/site-packages/marshmallow_sqlalchemy/convert.py", line 121, in _get_field_class_for_column
return self._get_field_class_for_data_type(column.type)
File "/home/arthas/env/wiki/lib/python3.5/site-packages/marshmallow_sqlalchemy/convert.py", line 143, in _get_field_class_for_data_type
'Could not find field column of type {0}.'.format(types[0]))
marshmallow_sqlalchemy.exceptions.ModelConversionError: Could not find field column of type <class 'flask_wiki.backend.custom_fields.GUIDField'>.
So, I finally ask: how to use marshmallow to serialize a custom sqlalchemy field?

You need to create your own Converter.
Try something like this:
import uuid
from flask_wiki.backend.models import Page
from flask_wiki.backend.backend import marsh
from marshmallow_sqlalchemy.convert import ModelConverter
from flask_wiki.backend.custom_fields import GUIDField
from marshmallow import fields
#Here you are overwriting the list of types of the SQLAlchemy, adding your custom type
class GUIDConverter(ModelConverter):
SQLA_TYPE_MAPPING = dict(
list(ModelConverter.SQLA_TYPE_MAPPING.items()) +
[(GUIDField, fields.Str)]
)
class GUIDSerializationField(fields.Field):
def _serialize(self, value, attr, obj):
if value is None:
return value
else:
if isinstance(value, uuid.UUID):
return str(value)
else:
return None
class PageSchema(marsh.ModelSchema):
class Meta:
model = Page
model_converter = GUIDConverter #Tell to Marshmallow to use your custom converter for this model
guid = GUIDSerializationField(attribute="guid")
You can also see this link for help.
I hope it helped and sorry for my bad english

According to what I've read in the docs, you can specify a model_converter attribute in your ModelSchema Meta class. The ModelConverter class has a SQLA_TYPE_MAPPING attribute you can override in a subclass to add your custom GUID field to the types detected by the automatic schema generator.
That said, I've never used it so I don't know if this will work or not.

SQLAlchemy mapping table with non-ascii columns to class

item = Table('Item', metadata, autoload=True, autoload_with=engine, encoding = 'cp1257')
class Item(object):
pass
from sqlalchemy.orm import mapper
mapper(Item, item)
I get error:
line 43, in <module>
mapper(Item, item)
File "C:\Python27\lib\site-packages\sqlalchemy\orm\__init__.py", line 890, in mapper
return Mapper(class_, local_table, *args, **params)
File "C:\Python27\lib\site-packages\sqlalchemy\orm\mapper.py", line 211, in __init__
self._configure_properties()
File "C:\Python27\lib\site-packages\sqlalchemy\orm\mapper.py", line 578, in _configure_properties
setparent=True)
File "C:\Python27\lib\site-packages\sqlalchemy\orm\mapper.py", line 618, in _configure_property
self._log("_configure_property(%s, %s)", key, prop.__class__.__name__)
File "C:\Python27\lib\site-packages\sqlalchemy\orm\mapper.py", line 877, in _log
(self.non_primary and "|non-primary" or "") + ") " +
File "C:\Python27\lib\site-packages\sqlalchemy\util.py", line 1510, in __get__
obj.__dict__[self.__name__] = result = self.fget(obj)
File "C:\Python27\lib\site-packages\sqlalchemy\sql\expression.py", line 3544, in description
return self.name.encode('ascii', 'backslashreplace')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xeb in position 7: ordinal not in range(128)
I am connecting to MSSQL. table autoload seems to work. I only get this error while trying to map.
Thank you all for help!

Mapping the table to a class creates mapped properties on the class. The properties have the same name of the columns, by default. Since python 2.x only allows ascii identifiers, that fails if you have non-ascii column names.
The only solution I can think of is to give the identifiers a different name when mapping the table to a class.
The example below does that. Note that I'm creating the table on the code for simplicity, so anyone can run the code without having existing table. But you could do the same with a reflected table.
#-*- coding:utf-8 -*-
import sqlalchemy as sa
import sqlalchemy.orm
engine = sa.create_engine('sqlite://', echo=True) # new memory-only database
metadata = sa.MetaData(bind=engine)
# create a table. This could be reflected from the database instead:
tb = sa.Table('foo', metadata,
sa.Column(u'id', sa.Integer, primary_key=True),
sa.Column(u'nomé', sa.Unicode(100)),
sa.Column(u'ãéìöû', sa.Unicode(100))
)
tb.create()
class Foo(object):
pass
# maps the table to the class, defining different property names
# for some columns:
sa.orm.mapper(Foo, tb, properties={
'nome': tb.c[u'nomé'],
'aeiou': tb.c[u'ãéìöû']
})
After that you can use Foo.nome to refer to the nomé column and Foo.aeiou to refer to the ãéìöû column.

I faced the same problem and finally managed to do it replacing table['column'].key after autoloading it, just make all your table classes inherit this one and then modify the column name replacement in mapTo method or override manually the desired names with a dictionary and columns_descriptor method. I don't know if this is not the right way to do it but after searching for hours is the best aproach I've got.
class SageProxy(object):
#classmethod
def ismapped(cls, table_name=None):
if mappings:
if table_name:
if mappings.has_key(table_name):
tmap=mappings[table_name]
if tmap.has_key('class'):
tclass=tmap['class']
if tclass is cls:
return True
else:
for m in mappings:
if cls is m['class']:
return True
return False
#classmethod
def mappingprops(cls):
#override this to pass properties to sqlalchemy mapper function
return None
#classmethod
def columns_descriptors(cls):
#override this to map columns to different class properties names
#return dictionary where key is the column name and value is the desired property name
return {}
#classmethod
def mapTo(cls, table_name, map_opts=None):
if not cls.ismapped(table_name):
tab_obj=Table(table_name,sage_md,autoload=True)
for c in tab_obj.c:
#clean field names
tab_obj.c[c.name].key=c.key.replace(u'%',u'Porcentaje').replace(u'ñ',u'ny').replace(u'Ñ',u'NY').replace(u'-',u'_')
for k,v in cls.columns_descriptors():
if tab_obj.c[k]:
tab_obj.c[k].key=v
mapper(cls, tab_obj, properties=cls.mappingprops())
mappings[table_name]={'table':tab_obj,'class':cls}
return cls
I expect it will be usefull

I found that I could do this with a simple addition to my reflected class:
metadata = MetaData(bind=engine, reflect=True)
sm = sessionmaker(bind=engine)
class tblOrders(Base):
__table__ = metadata.tables['tblOrders']
meter = __table__.c['Meter#']
meter is now mapped to the underlying Meter# column, which allows this code to work:
currOrder = tblOrders()
currOrder.meter = '5'
Without the mapping, python sees it as a broken statement becase Meter followed by a comment does not exist in the object.

How to get sqlalchemy length of a string column

Consider this simple table definition (using SQLAlchemy-0.5.6)
from sqlalchemy import *
db = create_engine('sqlite:///tutorial.db')
db.echo = False # Try changing this to True and see what happens
metadata = MetaData(db)
user = Table('user', metadata,
Column('user_id', Integer, primary_key=True),
Column('name', String(40)),
Column('age', Integer),
Column('password', String),
)
from sqlalchemy.ext.declarative import declarative_base
class User(declarative_base()):
__tablename__ = 'user'
user_id = Column('user_id', Integer, primary_key=True)
name = Column('name', String(40))
I want to know what is the max length of column name e.g. from user table and from User (declarative class)
print user.name.length
print User.name.length
I have tried (User.name.type.length) but it throws exception
Traceback (most recent call last):
File "del.py", line 25, in <module>
print User.name.type.length
File "/usr/lib/python2.5/site-packages/SQLAlchemy-0.5.6-py2.5.egg/sqlalchemy/orm/attributes.py", line 135, in __getattr__
key)
AttributeError: Neither 'InstrumentedAttribute' object nor 'Comparator' object has an attribute 'type'

User.name.property.columns[0].type.length
Note, that SQLAlchemy supports composite properties, that's why columns is a list. It has single item for simple column properties.

This should work (tested on my machine) :
print user.columns.name.type.length

I was getting errors when fields were too big so I wrote a generic function to trim any string down and account for words with spaces. This will leave words intact and trim a string down to insert for you. I included my orm model for reference.
class ProductIdentifierTypes(Base):
__tablename__ = 'prod_id_type'
id = Column(Integer, primary_key=True, autoincrement=True)
name = Column(String(length=20))
description = Column(String(length=100))
def trim_for_insert(field_obj, in_str) -> str:
max_len = field_obj.property.columns[0].type.length
if len(in_str) <= max_len:
return in_str
logger.debug(f'Trimming {field_obj} to {max_len} max length.')
trim_str = in_str[:(max_len-1)]
if ' ' in trim_str[:int(max_len*0.9)]:
return(str.join(' ', trim_str.split(' ')[:-1]))
return trim_str
def foo_bar():
from models.deals import ProductIdentifierTypes, ProductName
_str = "Foo is a 42 year old big brown dog that all the kids call bar."
print(_str)
print(trim_for_insert(ProductIdentifierTypes.name, _str))
_str = "Full circle from the tomb of the womb to the womb of the tomb we come, an ambiguous, enigmatical incursion into a world of solid matter that is soon to melt from us like the substance of a dream."
print(_str)
print(trim_for_insert(ProductIdentifierTypes.description, _str))```

If you have access to the class:
TableClass.column_name.type.length
If you have access to an instance, you access the Class using the __class__ dunder method.
table_instance.__class__.column_name.type.length
So in your case:
# Via Instance
user.__class__.name.type.length
# Via Class
User.name.type.length
My use case is similar to #Gregg Williamson
However, I implemented it differently:
def __setattr__(self, attr, value):
column = self.__class__.type
if length := getattr(column, "length", 0):
value = value[:length]
super().__setattr__(name, value)

SQLAlchemy - MappedCollection problem

I have some problems with setting up the dictionary collection in Python's SQLAlchemy:
I am using declarative definition of tables. I have Item table in 1:N relation with Record table. I set up the relation using the following code:
_Base = declarative_base()
class Record(_Base):
__tablename__ = 'records'
item_id = Column(String(M_ITEM_ID), ForeignKey('items.id'))
id = Column(String(M_RECORD_ID), primary_key=True)
uri = Column(String(M_RECORD_URI))
name = Column(String(M_RECORD_NAME))
class Item(_Base):
__tablename__ = 'items'
id = Column(String(M_ITEM_ID), primary_key=True)
records = relation(Record, collection_class=column_mapped_collection(Record.name), backref='item')
Now I want to work with the Items and Records. Let's create some objects:
i1 = Item(id='id1')
r = Record(id='mujrecord')
And now I want to associate these objects using the following code:
i1.records['source_wav'] = r
but the Record r doesn't have set the name attribute (the foreign key). Is there any solution how to automatically ensure this? (I know that setting the foreign key during the Record creation works, but it doesn't sound good for me).
Many thanks

You want something like this:
from sqlalchemy.orm import validates
class Item(_Base):
[...]
#validates('records')
def validate_record(self, key, record):
assert record.name is not None, "Record fails validation, must have a name"
return record
With this, you get the desired validation:
>>> i1 = Item(id='id1')
>>> r = Record(id='mujrecord')
>>> i1.records['source_wav'] = r
Traceback (most recent call last):
[...]
AssertionError: Record fails validation, must have a name
>>> r.name = 'foo'
>>> i1.records['source_wav'] = r
>>>

I can't comment yet, so I'm just going to write this as a separate answer:
from sqlalchemy.orm import validates
class Item(_Base):
[...]
#validates('records')
def validate_record(self, key, record):
record.name=key
return record
This is basically a copy of Gunnlaugur's answer but abusing the validates decorator to do something more useful than exploding.

You have:
backref='item'
Is this a typo for
backref='name'
?