I have simple classes mapped to python dictionies, WHen trying to add them to an in memory SQLite Db using SQLalchemy i get troubles running filtered queries.
As the objects were already in dicts I added a class method to my mapped objects, that could create an instance of the mapped class from the dict.
here is the set up:
from sqlalchemy import Column, Integer, String, text
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import create_engine
from sqlalchemy.orm.session import sessionmaker
Base = declarative_base()
class User(Base):
__tablename__ = 'Users'
mailbox = Column(String, primary_key=True)
uid = Column(Integer, primary_key=True)
name = Column(String)
#classmethod
def from_dict(cls, dict):
cls.mailbox = dict["mailbox"]
cls.uid = dict["uid"]
cls.name = dict["name"]
return cls(mailbox=cls.mailbox,
uid=cls.uid,
name=cls.name)
The classmethod of User returns the User class. All the other set up is pretty normal.
For demonstration i created two lists, one of mapped User objects. One of dictionaries with User Data. Despite the types, the info inside each is identical.
def dict_user_list():
return [{'mailbox':"inbox", 'uid':1, 'name':"tim"},
{'mailbox':"inbox", 'uid':2, 'name':"mark"},
{'mailbox':"inbox", 'uid':3, 'name':"susan"},
{'mailbox':"mybox", 'uid':1, 'name':"geoff"},
{'mailbox':"mybox", 'uid':2, 'name':"kim"},
{'mailbox':"mybox", 'uid':3, 'name':"trev"},
{'mailbox':"mybox", 'uid':4, 'name':"rick"}]
def user_list():
return [User(mailbox="inbox", uid=1, name="tim"),
User(mailbox="inbox", uid=2, name="mark"),
User(mailbox="inbox", uid=3, name="susan"),
User(mailbox="mybox", uid=1, name="geoff"),
User(mailbox="mybox", uid=2, name="kim"),
User(mailbox="mybox", uid=3, name="trev"),
User(mailbox="mybox", uid=4, name="rick")]
Here is the rest of the setup:
ENGINE = create_engine('sqlite://')
Base.metadata.create_all(ENGINE)
Base.metadata.bind = ENGINE
session_maker = sessionmaker()
session = session_maker()
Now if i add the Users via the .from_dict class method:
session.add_all([User.from_dict(user) for user in dict_user_list()])
Then run a query and filter:
query = session.query(User)
print("inbox filter: ", query.filter(User.mailbox == 'inbox').count())
print("mybox filter: ", query.filter(User.mailbox == 'mybox').count())
print("inbox filter_by: ", query.filter_by(mailbox='inbox').count())
print("mybox filter_by: ", query.filter_by(mailbox='mybox').count())
The output is:
inbox filter: 0
mybox filter: 7
inbox filter_by: 0
mybox filter_by: 7
This is wrong. But i can get the correct answers by using text()
print("inbox filter: ", query.filter(text('Users.mailbox = "inbox"')).count())
print("mybox filter: ", query.filter(text('Users.mailbox = "mybox"')).count())
The output gives the correct results:
inbox filter: 3
mybox filter: 4
If I change my .add_all() argument to create my instances via User(....):
session.add_all([user for user in user_list()])
Then run mu queries and filters again without using text():
query = session.query(User)
print("inbox filter: ", query.filter(User.mailbox == 'inbox').count())
print("mybox filter: ", query.filter(User.mailbox == 'mybox').count())
print("inbox filter_by: ", query.filter_by(mailbox='inbox').count())
print("mybox filter_by: ", query.filter_by(mailbox='mybox').count())
The output is correct:
inbox filter: 3
mybox filter: 4
inbox filter_by: 3
mybox filter_by: 4
Obviously the solution is not to invoke instances of my mapped class from a dict. But As i spend so long trying to work out what was going on here I would love to know what is going on here, Is it a lack of understanding of SQLalchemy? a lack of understanding of classmethods? or just a lack of understanding of python in general?
Related
Question
I created a hybrid property which is composed of a string and a decimal formatted as percentage but I am getting a TypeError when using the hybrid expression. I've tried several variations on the f-string including converting it to float first but I still get the error on the same line. What is the best way to do this string formatting and concatenation on the hybrid property expression?
I want to know why 'result_1' is producing an error, and 'result_2' works correctly
Model
from decimal import Decimal as D
class SupplierDiscount(Base):
__tablename__ = "tblSupplierDiscount"
id = Column(Integer, primary_key=True)
discount = Column(DECIMAL(5, 4), nullable=False)
description = Column(String, nullable=False)
#hybrid_property
def disc_desc(self):
return f'{self.description}: {self.discount * 100:.4f}%'
#disc_desc.expression
def disc_desc(cls):
return f'{cls.description}: {cls.discount * 100:.4f}%' # Error generated here
result_1 - Preferred method - but results in error
result_1 = session.query(SupplierDiscount.id.label("SDId"),
SupplierDiscount.disc_desc.label("SDDDesc")
).all()
print('Below is from result_1')
print(result_1)
for i in result_1:
print(i.id, i.disc_desc)
Error produced in result_1
TypeError: unsupported format string passed to BinaryExpression.__format__
result_2 - This works but this is not the preferred method
result_2 = session.query(SupplierDiscount).all()
print('Below is from result_2')
print(result_2)
for i in result_2:
print(i.id, i.disc_desc)
Environment
SQLAlchemy==1.3.20
PostgreSQL 13
There is a distinction between class-level and instance-level access. The hybrid property is used on instance-level and uses python string formatting to get the desired result. As the expression is used for class-level access this needs to be defined as an SQL expression to return the desired result. See the docs for reference.
Knowing this we can rewrite the code to the following to achieve the same result on class-level and instance-level:
from sqlalchemy import Column, Integer, create_engine, DECIMAL, String, func
from sqlalchemy.engine import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.ext.hybrid import hybrid_property
from sqlalchemy.orm import Session
Base = declarative_base()
class Discount(Base):
__tablename__ = 'Discount'
id = Column(Integer, primary_key=True)
description = Column(String)
discount = Column(DECIMAL(5,4))
#hybrid_property
def disc_desc(self):
return f'{self.description}: {self.discount * 100:.4f}%'
#disc_desc.expression
def disc_desc(cls):
return cls.description + ' ' + func.cast(cls.discount * 100, String) + '%'
engine = create_engine(dburl)
Base.metadata.drop_all(engine)
Base.metadata.create_all(engine)
with Session(engine) as session:
discount = Discount(description='test', discount=5.4)
session.add(discount)
session.commit()
disc = session.query(Discount).first()
print('Instance-level access')
print(disc.disc_desc)
print('')
disc = session.query(Discount.disc_desc).first()
print('Class-level access')
print(disc.disc_desc)
This results in:
Instance-level access
test: 540.0000%
Class-level access
test: 540.0000%
I'm looking to create a new object from q2, which fails because the Question class is expecting options to be a dictionary of Options, and it's receiving a dict of dicts instead.
So, unpacking obviously fails with a nested model.
What is the best approach to handle this? Is there something that's equivalent to the elegance of the **dict for a nested model?
main.py
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
import models.base
from models.question import Question
from models.option import Option
engine = create_engine('sqlite:///:memory:')
models.base.Base.metadata.create_all(engine, checkfirst=True)
Session = sessionmaker(bind=engine)
session = Session()
def create_question(q):
# The following hard coding works:
# q = Question(text='test text',
# frequency='test frequency',
# options=[Option(text='test option')]
# )
question = Question(**q)
session.add(question)
session.commit()
q1 = {
'text': 'test text',
'frequency': 'test frequency'
}
q2 = {
'text': 'test text',
'frequency': 'test frequency',
'options': [
{'text': 'test option 123'},
]
}
create_question(q1)
# create_question(q2) FAILS
base.py
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
question.py
from sqlalchemy import *
from sqlalchemy.orm import relationship
from .base import Base
class Question(Base):
__tablename__ = 'questions'
id = Column(Integer, primary_key=True)
text = Column(String(120), nullable=False)
frequency = Column(String(20), nullable=False)
active = Column(Boolean(), default=True, nullable=False)
options = relationship('Option', back_populates='question')
def __repr__(self):
return "<Question(id={0}, text={1}, frequency={2}, active={3})>".format(self.id, self.text, self.frequency, self.active)
option.py
from sqlalchemy import *
from sqlalchemy.orm import relationship
from .base import Base
class Option(Base):
__tablename__ = 'options'
id = Column(Integer, primary_key=True)
question_id = Column(Integer, ForeignKey('questions.id'))
text = Column(String(20), nullable=False)
question = relationship('Question', back_populates='options')
def __repr__(self):
return "<Option(id={0}, question_id={1}, text={2})>".format(self.id, self.question_id, self.text)
I liked the answer provided by #Abdou, but wanted to see if I couldn't make it a bit more generic.
I ended up coming up with the following, which should handle any nested model.
from sqlalchemy import event, inspect
#event.listens_for(Question, 'init')
#event.listens_for(Option, 'init')
def received_init(target, args, kwargs):
for rel in inspect(target.__class__).relationships:
rel_cls = rel.mapper.class_
if rel.key in kwargs:
kwargs[rel.key] = [rel_cls(**c) for c in kwargs[rel.key]]
Listens for the init event of any specified models, checks for relationships that match the kwargs passed in, and then converts those to the matching class of the relationship.
If anyone knows how to set this up so it can work on all models instead of specifying them, I would appreciate it.
Given that you need to create an Option object every time there is an options key in the dictionary passed to the create_question function, you should use dictionary comprehension to create your options before passing the result to the Question instantiator. I would rewrite the function as follows:
def create_question(q):
# The following hard coding works:
# q = Question(text='test text',
# frequency='test frequency',
# options=[Option(text='test option')]
# )
q = dict((k, [Option(**x) for x in v]) if k == 'options' else (k,v) for k,v in q.items())
print(q)
question = Question(**q)
session.add(question)
session.commit()
The dictionary comprehension part basically checks if there is an options key in the given dictionary; and if there is one, then it creates Option objects with the values. Otherwise, it carries on as normal.
The above function generated the following:
# {'text': 'test text', 'frequency': 'test frequency'}
# {'text': 'test text', 'frequency': 'test frequency', 'options': [<Option(id=None, question_id=None, text=test option 123)>]}
I hope this helps.
For SQLAlchemy objects you can simply use Model.__dict__
Building on #Searle's answer, this avoids needing to directly list all models in the decorators, and also provides handling for when uselist=False (e.g. 1:1, many:1 relationships):
from sqlalchemy import event
from sqlalchemy.orm import Mapper
#event.listens_for(Mapper, 'init')
def received_init(target, args, kwargs):
"""Allow initializing nested relationships with dict only"""
for rel in db.inspect(target).mapper.relationships:
if rel.key in kwargs:
if rel.uselist:
kwargs[rel.key] = [rel.mapper.class_(**c) for c in kwargs[rel.key]]
else:
kwargs[rel.key] = rel.mapper.class_(**kwargs[rel.key])
Possible further improvements:
add handling for if kwargs[rel.key] is a model instance (right now this fails if you pass a model instance for relationships instead of a dict)
allow relationships to be specified as None (right now requires empty lists or dicts)
source: SQLAlchemy "event.listen" for all models
I've inherited a SQLAlchemy project and need some assistance figuring out why one relationship does not appear to be behaving like all the others. Our orm models are generated by a script and as such I would assume all would either work or not.
When I try to insert on the table with the FK I receive the below error message:
Foreign key associated with column 'RenewalBatchGroup.renewalbatchfrequencyid' could not find table 'RenewalBatchFrequency' with which to generate a foreign key to target column 'renewalbatchfrequencyid'
In the below code, as it is, I get the NoReferencedTableError exception. However if I change the import from Frequency importing Group to the other way around, it does work.
IE:
Remove the following from RenewalBatchFrequency.py
"from app.orm.LROCustomer.dbo.RenewalBatchGroup import RenewalBatchGroup"
and add the following to RenewalBatchGroup.py
from app.orm.LROCustomer.dbo.RenewalBatchFrequency import RenewalBatchFrequency
I need to know why this relationship is behaving differently from other relationships like the RenewalBatch relationship. It's setup the exact same way.
RenewalBatchGroup.py
from sqlalchemy.dialects.mssql import *
from sqlalchemy import Column, ForeignKey
from sqlalchemy.orm import relationship
from app.DatabaseCore import Base
from Utils.DateTimeUtils import getDateTimeString
from app.orm.LROCustomer.dbo.RenewalBatch import RenewalBatch
from app.orm.LROCustomer.dbo.RenewalBatchGroupProperty import RenewalBatchGroupProperty
class RenewalBatchGroup(Base):
__tablename__ = 'RenewalBatchGroup'
__table_args__ = {'implicit_returning': False} # http://docs.sqlalchemy.org/en/latest/dialects/mssql.html#triggers
batchleadtime = Column(SMALLINT)
batchsize = Column(SMALLINT)
groupname = Column(VARCHAR(400))
offerstatusid = Column(INTEGER)
renewalbatchfrequencyid = Column(INTEGER, ForeignKey('RenewalBatchFrequency.renewalbatchfrequencyid'))
renewalbatchfrequencyvalue = Column(VARCHAR(100))
renewalbatchgroupid = Column(INTEGER, primary_key=True)
updateuser = Column(VARCHAR(254))
RenewalBatches = relationship("RenewalBatch", backref="RenewalBatchGroup")
RenewalBatchGroupPropertys = relationship("RenewalBatchGroupProperty", backref="RenewalBatchGroup")
def jsonify(self):
return {"batchleadtime": self.batchleadtime,
"batchsize": self.batchsize,
"groupname": self.groupname,
"offerstatusid": self.offerstatusid,
"renewalbatchfrequencyid": self.renewalbatchfrequencyid,
"renewalbatchfrequencyvalue": self.renewalbatchfrequencyvalue,
"renewalbatchgroupid": self.renewalbatchgroupid,
"updateuser": self.updateuser
}
def update(self, row):
self.batchleadtime = row.batchleadtime
self.batchsize = row.batchsize
self.groupname = row.groupname
self.offerstatusid = row.offerstatusid
self.renewalbatchfrequencyid = row.renewalbatchfrequencyid
self.renewalbatchfrequencyvalue = row.renewalbatchfrequencyvalue
self.updateuser = row.updateuser
RenewalBatchFrequency.py
from sqlalchemy.dialects.mssql import *
from sqlalchemy import Column, ForeignKey
from sqlalchemy.orm import relationship
from app.DatabaseCore import Base
from Utils.DateTimeUtils import getDateTimeString
from app.orm.LROCustomer.dbo.RenewalBatchGroup import RenewalBatchGroup
class RenewalBatchFrequency(Base):
__tablename__ = 'RenewalBatchFrequency'
__table_args__ = {'implicit_returning': False} # http://docs.sqlalchemy.org/en/latest/dialects/mssql.html#triggers
name = Column(VARCHAR(100))
renewalbatchfrequencyid = Column(INTEGER, primary_key=True)
RenewalBatchGroups = relationship("RenewalBatchGroup", backref="RenewalBatchFrequency")
def jsonify(self):
return {"name": self.name,
"renewalbatchfrequencyid": self.renewalbatchfrequencyid
}
def update(self, row):
self.name = row.name
If any other information is needed just let me know.
Thank you.
As mentioned above, I can solve the issue by reversing the import statement. That's contrary to all the other relationships. I can only assume there is a circular reference somehow with the other relationships that SQLAlchemy is working out.
The way that I solved this issue is to put all of my model imports into the __ init__.py file and thus every model has access to all models. Now when I generate a new model I just need to make sure it's added to the __ init__.py file and I should be good to go.
A very frustrating experience.
I am trying to create a program that loads in over 100 tables from a database so that I can change all appearances of a user's user id.
Rather than map all of the tables individually, I decided to use a loop to map each of the tables using an array of objects. This way, the table definitions can be stored in a config file and later updated.
Here is my code so far:
def init_model(engine):
"""Call me before using any of the tables or classes in the model"""
meta.Session.configure(bind=engine)
meta.engine = engine
class Table:
tableID = ''
primaryKey = ''
pkType = sa.types.String()
class mappedClass(object):
pass
WIW_TBL = Table()
LOCATIONS_TBL = Table()
WIW_TBL.tableID = "wiw_tbl"
WIW_TBL.primaryKey = "PORTAL_USERID"
WIW_TBL.pkType = sa.types.String()
LOCATIONS_TBL.tableID = "locations_tbl"
LOCATIONS_TBL.primaryKey = "LOCATION_CODE"
LOCATIONS_TBL.pkType = sa.types.Integer()
tableList = ([WIW_TBL, LOCATIONS_TBL])
for i in tableList:
i.tableID = sa.Table(i.tableID.upper(), meta.metadata,
sa.Column(i.primaryKey, i.pkType, primary_key=True),
autoload=True,
autoload_with=engine)
orm.mapper(i.mappedClass, i.tableID)
The error that this code returns is:
sqlalchemy.exc.ArgumentError: Class '<class 'changeofname.model.mappedClass'>' already has a primary mapper defined. Use non_primary=True to create a non primary Mapper. clear_mappers() will remove *all* current mappers from all classes.
I cant use clear_mappers as it wipes all of the classes and the entity_name scheme doesn't seem to apply here.
It seems that every object wants to use the same class, although they all should have their own instance of it.
Does anyone have any ideas?
Well, in your case it *is the same Class you try to map to different Tables. To solve this, create a class dynamically for each Table:
class Table(object):
tableID = ''
primaryKey = ''
pkType = sa.types.String()
def __init__(self):
self.mappedClass = type('TempClass', (object,), {})
But I would prefer slightly cleaner version:
class Table2(object):
def __init__(self, table_id, pk_name, pk_type):
self.tableID = table_id
self.primaryKey = pk_name
self.pkType = pk_type
self.mappedClass = type('Class_' + self.tableID, (object,), {})
# ...
WIW_TBL = Table2("wiw_tbl", "PORTAL_USERID", sa.types.String())
LOCATIONS_TBL = Table2("locations_tbl", "LOCATION_CODE", sa.types.Integer())
I have some problems with setting up the dictionary collection in Python's SQLAlchemy:
I am using declarative definition of tables. I have Item table in 1:N relation with Record table. I set up the relation using the following code:
_Base = declarative_base()
class Record(_Base):
__tablename__ = 'records'
item_id = Column(String(M_ITEM_ID), ForeignKey('items.id'))
id = Column(String(M_RECORD_ID), primary_key=True)
uri = Column(String(M_RECORD_URI))
name = Column(String(M_RECORD_NAME))
class Item(_Base):
__tablename__ = 'items'
id = Column(String(M_ITEM_ID), primary_key=True)
records = relation(Record, collection_class=column_mapped_collection(Record.name), backref='item')
Now I want to work with the Items and Records. Let's create some objects:
i1 = Item(id='id1')
r = Record(id='mujrecord')
And now I want to associate these objects using the following code:
i1.records['source_wav'] = r
but the Record r doesn't have set the name attribute (the foreign key). Is there any solution how to automatically ensure this? (I know that setting the foreign key during the Record creation works, but it doesn't sound good for me).
Many thanks
You want something like this:
from sqlalchemy.orm import validates
class Item(_Base):
[...]
#validates('records')
def validate_record(self, key, record):
assert record.name is not None, "Record fails validation, must have a name"
return record
With this, you get the desired validation:
>>> i1 = Item(id='id1')
>>> r = Record(id='mujrecord')
>>> i1.records['source_wav'] = r
Traceback (most recent call last):
[...]
AssertionError: Record fails validation, must have a name
>>> r.name = 'foo'
>>> i1.records['source_wav'] = r
>>>
I can't comment yet, so I'm just going to write this as a separate answer:
from sqlalchemy.orm import validates
class Item(_Base):
[...]
#validates('records')
def validate_record(self, key, record):
record.name=key
return record
This is basically a copy of Gunnlaugur's answer but abusing the validates decorator to do something more useful than exploding.
You have:
backref='item'
Is this a typo for
backref='name'
?