I've inherited a SQLAlchemy project and need some assistance figuring out why one relationship does not appear to be behaving like all the others. Our orm models are generated by a script and as such I would assume all would either work or not.
When I try to insert on the table with the FK I receive the below error message:
Foreign key associated with column 'RenewalBatchGroup.renewalbatchfrequencyid' could not find table 'RenewalBatchFrequency' with which to generate a foreign key to target column 'renewalbatchfrequencyid'
In the below code, as it is, I get the NoReferencedTableError exception. However if I change the import from Frequency importing Group to the other way around, it does work.
IE:
Remove the following from RenewalBatchFrequency.py
"from app.orm.LROCustomer.dbo.RenewalBatchGroup import RenewalBatchGroup"
and add the following to RenewalBatchGroup.py
from app.orm.LROCustomer.dbo.RenewalBatchFrequency import RenewalBatchFrequency
I need to know why this relationship is behaving differently from other relationships like the RenewalBatch relationship. It's setup the exact same way.
RenewalBatchGroup.py
from sqlalchemy.dialects.mssql import *
from sqlalchemy import Column, ForeignKey
from sqlalchemy.orm import relationship
from app.DatabaseCore import Base
from Utils.DateTimeUtils import getDateTimeString
from app.orm.LROCustomer.dbo.RenewalBatch import RenewalBatch
from app.orm.LROCustomer.dbo.RenewalBatchGroupProperty import RenewalBatchGroupProperty
class RenewalBatchGroup(Base):
__tablename__ = 'RenewalBatchGroup'
__table_args__ = {'implicit_returning': False} # http://docs.sqlalchemy.org/en/latest/dialects/mssql.html#triggers
batchleadtime = Column(SMALLINT)
batchsize = Column(SMALLINT)
groupname = Column(VARCHAR(400))
offerstatusid = Column(INTEGER)
renewalbatchfrequencyid = Column(INTEGER, ForeignKey('RenewalBatchFrequency.renewalbatchfrequencyid'))
renewalbatchfrequencyvalue = Column(VARCHAR(100))
renewalbatchgroupid = Column(INTEGER, primary_key=True)
updateuser = Column(VARCHAR(254))
RenewalBatches = relationship("RenewalBatch", backref="RenewalBatchGroup")
RenewalBatchGroupPropertys = relationship("RenewalBatchGroupProperty", backref="RenewalBatchGroup")
def jsonify(self):
return {"batchleadtime": self.batchleadtime,
"batchsize": self.batchsize,
"groupname": self.groupname,
"offerstatusid": self.offerstatusid,
"renewalbatchfrequencyid": self.renewalbatchfrequencyid,
"renewalbatchfrequencyvalue": self.renewalbatchfrequencyvalue,
"renewalbatchgroupid": self.renewalbatchgroupid,
"updateuser": self.updateuser
}
def update(self, row):
self.batchleadtime = row.batchleadtime
self.batchsize = row.batchsize
self.groupname = row.groupname
self.offerstatusid = row.offerstatusid
self.renewalbatchfrequencyid = row.renewalbatchfrequencyid
self.renewalbatchfrequencyvalue = row.renewalbatchfrequencyvalue
self.updateuser = row.updateuser
RenewalBatchFrequency.py
from sqlalchemy.dialects.mssql import *
from sqlalchemy import Column, ForeignKey
from sqlalchemy.orm import relationship
from app.DatabaseCore import Base
from Utils.DateTimeUtils import getDateTimeString
from app.orm.LROCustomer.dbo.RenewalBatchGroup import RenewalBatchGroup
class RenewalBatchFrequency(Base):
__tablename__ = 'RenewalBatchFrequency'
__table_args__ = {'implicit_returning': False} # http://docs.sqlalchemy.org/en/latest/dialects/mssql.html#triggers
name = Column(VARCHAR(100))
renewalbatchfrequencyid = Column(INTEGER, primary_key=True)
RenewalBatchGroups = relationship("RenewalBatchGroup", backref="RenewalBatchFrequency")
def jsonify(self):
return {"name": self.name,
"renewalbatchfrequencyid": self.renewalbatchfrequencyid
}
def update(self, row):
self.name = row.name
If any other information is needed just let me know.
Thank you.
As mentioned above, I can solve the issue by reversing the import statement. That's contrary to all the other relationships. I can only assume there is a circular reference somehow with the other relationships that SQLAlchemy is working out.
The way that I solved this issue is to put all of my model imports into the __ init__.py file and thus every model has access to all models. Now when I generate a new model I just need to make sure it's added to the __ init__.py file and I should be good to go.
A very frustrating experience.
Related
Short Description: I want to add insertion_date and last_update columns to a table in a Postgresql database using SQLalchemy ORM. I followed the documentation and the examples on SO, but it didn't work for me.
Here is my example:
base.py
from sqlalchemy.ext.declarative.api import declarative_base
Base = declarative_base()
movie_model.py
from __future__ import annotations
import datetime
from sqlalchemy import func
from base import Base
from sqlalchemy.sql.schema import Column
from typing import Optional, Dict, Union
from sqlalchemy.sql.sqltypes import String, Date, DateTime, Integer
class Movie(Base):
__tablename__ = 'movies'
movie_id = Column(String, primary_key=True)
movie_name = Column(String)
release_date = Column(Date)
movie_revenue = Column(Integer)
last_update = Column(DateTime, default=datetime.datetime.now, onupdate=datetime.datetime.now)
insertion_date = Column(DateTime, server_default=func.now())
# __mapper_args__ = {"eager_defaults": True}
#staticmethod
def from_dict(movie_dict: Dict[str, Union[str, int, datetime]]) -> Optional[Movie]:
if not movie_dict:
return None
return Movie(
movie_id=movie_dict.get('movie_id'),
movie_name=movie_dict.get('movie_name'),
release_date=movie_dict.get('release_date'),
movie_revenue=movie_dict.get('movie_revenue')
)
db_manager.py
from movie_model import Movie
from sqlalchemy.dialects.postgresql import insert
from sqlalchemy.inspection import inspect
from typing import List
class DBManager:
def __init__(self, session):
self.session = session
self.columns = Movie.__table__.columns.keys()
self.primary_key = inspect(Movie).primary_key[0]
self.primary_key_name = self.primary_key.name
def add_many_on_conflict_do_update(self, movies: List[Movie]) -> None:
# Used self.columns[:-2] to exclude the last two columns (last_update and insertion_date) from the explicit insertion, so the ORM handles it
# on its own as mentioned here: https://docs.sqlalchemy.org/en/14/core/metadata.html#sqlalchemy.schema.Column.params.onupdate.
statement = insert(Movie.__table__).values([{attr: getattr(movie, attr) for attr in self.columns[:-2]} for movie in movies])
statement = statement.on_conflict_do_update(
index_elements=[self.primary_key_name], set_={attr: getattr(statement.excluded, attr) for attr in self.columns[:-2]}
)
self.session.execute(statement)
self.session.commit()
def get_all(self) -> List[Movie]:
return self.session.query(Movie).all()
main.py
from movie_model import Movie
from db_manager import DBManager
import datetime
from base import Base
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker, scoped_session
def main():
db_string = "postgresql+psycopg2://db_user:password#localhost:5432/db_name"
engine = create_engine(db_string, echo=False)
session = scoped_session(sessionmaker(bind=engine))
Base.metadata.create_all(engine, checkfirst=True)
movie1 = {
'movie_id': '1',
'movie_name': 'movie1',
'release_date': datetime.datetime.strptime('2010-09-23', '%Y-%m-%d').date(),
'movie_revenue': 1000
}
movie2 = {
'movie_id': '2',
'movie_name': 'movie2',
'release_date': datetime.datetime.strptime('2010-09-23', '%Y-%m-%d').date(),
'movie_revenue': 2000
}
movie3 = {
'movie_id': '3',
'movie_name': 'movie3',
'release_date': datetime.datetime.strptime('2010-09-24', '%Y-%m-%d').date(),
'movie_revenue': 3000
}
movie4 = {
'movie_id': '4',
'movie_name': 'movie4',
'release_date': datetime.datetime.strptime('2010-09-24', '%Y-%m-%d').date(),
'movie_revenue': 4000
}
movies = [movie1, movie2, movie3, movie4]
movies_models = []
for movie in movies:
movies_models.append(Movie.from_dict(movie))
db_manager = DBManager(session=session)
db_manager.add_many_on_conflict_do_update(movies_models)
retrieved_movies = db_manager.get_all()
for movie in retrieved_movies:
print(movie.last_update)
if __name__ == '__main__':
main()
Packages:
psycopg2-binary==2.9.1
SQLAlchemy==1.3.24
**Postgres version: ** 12.
When I run python main.py the insertion_date is inserted properly even if I run it for many times adding new movie each time. But the last_update is only inserted the first time I add a movie, while it is expected to be updated whenever an update is applied to the already inserted movie.
I tried replacing self.columns[:-2] by self.columns[:-1] or by self.columns in the insertion statement and/or in the _set parameter. But it ended up that last_update column is null or it is being updated every time I run main.py even if there are no updates/insertions to apply.
I also replaced onupdate by server_onupdate and datetime.datetime.now by func.now() in movie_model.py but nothing worked.
I even added __mapper_args__ = {"eager_defaults": True} to Movie() in movie_model.py as described here but still didn't work.
I checked the following questions on SO but nothing worked too:
- Is possible to create Column in SQLAlchemy which is going to be automatically populated with time when it inserted/updated last time?
- onupdate not overridinig current datetime value
Any suggestions to make it work?
Edit for clarification:
What I am looking for is to make the last_update column get updated with the time of the transaction whenever a row is updated (using on_conflict_do_update).
I'm looking to create a new object from q2, which fails because the Question class is expecting options to be a dictionary of Options, and it's receiving a dict of dicts instead.
So, unpacking obviously fails with a nested model.
What is the best approach to handle this? Is there something that's equivalent to the elegance of the **dict for a nested model?
main.py
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
import models.base
from models.question import Question
from models.option import Option
engine = create_engine('sqlite:///:memory:')
models.base.Base.metadata.create_all(engine, checkfirst=True)
Session = sessionmaker(bind=engine)
session = Session()
def create_question(q):
# The following hard coding works:
# q = Question(text='test text',
# frequency='test frequency',
# options=[Option(text='test option')]
# )
question = Question(**q)
session.add(question)
session.commit()
q1 = {
'text': 'test text',
'frequency': 'test frequency'
}
q2 = {
'text': 'test text',
'frequency': 'test frequency',
'options': [
{'text': 'test option 123'},
]
}
create_question(q1)
# create_question(q2) FAILS
base.py
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
question.py
from sqlalchemy import *
from sqlalchemy.orm import relationship
from .base import Base
class Question(Base):
__tablename__ = 'questions'
id = Column(Integer, primary_key=True)
text = Column(String(120), nullable=False)
frequency = Column(String(20), nullable=False)
active = Column(Boolean(), default=True, nullable=False)
options = relationship('Option', back_populates='question')
def __repr__(self):
return "<Question(id={0}, text={1}, frequency={2}, active={3})>".format(self.id, self.text, self.frequency, self.active)
option.py
from sqlalchemy import *
from sqlalchemy.orm import relationship
from .base import Base
class Option(Base):
__tablename__ = 'options'
id = Column(Integer, primary_key=True)
question_id = Column(Integer, ForeignKey('questions.id'))
text = Column(String(20), nullable=False)
question = relationship('Question', back_populates='options')
def __repr__(self):
return "<Option(id={0}, question_id={1}, text={2})>".format(self.id, self.question_id, self.text)
I liked the answer provided by #Abdou, but wanted to see if I couldn't make it a bit more generic.
I ended up coming up with the following, which should handle any nested model.
from sqlalchemy import event, inspect
#event.listens_for(Question, 'init')
#event.listens_for(Option, 'init')
def received_init(target, args, kwargs):
for rel in inspect(target.__class__).relationships:
rel_cls = rel.mapper.class_
if rel.key in kwargs:
kwargs[rel.key] = [rel_cls(**c) for c in kwargs[rel.key]]
Listens for the init event of any specified models, checks for relationships that match the kwargs passed in, and then converts those to the matching class of the relationship.
If anyone knows how to set this up so it can work on all models instead of specifying them, I would appreciate it.
Given that you need to create an Option object every time there is an options key in the dictionary passed to the create_question function, you should use dictionary comprehension to create your options before passing the result to the Question instantiator. I would rewrite the function as follows:
def create_question(q):
# The following hard coding works:
# q = Question(text='test text',
# frequency='test frequency',
# options=[Option(text='test option')]
# )
q = dict((k, [Option(**x) for x in v]) if k == 'options' else (k,v) for k,v in q.items())
print(q)
question = Question(**q)
session.add(question)
session.commit()
The dictionary comprehension part basically checks if there is an options key in the given dictionary; and if there is one, then it creates Option objects with the values. Otherwise, it carries on as normal.
The above function generated the following:
# {'text': 'test text', 'frequency': 'test frequency'}
# {'text': 'test text', 'frequency': 'test frequency', 'options': [<Option(id=None, question_id=None, text=test option 123)>]}
I hope this helps.
For SQLAlchemy objects you can simply use Model.__dict__
Building on #Searle's answer, this avoids needing to directly list all models in the decorators, and also provides handling for when uselist=False (e.g. 1:1, many:1 relationships):
from sqlalchemy import event
from sqlalchemy.orm import Mapper
#event.listens_for(Mapper, 'init')
def received_init(target, args, kwargs):
"""Allow initializing nested relationships with dict only"""
for rel in db.inspect(target).mapper.relationships:
if rel.key in kwargs:
if rel.uselist:
kwargs[rel.key] = [rel.mapper.class_(**c) for c in kwargs[rel.key]]
else:
kwargs[rel.key] = rel.mapper.class_(**kwargs[rel.key])
Possible further improvements:
add handling for if kwargs[rel.key] is a model instance (right now this fails if you pass a model instance for relationships instead of a dict)
allow relationships to be specified as None (right now requires empty lists or dicts)
source: SQLAlchemy "event.listen" for all models
I am trying to write a logging system, which uses dynamic classes to make tables. Getting the classes created, and the tables created seems to be working fine, but trying to put entries into them is lead to an error message regarding mapping, below is the sample code and the error message.
Base = declarative_base()
#my init function
def tableinit(self,keyargs):
self.__dict__ = dict(keyargs)
#table creation
tableName = "newTable"
columnsDict["__tablename__"] = tableName
columnsDict["__init__"] = tableinit
columnsDict["id"] = Column("id",Integer, autoincrement = True, nullable = False, primary_key=True)
columnsDict["pid"] = Column("pid",Integer, ForeignKey('someparenttable.id')) #someparenttable is created with a hard coded class
newTable = type(tableName,(Base,),columnsDict)
tableClassDict[tableName]=newTable
#when doing an entry
newClassInst = subEntryClassDict[tableName]
newEntry = newClassInst(dataDict)
entryList.append(newEntry) # this is called in a for loop with the entries for someparenttable's entries also
self.session.add_all(entryList) # at this point the error occurs
The error:
UnmappedInstanceError: Class 'newTable' is mapped, but this instance lacks instrumentation. This occurs when the instance is created before sqlalchemy.orm.mapper(module.newTable) was called.
This is easier if you create a function to return a class that you set up normally. I've tried something like this and it works:
def getNewTable( db, table ):
class NewTable( Base ):
__tablename__ = table
__table_args__ = { 'schema': db }
id = Column( ...
return NewTable
newClassInst = getNewTable( 'somedb', 'sometable' )
newRow = newClassInst( data )
This problem is caused by lack of instruments function interfaces for the orm as the error description says. And it is actually caused by self.__dict__ = dict(keyargs) I think.
So this can be solved by reconstruct the init, which do not modify the injected functions by ORM.
Turn this
#my init function
def tableinit(self,keyargs):
self.__dict__ = dict(keyargs)
To
#my init function
def tableinit(self,**kwargs):
self.__dict__.update(kwargs)
I'm trying to make some generic apps using Sql Alchemy, such as tags or rating for any model. But I couldn't find any help in the docs. I really liked what I could do with the django contenttypes framework ? Is there any similar functionality in Sql Alchemy ?
I once wrote some example code about something similar to this (see http://taketwoprogramming.blogspot.com/2009/08/reusable-sqlalchemy-models.html).
The basic idea is that you can create a model like this:
#commentable
class Post(Base):
__tablename__ = 'posts'
id = sa.Column(sa.Integer, primary_key=True)
text = sa.Column(sa.String)
...where commentable is defined like this...
class BaseComment(object):
pass
def build_comment_model(clazz):
class_table_name = str(class_mapper(clazz).local_table)
metadata = clazz.metadata
comment_class_name = clazz.__name__ + 'Comment'
comment_class = type(comment_class_name, (BaseComment,), {})
comment_table_name = class_table_name + '_comments'
comment_table = sa.Table(comment_table_name, metadata,
sa.Column('id', sa.Integer, primary_key=True),
sa.Column(class_table_name + '_id',
sa.Integer,
sa.ForeignKey(class_table_name + '.id')),
sa.Column('text', sa.String),
sa.Column('name', sa.String(100)),
sa.Column('url', sa.String(255)),
)
mapper(comment_class, comment_table)
return comment_class, comment_table
def commentable(clazz):
comment_class, comment_table = build_comment_model(clazz)
clazz.Comment = comment_class
setattr(clazz, 'comments', relation(comment_class))
def add_comment(self, comment):
self.comments.append(comment)
setattr(clazz, 'add_comment', add_comment)
return clazz
Basically, the commentable decorator dynamically creates a new type and table, along with some helper methods to the decorated class. This is the test I used to test that the code works, which shows some example of how it would work...
class TestModels(SATestCase):
def test_make_comment(self):
p = Post()
p.text = 'SQLAlchemy is amazing!'
text = 'I agree!'
name = 'Mark'
url = 'http://www.sqlalchemy.org/'
c = Post.Comment()
c.text = text
c.name = name
c.url = url
p.add_comment(c)
Session.add(p)
# This is a method I use to force the reload of the objects from
# the database to make sure that when I test them, I'm actually
# pulling from the database rather than just getting the data
# of the object still in the session.
p = self.reload(p)
self.assertEquals(len(p.comments), 1)
c = p.comments[0]
self.assertEquals(c.text, text)
self.assertEquals(c.name, name)
self.assertEquals(c.url, url)
I wrote this awhile ago, but I don't think there's anything in SQLA that will do this kind of thing for you, but you can create something similar without too much trouble. In my example, I created new mapped classes and methods to use it on the fly in a class decorator.
I never really made much use out of it, but it might give you some ideas.
I have some problems with setting up the dictionary collection in Python's SQLAlchemy:
I am using declarative definition of tables. I have Item table in 1:N relation with Record table. I set up the relation using the following code:
_Base = declarative_base()
class Record(_Base):
__tablename__ = 'records'
item_id = Column(String(M_ITEM_ID), ForeignKey('items.id'))
id = Column(String(M_RECORD_ID), primary_key=True)
uri = Column(String(M_RECORD_URI))
name = Column(String(M_RECORD_NAME))
class Item(_Base):
__tablename__ = 'items'
id = Column(String(M_ITEM_ID), primary_key=True)
records = relation(Record, collection_class=column_mapped_collection(Record.name), backref='item')
Now I want to work with the Items and Records. Let's create some objects:
i1 = Item(id='id1')
r = Record(id='mujrecord')
And now I want to associate these objects using the following code:
i1.records['source_wav'] = r
but the Record r doesn't have set the name attribute (the foreign key). Is there any solution how to automatically ensure this? (I know that setting the foreign key during the Record creation works, but it doesn't sound good for me).
Many thanks
You want something like this:
from sqlalchemy.orm import validates
class Item(_Base):
[...]
#validates('records')
def validate_record(self, key, record):
assert record.name is not None, "Record fails validation, must have a name"
return record
With this, you get the desired validation:
>>> i1 = Item(id='id1')
>>> r = Record(id='mujrecord')
>>> i1.records['source_wav'] = r
Traceback (most recent call last):
[...]
AssertionError: Record fails validation, must have a name
>>> r.name = 'foo'
>>> i1.records['source_wav'] = r
>>>
I can't comment yet, so I'm just going to write this as a separate answer:
from sqlalchemy.orm import validates
class Item(_Base):
[...]
#validates('records')
def validate_record(self, key, record):
record.name=key
return record
This is basically a copy of Gunnlaugur's answer but abusing the validates decorator to do something more useful than exploding.
You have:
backref='item'
Is this a typo for
backref='name'
?