We Keep Coding

SQLAlchemy mapping class relationship error

I am facing this InvalidRequestError that my relationships between two classes are not mapped properly. This is one(Question) to Many(Choice) relationship.
from sqlalchemy.ext.declarative import declartive_base
Base = declarative_base()
class ChoiceModel(Base):
__tablename__ = 'choice'
id_seq = Sequence('choice_id_seq', metadata=Base.metadata)
id = Column(Integer, id_seq, server_default=id_seq.next_value(), primary_key=True, index=True)
choice_context = Column(Text, nullable=False)
question_id = Column(Integer, ForeignKey('question.id'))
question = relationship('QuestionModel', back_populates='choices')
class QuestionModel(Base):
__tablename__ = 'question'
id_seq = Sequence('question_id_seq', metadata=Base.metadata)
id = Column(Integer, id_seq, server_default=id_seq.next_value(), primary_key=True, index=True)
correct = Column(Boolean, nullable=False)
choice_list = Column(ARRAY(Integer))
choices = relationship("ChoiceModel", back_populates='question', cascade="all, delete-orphan")
And the error I am getting,
sqlalchemy.exc.InvalidRequestError:
When initializing mapper mapped class ChoiceModel->choice,
expression 'Question' failed to locate a name ('Question').
If this is a class name, consider adding this relationship() to the
<class 'project.core.models.choice_model.ChoiceModel'> class after both dependent classes have been defined.
I have read SQLAlchemy official document similar questions in stackoverflow that most of them used wrong class name, or didn't have back_populates.
My questions are,
should I have to change my classes name, like Choice, Question instead of having postfix Model ??
If I mapped correctly, what causes this error?
I have set my PostgreSQL tables name, choice and question
*python==3.10
*ubuntu==20.04
*fastapi==0.73.0
*SQLAlchemy==1.4.31
*postgresql==14.2
Thanks in advance!

How do I change the schema for both a table and a foreign key?

I have the following simplified database access layer and two tables:
class DataAccessLayer():
def __init__(self):
conn_string = "mysql+mysqlconnector://root:root#localhost/"
self.engine = create_engine(conn_string)
Base.metadata.create_all(self.engine)
Session = sessionmaker()
Session.configure(bind=self.engine)
self.session = Session()
class MatchesATP(Base):
__tablename__ = "matches_atp"
__table_args__ = {"schema": "belgarath", "extend_existing": True}
ID_M = Column(Integer, primary_key=True)
ID_T_M = Column(Integer, ForeignKey("oncourt.tours_atp.ID_T"))
class TournamentsATP(Base):
__tablename__ = "tours_atp"
__table_args__ = {"schema": "oncourt", "extend_existing": True}
ID_T = Column(Integer, primary_key=True)
NAME_T = Column(String(255))
I want to be able to switch the schema names for the two tables to test databases as follows:
belgarath to belgarath_test
oncourt to oncourt_test
I've tried adding:
self.session.connection(execution_options={"schema_translate_map": {"belgarath": belgarath, "oncourt": oncourt}})
To the bottom of DataAccessLayer and then initialising the class with two variables as follows:
def __init__(self, belgarath, oncourt):
However, when I build the following query:
dal = DataAccessLayer("belgarath_test", "oncourt_test")
query = dal.session.query(MatchesATP)
print(query)
I get the following SQL:
SELECT belgarath.matches_atp.`ID_M` AS `belgarath_matches_atp_ID_M`, belgarath.matches_atp.`ID_T_M` AS `belgarath_matches_atp_ID_T_M`
FROM belgarath.matches_atp
This is still referencing the belgarath table.
I also can't figure out a way of changing the schema of the foreign key of oncourt.tours_atp.ID_T at the same time as the tables.
Are there individual solutions or a combined solution to my issues?

You might wanna decorate your subclassed Base declarative model with the #declared_attr decorator.
Try this--
In a base class for your models, say __init__.py...
from sqlalchemy.ext.declarative import declarative_base, declared_attr
SCHEMA_MAIN = 'belgarath' # figure out how you want to retrieve this
SCHEMA_TEST = 'belgarath_test'
class _Base(object):
#declared_attr
def __table_args__(cls):
return {'schema': SCHEMA_MAIN}
...
Base = declarative_base(cls=_Base)
Base.metadata.schema = SCHEMA_MAIN
Now that you have a Base that subclasses _Base with the main schema already defined, all your other models will subclass Base and do the following:
from . import Base, declared_attr, SCHEMA_TEST
class TestModel(Base):
#declared_attr
def __table_args__(cls):
return {'schema': SCHEMA_TEST}
Changing a schema for a foreign key could look like this:
class TournamentsATP(Base):
__tablename__ = "tours_atp"
__table_args__ = {"schema": "oncourt", "extend_existing": True}
ID_T = Column(Integer, primary_key=True)
NAME_T = Column(String(255))
match_id = Column('match_id', Integer, ForeignKey(f'{__table_args__.get("schema")}.matches_atp.id'))
Where match_id is a foreign key to matches_atp.id by using the __table_args[schema] element defined at the class level via #declared_attr.

It only took me 18 months to figure this out. Turns out I needed to add the schema_translate_map to an engine and then create the session with this engine:
from sqlalchemy import create_engine
engine = create_engine(conn_str, echo=False)
schema_engine = engine.execution_options(schema_translate_map={<old_schema_name>: <new_schema_name>})
NewSession = sessionmaker(bind=schema_engine)
session = NewSession()
All ready to roll...

Assuming your goal is to:
have dev/test/prod schemas on a single mysql host
allow your ORM classes to be flexible enough to be used in three different environments without modification
Then John has you most of the way to one type of solution. You could use #declared_attr to dynamically generate __table_args__ as he has suggested.
You could also consider using something like flask-sqlalchemy that comes with a built-in solution for this:
import os
DB_ENV = os.getenv(DB_ENV)
SQLALCHEMY_BINDS = {
'belgarath': 'mysql+mysqlconnector://root:root#localhost/belgarath{}'.format(DB_ENV),
'oncourt': 'mysql+mysqlconnector://root:root#localhost/oncourt{}'.format(DB_ENV)
}
class MatchesATP(Base):
__bind_key__ = "belgarath"
ID_M = Column(Integer, primary_key=True)
ID_T_M = Column(Integer, ForeignKey("oncourt.tours_atp.ID_T"))
class TournamentsATP(Base):
__bind_key__ = "oncourt"
ID_T = Column(Integer, primary_key=True)
NAME_T = Column(String(255))
Basically this method allows you to create a link to a schema (a bind key), and that schema is defined at run-time via the connection string. More information at the flask-sqlalchemy link.

How to properly define a many-to-many relationship in SQLAlchemy?

In my Flask application I am using SQLAlchemy, all tables are defined in one single file models.py:
training_ids_association_table = db.Table(
"training_ids_association",
db.Model.metadata,
Column("training_id", Integer, ForeignKey("training_sessions.id")),
Column("ids_id", Integer, ForeignKey("image_data_sets.id")),
)
class ImageDataSet(db.Model):
__tablename__ = "image_data_sets"
id = Column(Integer, primary_key=True)
trainings = relationship("Training", secondary=training_ids_association_table, back_populates="image_data_sets")
class TrainingSession(db.Model):
__tablename__ = "training_sessions"
id = Column(Integer, primary_key=True)
image_data_sets = relationship("DataSet", secondary=training_ids_association_table, back_populates="trainings")
So what I want to achieve here is a many-to-many relationship:
One ImageDataSet can belong to multiple TrainingSession's
One TrainingSession can include multiple ImageDataSet's
However, as soon as I call TrainingSession.query() in my code, the following error is raised:
Exception has occurred: InvalidRequestError
When initializing mapper mapped class ImageDataSet->image_data_sets, expression 'Training' failed to locate a name ('Training'). If this is a class name, consider adding this relationship() to the <class 'app.base.models.ImageDataSet'> class after both dependent classes have been defined.
I found some related threads here, but they are either asking for one-to-many relationships, or they define their tables in different files. Both is not the case here.
Any ideas what I am doing wrong?

You mispelled the names of the models, try this:
training_ids_association_table = db.Table(
"training_ids_association",
db.Model.metadata,
Column("training_id", Integer, ForeignKey("training_sessions.id")),
Column("ids_id", Integer, ForeignKey("image_data_sets.id")),
)
class ImageDataSet(db.Model):
__tablename__ = "image_data_sets"
id = Column(Integer, primary_key=True)
trainings = relationship("TrainingSession", secondary=training_ids_association_table, back_populates="image_data_sets")
class TrainingSession(db.Model):
__tablename__ = "training_sessions"
id = Column(Integer, primary_key=True)
image_data_sets = relationship("ImageDataSet", secondary=training_ids_association_table, back_populates="trainings")

Order by association proxy: invalid sql

I want to attach a column of a related model to another model and use it like any other sqlalchemy column attribute. As mentioned in docs, association_proxy makes it easy to use.
from sqlalchemy.ext.associationproxy import association_proxy
from app import db
class User(db.Model):
id = db.Column(...)
name = db.Column(...)
class Book(db.Model):
author_id = db.Column(...)
author = db.relationship('User', foreign_keys=[author_id])
author_name = association_proxy('author', 'name')
But when I call Book.query.order_by(Book.author_name.asc()).all() I got this:
psycopg2.errors.SyntaxError: syntax error at or near "ASC"
LINE 4: ...user.id = book.author_id AND user.name ASC)
Sqlalchemy generates EXISTS statement:
...
EXISTS (SELECT 1
FROM user
WHERE user.id = book.author_id AND user.name ASC)
So how to use ordering properly with an associated attribute?

In this case it is probably best to use a column_property:
Base = declarative_base()
class User(Base):
__tablename__ = 'user'
id = Column(Integer(), primary_key=True)
name = Column(Unicode())
class Book(Base):
__tablename__ = 'book'
id = Column(Integer(), primary_key=True)
author_id = Column(Integer(), ForeignKey('user.id'), nullable=False)
author = relationship('User')
Book.author_name = column_property(select([User.name]).where(User.id == Book.author_id))
# sample ORM query
books = session.query(Book).order_by(Book.author_name)

SQLAlchemy generic relationship mixin

I'm working on a SQLAlchemy defining a bunch of mixin classes that applications should be able to import and extend their model.
When looking at the documentation, mixin classes are create knowing the final table name however, in the case of a generic library, the final table name that will be used by the application is not known.
Take the following mixin classes:
import sqlalchemy as sa
class UserMixin(object):
id = sa.Column(sa.Integer(), primary_key=True)
first_name = sa.Column(sa.Unicode(255))
last_name = sa.Column(sa.Unicode(255))
class ItemMixin(object):
id = sa.Column(sa.Integer(), primary_key=True)
name = sa.Column(sa.Unicode(255))
short_description = sa.Column(sa.Unicode(255))
class OrdersMixin(object):
id = sa.Column(sa.Integer(), primary_key=True)
user_id = sa.Column(sa.Integer(), sa.ForeignKey('???'))
item_id = sa.Column(sa.Integer(), sa.ForeignKey('???'))
Then an application defining its models:
import sqlalchemy as sa
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class MyUser(UserMixin, Base):
__tablename__ = 'myuser'
class MyItem(ItemMixin, Base):
__tablename__ = 'myitem'
total = sa.Column(sa.Integer())
class MyOrders(OrdersMixin, Base):
__tablename__ = 'myorders'
I have two issues with this model:
Except from redefining the relationship columns in the extending models, how can the mixin class build the relationship on its own.
Type of the foreign key is assumed by the mixin class, but the id of the table may come from the application itself or from another mixin class.
Is the model I'm trying to implement correct? What would be the right way to tackle this problem?

A Mixin is a class that copies its data into a table, but for SQL it matters if you're the owner of that data (the table) vs being a reference (the foreign key).
It looks like you're attempting to create Mixins that are both sources of truth and references. Which isn't possible in SQL.
Taking your example one step further and defining OrdersMixin like this will make the issues more obvious I think.
class OrdersMixin(UserMixin, ItemMixin):
id = sa.Column(sa.Integer(), primary_key=True)
For example MyOrders would end up like this once things are resolved.
class MyOrders(Base):
__tablename__ = 'myorders'
# This is from UserMixin
id = sa.Column(sa.Integer(), primary_key=True)
first_name = sa.Column(sa.Unicode(255))
last_name = sa.Column(sa.Unicode(255))
# This is from ItemMixin
id = sa.Column(sa.Integer(), primary_key=True)
name = sa.Column(sa.Unicode(255))
short_description = sa.Column(sa.Unicode(255))
# From OrdersMixin
id = sa.Column(sa.Integer(), primary_key=True) # This is defined last so it overrides all others with the same name.
user_id = sa.Column(sa.Integer(), sa.ForeignKey('???'))
item_id = sa.Column(sa.Integer(), sa.ForeignKey('???'))
With how you have the Mixin defined any table that used that Mixin would have primary_keys for the id column, which would conflict. Additionally you are duplicating every column in the Mixin, which in general you want to avoid in SQL (see Database normal form).
The final result would be something like this. Which is a whole bunch of columns meaning you wouldn't need to refer to any other tables and all of the references id you had were overwritten, meaning you wouldn't be able to join them anyway.
class MyOrders(Base):
__tablename__ = 'myorders'
first_name = sa.Column(sa.Unicode(255))
last_name = sa.Column(sa.Unicode(255))
name = sa.Column(sa.Unicode(255))
short_description = sa.Column(sa.Unicode(255))
id = sa.Column(sa.Integer(), primary_key=True) # This is defined last so it overrides all others with the same name.
user_id = sa.Column(sa.Integer(), sa.ForeignKey('???'))
item_id = sa.Column(sa.Integer(), sa.ForeignKey('???'))
To avoid that I keep my Mixins separate from initial table definition. I.e. I use a Mixin for when I want another table to refer to that table.
The following is close to what I think you were hoping to achieve.
import sqlalchemy as sa
from sqlalchemy import orm
class UserMixin(object):
user_id = sa.Column(sa.Integer(), ForeignKey("myuser.id"), index=True)
user = orm.relationship("MyUser")
class ItemMixin(object):
item_id = sa.Column(sa.Integer(), ForeignKey("myitem.id"), index=True)
item = orm.relationship("MyItem")
class OrdersMixin(UserMixin, ItemMixin):
order_id = sa.Column(sa.Integer(), sa.ForeignKey('myorders.id'))
user_id = sa.Column(sa.Integer(), sa.ForeignKey('myorders.user_id'))
item_id = sa.Column(sa.Integer(), sa.ForeignKey('myorders.item_id'))
Note in the Mixins I gave every column a unique name so that there aren't conflicts and in OrdersMixin even though I'm using UserMixin and ItemMixin I'm overriding the user_id and item_id columns because otherwise anything using the OrdersMixin would have foreign keys pointing to three different tables which would confuse the automatic query builder. But it will still add the user and item relations (and since they are defined as foreign keys to the original tables in MyOrders table I think the relationship will just work).
Then I would change your tables to look like this.
import sqlalchemy as sa
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class MyUser(Base):
__tablename__ = "myuser"
id = sa.Column(sa.Integer(),primary_key=True)
first_name = sa.Column(sa.Unicode(255))
last_name = sa.Column(sa.Unicode(255))
class MyItem(Base):
__tablename__ = "myitem"
id = sa.Column(sa.Integer(),primary_key=True)
name = sa.Column(sa.Unicode(255))
short_description = sa.Column(sa.Unicode(255))
class MyOrders(Base, UserMixin, OrdersMixin):
__tablename__ = "myorders"
id = sa.Column(sa.Integer(),primary_key=True)
The original table definition owns the columns (source of truth) defining them individually and Mixins (of this kind) are good to define references so subsequent references don't need define each of them individually. A Mixin can't be defined to be both a reference and a source of truth. In light of that instead of overriding the column each time like OrdersMixin it's better to just define it once canonically (the table) and once as a reference (the Mixin).