I'm using a bidirectional association_proxy to associate properties Group.members and User.groups. I'm having issues with removing a member from Group.members. In particular, Group.members.remove will successfully remove an entry from Group.members, but will leave a None in place of the corresponding entry in User.groups.
More concretely, the following (minimal-ish) representative code snippet fails its last assertion:
import sqlalchemy as sa
from sqlalchemy.orm import Session
from sqlalchemy.ext.associationproxy import association_proxy
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class Group(Base):
__tablename__ = 'group'
id = sa.Column(sa.Integer, autoincrement=True, primary_key=True)
name = sa.Column(sa.UnicodeText())
members = association_proxy('group_memberships', 'user',
creator=lambda user: GroupMembership(user=user))
class User(Base):
__tablename__ = 'user'
id = sa.Column(sa.Integer, autoincrement=True, primary_key=True)
username = sa.Column(sa.UnicodeText())
groups = association_proxy('group_memberships', 'group',
creator=lambda group: GroupMembership(group=group))
class GroupMembership(Base):
__tablename__ = 'user_group'
user_id = sa.Column(sa.Integer, sa.ForeignKey('user.id'), primary_key=True)
group_id = sa.Column(sa.Integer, sa.ForeignKey('group.id'), primary_key=True)
user = sa.orm.relationship(
'User',
backref=sa.orm.backref('group_memberships', cascade="all, delete-orphan"))
group = sa.orm.relationship(
'Group',
backref=sa.orm.backref('group_memberships', cascade="all, delete-orphan"),
order_by='Group.name')
if __name__ == '__main__':
engine = sa.create_engine('sqlite://')
Base.metadata.create_all(engine)
session = Session(engine)
group = Group(name='group name')
user = User(username='user name')
group.members.append(user)
session.add(group)
session.add(user)
session.flush()
assert group.members == [user]
assert user.groups == [group]
group.members.remove(user)
session.flush()
assert group.members == []
assert user.groups == [] # This assertion fails, user.groups is [None]
I've tried to follow the answers to SQLAlchemy relationship with association_proxy problems and How can SQLAlchemy association_proxy be used bi-directionally? but they do not seem to help.
I discovered your problem almost entirely by accident, as I was trying to figure out what's going on.
Because there wasn't any data in the db, I added a session.commit(). It turns out that (from the linked answer):
The changes aren't persisted permanently to disk, or visible to other transactions until the database receives a COMMIT for the current transaction (which is what session.commit() does).
Because you are just .flush()ing the changes, sqlalchemy never re-queries the database. You can verify this by adding:
import logging
logging.getLogger('sqlalchemy').setLevel(logging.INFO)
logging.getLogger('sqlalchemy').addHandler(logging.StreamHandler())
And then simply running your code. It will display all of the queries that are run as they happen. Then you can change session.flush() to session.commit() and then re-run, and you'll see that several SELECT statements are run after your commit.
It looks like either session.expire(user) or session.refresh(user) will force a refresh of the user, as well. I'm not sure if there's a way to force the update to propagate to the other object without being explicit about it (or if that's even desirable).
Related
We use SQLAlchemy to read/write data, but not create tables (as it done by DBAs). Due to this, some of the definitions which are incorrect have not yet been caught (though it works for reads/writes).
Is there a way to override them for testing purposes (create tables on the fly etc.) without touching the original definition? A simple class-overriding doesn't seem to work, and I don't see any other solution to this problem:
from sqlalchemy import Column, Integer, Numeric, String
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class Product(Base):
__tablename__ = "product"
idn = Column(Numeric, primary_key=True) # should be Integer
code = Column(String, primary_key=True) # should be unique (not primary)
class Product(Product):
__tablename__ = "product"
__table_args__ = {"extend_existing": True} # want to override, not extend
idn = Column(Integer, primary_key=True)
code = Column(String, unique=True)
When I have two objects, linked with a "relation" in SQLAlchemy, I realised that simply assigning to that relation is not enough to propagate the values to the other object. For example (see below), if I have a "user" table and a "contact" table (both are highly contrived, but demonstrate the issue well), and a "user" can have multiple "contacts". In that case I will have a foreign key between the users and contacts. If I create both an instance of User and Contact and later assign the user to the contact, I would expect the attributes of the FK to be updated (even without a DB flush) but they are not. Why? And how can I tell SA to do this automatically?
This would be something I would expect to work, but as you can see in the full example below, it does not:
user = User(name='a', lname='b')
contact(type='email', value='foo#bar.com')
contact.user = user
assert contact.username == 'a' # <-- Fails because the attribute is still `None`
Full runnable example:
"""
This code example shows two tables related to each other by a composite key,
using an SQLAlchemy "relation" to allow easy access to related items.
However, as the last few lines show, simply assigning an object A to the
relation of object B does not update the attributes of object B until at least
a "flush" is called.
"""
from sqlalchemy import Column, ForeignKeyConstraint, Unicode, create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import relation, sessionmaker
Base = declarative_base()
class User(Base):
__tablename__ = "user"
name = Column(Unicode, primary_key=True)
lname = Column(Unicode, primary_key=True)
class Contact(Base):
__tablename__ = "contact"
__table_args__ = (
ForeignKeyConstraint(
['username', 'userlname'],
['user.name', 'user.lname']
),
)
username = Column(Unicode, primary_key=True)
userlname = Column(Unicode, primary_key=True)
type = Column(Unicode)
value = Column(Unicode)
user = relation(User)
engine = create_engine('sqlite://')
Base.metadata.create_all(engine)
Session = sessionmaker(bind=engine)
session = Session()
user = User(name="John", lname="Doe")
contact = Contact(type='email', value='john.doe#example.com')
contact.user = user # <-- How can I tell SA to set the FKs on *contact* here?
session.add(contact)
print('Before flush: contact.username user=%r' % contact.username)
session.flush()
print('After flush : contact.username user=%r' % contact.username)
According to this answer - https://stackoverflow.com/a/52911047/4981223 it is not possible:
The FK of the child object isn't updated until you issue a flush() either explicitly or through a commit(). I think the reason for this is that if the parent object of a relationship is also a new instance with an auto-increment PK, SQLAlchemy needs to get the PK from the database before it can update the FK on the child object (but I stand to be corrected!).
I have been fighting this error for a few days now in a variety of configurations.
I tried adding Relationship lines under each of the ForeignKeys with no luck.
I also swapped my queries from using the declarative_base style calls to SQL queries.
The error only appears once I add the second ForeignKey to Task. (I originally had several foreign keys in Task and Project but pared things down until it worked and tried slowly adding things back. Adding the second ForeignKey did it no matter which table it was added to.)
My current Flask method is:
#app.route('/project/<int:project_id>/')
def showProject(project_id):
project = session.query(Project).from_statement(text("SELECT * FROM project WHERE project.id = project_id ORDER BY project.projnum DESC"))
tasks = session.query(Task).filter_by(project_id = project_id).all()
return render_template('project.html', project = project, tasks = tasks)
And I get a "no such column" error:
OperationalError: (sqlite3.OperationalError) no such column: task.project_id [SQL: u'SELECT task.id AS task_id, task.name AS task_name, task.description AS task_description, task.assigned_id AS task_assigned_id, task.project_id AS task_project_id, task.due AS task_due \nFROM task \nWHERE task.project_id = ?'] [parameters: (2,)]
This is the setup file for my database:
import os
import sys
import time
from sqlalchemy import Column, ForeignKey, Integer, String, Float
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import relationship
from sqlalchemy import create_engine
Base = declarative_base()
engine = create_engine('sqlite:///gmnpm.db')
class User(Base):
__tablename__ = 'user'
id = Column(Integer, primary_key=True)
name = Column(String(250), nullable=False)
email = Column(String(250), nullable=False)
picture = Column(String(250))
class Task(Base):
__tablename__ = 'task'
id = Column(Integer, primary_key=True)
name = Column(String(250), nullable=False)
description = Column(String(2000))
assigned_id = Column(Integer, ForeignKey('user.id'))
project_id = Column(Integer, ForeignKey('project.id'))
due = Column(Integer, default=int(time.time()))
class Project(Base):
__tablename__ = 'project'
id = Column(Integer, primary_key=True)
name = Column(String(250), nullable=False)
description = Column(String(1000))
projnum = Column(Integer)
Base.metadata.create_all(engine)
After reading a bunch of documentation and going crazy trying random things on the off chance they work, I'm grateful for any suggestions!
I'm learning SQLAlchemy and I want to make sure that I've understood the backref parameter in relationship correctly.
For example
from app import db
class User(db.Model):
id = db.Column(db.Integer, primary_key=True)
username = db.Column(db.String(64), unique=True)
posts = db.relationship('Post', backref='author', lazy=True)
class Post(db.Model):
id = db.Column(db.Integer, primary_key=True)
body = db.Column(db.String(140))
user_id = db.Column(db.Integer, db.ForeignKey('user.id'))
Say I have a User object j = models.User.query.get(1). My question is, is there any difference between the following things?
j.posts
Post.query.filter_by(author=j).all()
Post.query.with_parent(j).all()
Post.query.with_parent(j, property='posts').all()
Post.query.with_parent(j, property=User.posts).all()
The results returned are same, but I don't know whether the SQL statements executed are identical.
What I've tried
The SQLAlchemy docs says:
with_parent(instance, property=None, from_entity=None)
...the given property can be None, in which case a search is performed against this Query object’s target mapper.
So the last three statements seem same, but I don't really understand what does this Query object’s target mapper refer to. Is it Post in this case, for this query is performed on Post?
Even if the generated SQL statements are identical, the commands you enlisted may have a different impact on your application, e.g. j.posts will cache (memoize, do not confuse with Werkzeug caching) results you have got, while others will fetch them every single time.
If you remove .all() from your queries you can simply print them:
query = Post.query.filter_by(author=j)
print(query)
Which would result in:
SELECT post.id AS post_id, post.body AS post_body, post.user_id AS post_user_id
FROM post
WHERE ? = post.user_id
Using .all() is essentially like getting [m for m in query]).
The trick with query-printing will not work for j.posts which will return something like:
> print(j.posts)
> [Post(...), Post(..)]
Still, you can see all the silently emitted queries using built-in sqlalchemy loggers. See the following code:
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from sqlalchemy.engine import Engine
from sqlalchemy import event
import logging
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:////tmp/tests.db'
db = SQLAlchemy(app)
logging.basicConfig()
logger = logging.getLogger('sqlalchemy.engine')
class User(db.Model):
id = db.Column(db.Integer, primary_key=True)
username = db.Column(db.String(64), unique=True)
posts = db.relationship('Post', backref='author', lazy=True)
class Post(db.Model):
id = db.Column(db.Integer, primary_key=True)
body = db.Column(db.String(140))
user_id = db.Column(db.Integer, db.ForeignKey('user.id'))
db.drop_all()
db.create_all()
user = User(username='test', posts=[Post(body='some body')])
db.session.add(user)
db.session.commit()
# start logging
logger.setLevel(logging.DEBUG)
j = User.query.get(1)
queries = {
"j.posts",
"Post.query.filter_by(author=j)",
"Post.query.with_parent(j)",
"Post.query.with_parent(j, property='posts')",
"Post.query.with_parent(j, property=User.posts)",
}
def test_queries():
for name in queries:
print('\n=======')
print('Executing %s:' % name)
query = eval(name)
print(query)
test_queries() # you should see j.posts query here
print('Second test')
test_queries() # but not here
Getting back to your question: yes, the emitted SQL queries are identical.
In Query object’s target mapper, Query object's target refers to Post in your example. Decoupling this, when you declare Post class, inheriting from db.Model, for SQLAlchemy it is like creating an object Post and mapping the properties of this object to columns of specially created table.
Underneath there is an instance of Mapper class, which is responsible for the mapping for every single model that you create (learn more about mapping here: Types of Mappings). You can simply get this mapper calling class_mapper on your model or object_mapper on an instance of your model:
from sqlalchemy.orm import object_mapper, class_mapper,
from sqlalchemy.orm.mapper import Mapper
assert object_mapper(j) is class_mapper(User)
assert type(class_mapper(User)) is Mapper
The Mapper has all the necessary information about the columns and relations you have in your model. When calling Post.query.with_parent(j) this information is used to find a property (i.e. relationship) relating Post and User objects, so in your case to populate 'property' with User.posts.
To see the queries you can run your python script with -i and then run each query individually and it will print out the SQL code it runs.
Example:
main.py:
import sqlalchemy
from sqlalchemy import create_engine, Column, Integer, String, Sequence
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
import os
engine = create_engine('sqlite:///:memory:', echo=True)
Base = declarative_base()
class User(Base):
__tablename__ = 'users'
id = Column(Integer, Sequence('user_id_seq'), primary_key=True)
name = Column(String(50))
fullname = Column(String(50))
password = Column(String(12))
def __repr__(self):
return "< User(name={}, fullname={}, password={} )>".format(self.name, self.fullname, self.password)
Base.metadata.create_all(engine)
ed_user= User(name='ed', fullname='Ed Jones', password='edpassword')
Session = sessionmaker(bind=engine, autoflush=False)
session = Session()
session.add(ed_user)
session.add_all([
User(name='wendy', fullname='Wendy Williams', password='foobar'),
User(name='mary', fullname='Mary Contraty', password='xxg527'),
User(name='fred', fullname='Fred Flinstone', password='blah')
])
session.commit()
os.system('clear')
Now you run it with python -i main.py, type: session.query(User).filter_by(name='ed').first() and you will see the SQL generated. After running all of your tests I concluded that they are all identical. With this method you can test any query and see if there is any difference.
p.s. I added the os.system('clear') to remove all the unnecessary output from creating the database and some other stuff.
I have two relationships to the same table. When I add an element to one relationship, this does not reflect to the other relationship until I submit the session. Is there a way to force "update" the relationships?
Concrete example:
class Event(ManagerBase):
"""Defines an event."""
__tablename__ = 'eventing_events'
id = Column(Integer, primary_key=True)
device_id = Column(Integer, ForeignKey(EventingDevice.id), nullable=False)
device = relation(EventingDevice)
type_id = Column(Integer, ForeignKey(EventType.id), nullable=False)
type = relation(EventType)
datetime = Column(DateTime, nullable=False)
summary = Column(String(500))
fields = relation("EventFieldValue",
viewonly=True,
collection_class=attribute_mapped_collection("field.name"))
class EventFieldValue(ManagerBase):
"""The value of a single field of an event."""
__tablename__ = 'eventing_event_field_values'
event_id = Column(Integer, ForeignKey(Event.id), primary_key=True)
event = relation(Event, backref=backref("field_values",
collection_class=attribute_mapped_collection("field")))
field_id = Column(Integer, ForeignKey(Field.id), primary_key=True)
field = relation(Field)
value = Column(Text)
I have two realations from Event to EventFieldValue: fields and field_values (via backref of event). When I add a EventFieldValue to event.field_values, it does not reflect in event.fields until I commit the session.
Because you have two relations, sqlalchemy have to make requests for each one, and doesn't share their cache in the session.
You should take a look at Association proxies, that seems to be exactly what you need. They allow you to define only one relation and to put proxies on the top of them to access stuff in the relation more easily.
Flushing the session should solve this problem. It updates your session with all the new state but doesn't do a commit. You can also look into Refresh/Expire which will reload your objects.