We Keep Coding

Self referencing many-to-many relationship with extra column in association object

I am new in Sqlalchemy and trying to achieve the following goal with relationship():
There is an User table which stores user data.
Every user is able to invite other user with an invite_code.
Every user keeps a list of invitation, every invitation includes the invite_code and the invitee User
I think the relationship between User and Invitation is one-to-many. Since Invitation contains User, then I think it is probably better to use self-referential relationship to represent the inviter-to-invitaions(invitees) relationship and use an association object to store the invite_code.
I checked the sqlalchemy documentation and the question, tried to implement the classed like this:
from sqlalchemy import Column, Integer, ForeignKey, create_engine, String
from sqlalchemy.orm import relationship, sessionmaker
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class Invitation(Base):
__tablename__ = 'invitation'
invite_code = Column(Integer)
inviter_id = Column(Integer, ForeignKey('user.id'), primary_key=True)
invitee_id = Column(Integer, ForeignKey('user.id'), primary_key=True)
invitee = relationship('User') #Need HELP here
class User(Base):
__tablename__ = 'user'
id = Column(Integer, primary_key=True)
name = Column(String)
inviters = relationship('User',
secondary='invitation',
primaryjoin=id==Invitation.invitee_id,
secondaryjoin=id==Invitation.inviter_id,
backref='invitees')
invitations = relationship('Invitation')# Need HELP here
def __repr__(self):
return f'User: {self.name}'
if __name__ == '__main__':
engine = create_engine('sqlite://')
Base.metadata.create_all(engine)
Session = sessionmaker(engine)
db = Session()
inviter1 = User(name='inviter1')
inviter2 = User(name='inviter2')
invitee1= User(name='invitee1')
invitee2 = User(name='invitee2')
inviter1.invitees = [invitee1, invitee2]
inviter2.invitees = [invitee1]
db.add(inviter1)
db.add(inviter2)
db.add(invitee1)
db.add(invitee2)
db.commit()
users = db.query(User).all()
for user in users:
print(user)
print(' Inviter: ', user.inviters)
print(' Invitee: ', user.invitees)
print()
If the lines with comment #Need HELP here are deleted, I can get the corresponding inviters and invitees, but cannot get the invite_code. If the #Need HELP here code are added, the error is:
Exception has occurred: AmbiguousForeignKeysError
Could not determine join condition between parent/child tables on relationship Invitation.invitee - there are multiple foreign key paths linking the tables. Specify the 'foreign_keys' argument, providing a list of those columns which should be counted as containing a foreign key reference to the parent table.
Is there a way to add extra data column in association object like association object for many-to-many relationship for self referential table?
Sorry for the too much text, I didn't find any reference document on the web.

Finally, I figured it out with the help of foreign_keys:
from sqlalchemy import Column, Integer, ForeignKey, create_engine, String
from sqlalchemy.orm import relationship, sessionmaker
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class User(Base):
__tablename__ = 'user'
id = Column(Integer, primary_key=True)
name = Column(String)
sent_invitations = relationship('Invitation', foreign_keys='Invitation.inviter_id', back_populates='inviter', cascade='all, delete')
received_invitations=relationship('Invitation', foreign_keys='Invitation.invitee_id', back_populates='invitee', cascade='all, delete')
def __repr__(self):
return f'User: {self.name}'
class Invitation(Base):
__tablename__ = 'invitation'
id = Column(Integer, primary_key=True)
invite_code = Column(Integer)
inviter_id = Column(Integer, ForeignKey('user.id'))
invitee_id = Column(Integer, ForeignKey('user.id'))
inviter=relationship('User', foreign_keys=[inviter_id], back_populates='sent_invitations')
invitee=relationship('User', foreign_keys=[invitee_id], back_populates='received_invitations')
def __repr__(self):
return f'Invitation: {self.inviter} invited {self.invitee} with {self.invite_code}'
if __name__ == '__main__':
engine = create_engine('sqlite://')
Base.metadata.create_all(engine)
Session = sessionmaker(engine)
db = Session()
inviter1 = User(name='inviter1')
inviter2 = User(name='inviter2')
invitee1= User(name='invitee1')
invitee2 = User(name='invitee2')
invitation1 = Invitation(invite_code=50, inviter=inviter1, invitee=invitee1)
invitation2 = Invitation(invite_code=20, inviter=inviter2, invitee=invitee2)
invitation3 = Invitation(invite_code=22, inviter=inviter1, invitee=inviter2)
invitation4 = Invitation(invite_code=44, inviter=invitee1, invitee=inviter2)
db.add(inviter1)
db.add(inviter2)
db.add(invitee1)
db.add(invitee2)
db.commit()
users = db.query(User).all()
for user in users:
print(user)
print(' sent_invitation: ', user.sent_invitations)
print(' received_invitation: ', user.received_invitations)
print()
invitations = db.query(Invitation).all()
for invitation in invitations:
print(invitation)
db.delete(inviter1)
db.delete(invitee2)
db.commit()

SQLALchemy: Where does the attribute error after data import come from?

I wrote an application using SQLAlchemy's object relational mapper to store and access data from an SQLite3 database.
I can call add_userto add one or multiple users and call get_users to get them
I can import data from excel and get them with get_users
I can import data from excel and add a user with add_user
BUT I can't get the users with the get_users function afterwards, because I'm getting the following error for the entry created with add_user: AttributeError: 'NoneType' object has no attribute 'id'
What am I doing wrong?
Here's a simple version of the application:
orm_test.py
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
def orm_setup():
Base = declarative_base()
engine = create_engine('sqlite:///:main:', echo=True)
Base.metadata.create_all(bind=engine)
Session = sessionmaker(bind=engine)
session = Session()
return Base, engine, session
orm_test_class.py
from sqlalchemy import Column, Integer, String
from orm_test import orm_setup
Base = orm_setup()[0]
engine = orm_setup()[1]
class User(Base):
__tablename__ = 'person'
id = Column('id', Integer, primary_key=True)
username = Column('username', String, unique=True)
Base.metadata.create_all(bind=engine)
orm_test_functions.py
from orm_test_class import User
from orm_test import orm_setup
session = orm_setup()[2]
def add_user(name):
u = User()
user_name = str(name)
u.username = user_name
session.add(u)
session.commit()
def get_users():
users = session.query(User).all()
for user in users:
print(user.id, user.username)
session.close()
main.py
import fire
from orm_test_functions import add_user, get_users
if __name__ == '__main__':
fire.Fire()
data_import.py
import fire
import pandas as pd
from orm_test import orm_setup
# import engine from orm
engine = orm_setup()[1]
def data_import():
file = 'Data.xlsx'
df_user = pd.read_excel(file, sheet_name = 'User')
df_user.to_sql('person', engine, if_exists='replace', index=False)
# Command line interface
if __name__ == '__main__':
fire.Fire()

The problem is that df_to_sql drops the original table, which has a primary key defined, and replaces it with a table that does not define a primary key.
From the dataframe_to_sql docs
replace: Drop the table before inserting new values.
You can get around this by setting if_exists='append' instead of if_exists='replace'.
df_user.to_sql('person', engine, if_exists='append', index=False)
If necessary you can emulate the "replace" behaviour by deleting any existing records from the table before importing the data.
This is the code I used to reproduce and resolve:
import io
import sqlalchemy as sa
from sqlalchemy import orm
import pandas as pd
Base = orm.declarative_base()
class User(Base):
__tablename__ = 'person'
id = sa.Column('id', sa.Integer, primary_key=True)
username = sa.Column('username', sa.String, unique=True)
engine = sa.create_engine('sqlite://', echo=True, future=False)
# Drop all is redundant for in-memory db
Base.metadata.drop_all(engine)
Base.metadata.create_all(engine)
sessionmaker = orm.sessionmaker(engine)
def add_user(name):
session = sessionmaker()
u = User()
user_name = str(name)
u.username = user_name
session.add(u)
session.commit()
def get_users():
session = sessionmaker()
users = session.query(User).all()
for user in users:
print(user.id, user.username)
print()
session.close()
DATA = """\
id,username
1,Alice
2,Bob
"""
buf = io.StringIO(DATA)
df_user = pd.read_csv(buf)
df_user.to_sql('person', engine, if_exists='append', index=False)
users = get_users()
add_user('Carol')
users = get_users()
engine.dispose()

You should set the column id with AUTOINCREMENT keyword, see https://docs.sqlalchemy.org/en/14/dialects/sqlite.html#using-the-autoincrement-keyword

null value for primary key when trying to insert list of dictionaries

I'm trying to do a large number of inserts with one call, and the way someone here recommended was by giving .insert a list of dictionaries. This is using SQLAlchemy Core.
As an example:
try:
engine = db.create_engine(f"postgres://user:pass#myip/addressbook", connect_args={'connect_timeout': 5})
connection = engine.connect()
metadata = db.MetaData()
except exc.OperationalError:
print_error(f":: Could not connect to myip!")
sys.exit()
table_addressbook = db.Table('addressbook', metadata, autoload=True, autoload_with=engine)
list = []
list.append({'firstname': "John", 'lastname': "Doe"})
list.append({'firstname': "Jane", 'lastname': "Doe"})
query = db.insert(table_addressbook).values(list)
connection.execute(query)
But I'm getting an error saying the column id violates a non-null constraint. This is because insert normally auto-generates the primary-key id. How do I use this method but specify that id should be auto-generated? Or is there a different method I should use?
edit
Table name is addressbook.
Column id is type integer with default sequence 'untitled_table_id_seq', constraints are PRIMARY_KEY. This was autogenerated by Postico for Mac, but I've always been able to insert without including id and it auto increments from the last inserted ID.
Columns firstname and lastname are type text, no default, no constraints.

Without any information on your model and/or connection it is a bit difficult to answer your question. Please find below a piece of code which uses insert without throwing non-null constraint errors. Hopefully it helps you.
from sqlalchemy import create_engine, Column, Integer, String, Table
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
from sqlalchemy.sql import insert
engine = create_engine('sqlite:///:memory:', echo=True)
Base = declarative_base()
class User(Base):
__tablename__ = 'users'
id = Column(Integer, primary_key=True)
firstname = Column(String)
lastname = Column(String)
Base.metadata.create_all(engine)
Session = sessionmaker(bind=engine)
Session.configure(bind=engine) # once engine is available
session = Session()
new_users = []
new_users.append({'firstname': "John", 'lastname': "Doe"})
new_users.append({'firstname': "Jane", 'lastname': "Doe"})
i = insert(User).values(new_users)
session.execute(i)
PS: most of this is coming from the tutorial on: https://docs.sqlalchemy.org/en/13/orm/tutorial.html

from sqlalchemy import Column
from sqlalchemy import create_engine
from sqlalchemy import Integer
from sqlalchemy import String
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import Session
engine = create_engine('sqlite:///:memory:', echo=True)
Base = declarative_base()
# Example Model definition for the illustration
class Customer(Base):
__tablename__ = "customer"
id = Column(Integer, primary_key=True)
name = Column(String(255))
description = Column(String(255))
Base.metadata.create_all(engine)
######################################################
# Bulk insert using dictionaries.
######################################################
# Insert test records into `customer`table.
def bulk_insert_customers(n):
session = Session(bind=engine)
session.bulk_insert_mappings(
Customer,
[
dict(
name="customer name %d" % i,
description="customer description %d" % i,
)
for i in range(n)
],
)
session.commit()
Refer these for more examples of how to do bulk inserts in different ways:
https://docs.sqlalchemy.org/en/13/_modules/examples/performance/bulk_inserts.html

python sqlalchemy not filling in the row

Hello I am new with SQLalchemy and have some problems with inserting data in a column.
import sqlalchemy
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, Integer, String, ForeignKey, Boolean
from sqlalchemy.orm import relationship, backref
from sqlalchemy.orm import sessionmaker
engine = create_engine('sqlite:///test.db')
Base = declarative_base()
class Users(Base):
__tablename__ = 'users'
id = Column(Integer, primary_key=True)
name = Column(String, unique=True)
password = Column(String)
email = Column(String)
def __init__(self, name, password, email):
self._name = name
self._password = password
self._email = email
Base.metadata.create_all(engine)
Session = sessionmaker(bind=engine)
Session.configure(bind=engine)
session = Session()
dm_user = Users("Dungeonmaster", "123", "email")
print dm_user.name
print dm_user.password
print dm_user.email
session.add(dm_user)
our_user = session.query(Users).filter_by(name='Dungeonmaster').first()
session.commit()
I used sqlite studio to see if the data is added, and I see that a new row is being made (with a new id.) but the data name, password and email is not inserted.
With the print I see even before I try to add something goes wrong, but I don't know what. please help me out a bit
( dm_user = Users("Dungeonmaster", "123", "email") tried with single quotes as well (helped me before using sqlalchemy, but no difference here)

Your init method does not initialize the persistent columns. Try instead (remove underscores):
def __init__(self, name, password, email):
self.name = name
self.password = password
self.email = email

query related entities on sqlalchemy (many-to-many)

i have users and entities (many-to-many) , and using sqlalchemy, with this model:
from sqlalchemy import Table, Column, Unicode, Integer, ForeignKey
from sqlalchemy.orm import relationship
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import scoped_session, sessionmaker
from zope.sqlalchemy import ZopeTransactionExtension
DBSession = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))
Base = declarative_base()
users_entities = Table('users_entities', Base.metadata,
Column('userID', Integer, ForeignKey('users.id')),
Column('entitieID', Integer, ForeignKey('entities.id'))
)
class User(Base):
__tablename__ = 'users'
id = Column(Integer, primary_key=True)
username = Column(Unicode(20))
password = Column(Unicode(101))
entities = relationship("Entities", secondary=users_entities)
def __init__(self, username, password):
self.username = username
self.password = password
class Entities(Base):
__tablename__ = 'entities'
id = Column(Integer, primary_key=True)
name = Column(Unicode(20))
descr = Column(Unicode(101))
url = Column(Unicode(101))
def __init__(self, name, descr, url):
self.name = name
self.descr = descr
self.url = url
so, when i use:
user = dbsession.query(User).filter_by(id=session["userID"]).first()
entities = user.entities
i get the user with the user data, and the entities(user.entities) with all the user's entities.
but instead of getting all the entities, now i need to get an entity from the user where the id is = X
something that would work like this:
user.query(Entities).filter_by(id=X)
i can't find a simple("best") way to do this, am i missing something?

You are looking for Dynamic Relationship Loaders, they're exactly what you want to do, and enable you to write:
user.entities.filter_by(id=X)
By the way, you can write user = dbsession.query(User).get(session["userID"]). It does the same query, but only if the object is not already in the session cache (plus, it's shorter).