I have a question concerning the mapping of entities in SQLAlchemy.
I have a transient object, which already contains foreign keys to some persistent objects. I want that SQLAlchemy fetches the referenced objects and assigns them to their relationship-attributes. From the SQLAlchemy documentation, I thought that I have to use the merge-operation on the session to achieve this. But in my configuration, it doesn't work.
This is a minimum example demonstrating my problem:
# -*- coding: utf-8 -*-
from sqlalchemy import create_engine
from sqlalchemy.orm import mapper
from sqlalchemy import Table, Column, Integer, String, MetaData, ForeignKey
from sqlalchemy.orm import relationship, sessionmaker
class User(object):
def __init__(self, id, name, fullname, password, best_friend_id=None):
self.id = id
self.name = name
self.fullname = fullname
self.password = password
self.best_friend_id = best_friend_id
def __repr__(self):
return "<User('%s','%s', '%s')>" % (self.name, self.fullname, self.password)
class Dog(object):
def __init__(self, id, name):
self.id = id
self.name = name
def __repr__(self):
return "<User('%s','%s', '%s')>" % (self.name, self.fullname, self.password)
engine = create_engine('sqlite:///:memory:', echo=True)
Session = sessionmaker(bind=engine)
session = Session()
metadata = MetaData()
dogs_table = Table('dogs', metadata,
Column('id', Integer, primary_key=True),
Column('name', String),
)
users_table = Table('users', metadata,
Column('id', Integer, primary_key=True),
Column('name', String),
Column('fullname', String),
Column('password', String),
Column('best_friend_id', Integer, ForeignKey('dogs.id'))
)
metadata.create_all(engine)
mapper(User, users_table, properties={'best_friend': relationship(Dog, uselist=False)})
mapper(Dog, dogs_table)
dog = Dog(id=1, name='Hasso')
lordling = User(id=2, name='John', fullname='Miller', password='very_secret', best_friend_id=1)
session.add(dog)
session.commit()
merged_lordling = session.merge(lordling)
print str(merged_lordling.best_friend.name)
I expect that merged_lordling.best_friend contains the dog 'Hasso'. But it is still None.
I was bit by this same problem recently. Once you established a relationship, you should simply assign your Dog instance to User.best_friend directly, not explicitly using the foreign key. I don't know why exactly that happens, but while investigating a similar problem I realized that if you do that, SQLAlchemy doesn't populate the relationship property until you flushed all the related instances.
So, instead of:
dog = Dog(id=1, name='Hasso')
lordling = User(id=2, name='John', fullname='Miller', password='very_secret',
best_friend_id=1)
session.add(dog)
Simply do:
dog = Dog(id=1, name='Hasso')
lordling = User(id=2, name='John', fullname='Miller', password='very_secret',
best_friend=dog)
session.add(lordling)
Or even:
lordling = User(id=2, name='John', fullname='Miller', password='very_secret',
best_friend=Dog(id=1, name='Hasso'))
session.add(lordling)
As a general rule, avoid using the foreign key columns directly when you have a relationship established. Embrace the ORM, and only assign or query directly from foreign keys when you really have no other choice. I learned that the hard way.
Related
I'm trying to do a large number of inserts with one call, and the way someone here recommended was by giving .insert a list of dictionaries. This is using SQLAlchemy Core.
As an example:
try:
engine = db.create_engine(f"postgres://user:pass#myip/addressbook", connect_args={'connect_timeout': 5})
connection = engine.connect()
metadata = db.MetaData()
except exc.OperationalError:
print_error(f":: Could not connect to myip!")
sys.exit()
table_addressbook = db.Table('addressbook', metadata, autoload=True, autoload_with=engine)
list = []
list.append({'firstname': "John", 'lastname': "Doe"})
list.append({'firstname': "Jane", 'lastname': "Doe"})
query = db.insert(table_addressbook).values(list)
connection.execute(query)
But I'm getting an error saying the column id violates a non-null constraint. This is because insert normally auto-generates the primary-key id. How do I use this method but specify that id should be auto-generated? Or is there a different method I should use?
edit
Table name is addressbook.
Column id is type integer with default sequence 'untitled_table_id_seq', constraints are PRIMARY_KEY. This was autogenerated by Postico for Mac, but I've always been able to insert without including id and it auto increments from the last inserted ID.
Columns firstname and lastname are type text, no default, no constraints.
Without any information on your model and/or connection it is a bit difficult to answer your question. Please find below a piece of code which uses insert without throwing non-null constraint errors. Hopefully it helps you.
from sqlalchemy import create_engine, Column, Integer, String, Table
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
from sqlalchemy.sql import insert
engine = create_engine('sqlite:///:memory:', echo=True)
Base = declarative_base()
class User(Base):
__tablename__ = 'users'
id = Column(Integer, primary_key=True)
firstname = Column(String)
lastname = Column(String)
Base.metadata.create_all(engine)
Session = sessionmaker(bind=engine)
Session.configure(bind=engine) # once engine is available
session = Session()
new_users = []
new_users.append({'firstname': "John", 'lastname': "Doe"})
new_users.append({'firstname': "Jane", 'lastname': "Doe"})
i = insert(User).values(new_users)
session.execute(i)
PS: most of this is coming from the tutorial on: https://docs.sqlalchemy.org/en/13/orm/tutorial.html
from sqlalchemy import Column
from sqlalchemy import create_engine
from sqlalchemy import Integer
from sqlalchemy import String
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import Session
engine = create_engine('sqlite:///:memory:', echo=True)
Base = declarative_base()
# Example Model definition for the illustration
class Customer(Base):
__tablename__ = "customer"
id = Column(Integer, primary_key=True)
name = Column(String(255))
description = Column(String(255))
Base.metadata.create_all(engine)
######################################################
# Bulk insert using dictionaries.
######################################################
# Insert test records into `customer`table.
def bulk_insert_customers(n):
session = Session(bind=engine)
session.bulk_insert_mappings(
Customer,
[
dict(
name="customer name %d" % i,
description="customer description %d" % i,
)
for i in range(n)
],
)
session.commit()
Refer these for more examples of how to do bulk inserts in different ways:
https://docs.sqlalchemy.org/en/13/_modules/examples/performance/bulk_inserts.html
I tired to use the SqlAlchemy orm to build the api to insert the values into database from uploaded excel files. when I tested on the codes it kept showing the error:
TypeError: __init__() missing 1 required positional argument: 'id'
I've updated the id key to primary key, auto increment, unique and unsigned in my local MySql data base. I believe the system cannot insert the primary key automatically because it works if I assign the value to id manually
transaction_obj = Transaction(id=1, name="David", date="2018-03-03",
product="fruit", quantity=20, amount=12.55)
Here is model.py
from sqlalchemy import Table, MetaData, Column, Integer, String, DATE, DECIMAL,ForeignKey, DateTime
from sqlalchemy.orm import mapper
metadata = MetaData()
customers = Table('customers', metadata,
Column('id', Integer, primary_key=True),
Column('name', String(20)),
Column('phone', String(20)),
Column('address', String(45)),
Column('source_from', String(45))
)
class Customers(object):
def __init__(self, name, phone, address, source_from):
self.name = name
self.phone = phone
self.address = address
self.source_from = source_from
def __repr__(self):
return "<Customer(name='%s', phone='%s', address='%s', " \
"source_from='%s')" % (self.name, self.phone, self.address,
self.source_from)
mapper(Customers, customers)
transaction = Table('transaction', metadata,
Column('id', Integer, primary_key=True),
Column('name', String(20)),
Column('date', DateTime),
Column('product', String(20)),
Column('quantity', Integer),
Column('amount',DECIMAL(2))
)
class Transaction(object):
def __index__(self, name, date, product, quantity, amount):
self.name = name
self.date = date
self.product = product
self.quantity = quantity
self.amount = amount
def __repr__(self):
return "<Transaction(name='%s', date='%s', product='%s'," \
"quantity='%s', amount='%s')>" % (self.name, self.date,
self.product, self.quantity,
self.amount)
mapper(Transaction, transaction)
Here is my test coding: test.py
import json
import os
import os
import json
from sqlalchemy import create_engine
import config
import pandas as pd
conn = config.conn_str
def tran_test():
engine = create_engine(conn)
Session_class = sessionmaker(bind=engine)
Session = Session_class
# generate the object for the data we would like to insert
transaction_obj = Transaction(name="David", date="2018-03-03",
product="fruit", quantity=20, amount=12.55)
Session.add(transaction_obj)
Session.commit()
def test_uploaded_file(file):
df = pd.read_excel(file)
return df.info()
if __name__ == '__main__':
# set_env_by_setting('prod')
# conn_str = os.environ.get('ConnectionString')
# print(conn_str)
# test_uploaded_file("-1.xlsx")
tran_test()
I'm using SQLAlchemy==1.2.10, PyMySQL==0.9.2.
I'm doubting if I'm using the wrong format in model.py. Please advise. Thx.
While I'm not sure about the pattern you are using, (manually mapping to your table classes) I think you would have a much easier time making use of declarative_base which does this for you.
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
Then make sure your models inherit Base
from sqlalchemy import (
Column,
Integer,
String
)
class Customers(Base):
__tablename__ = 'customer'
id = Column(Integer, primary_key=True) # Auto-increment should be default
name = Column(String(20))
# Etc.
def __repr__(self):
return "<Customer(name='%s', phone='%s', address='%s', " \
"source_from='%s')" % (self.name, self.phone, self.address,
self.source_from)
And finally use Base to create your table:
Base.metadata.create_all(engine)
Here is a good reference to basic declarative use cases. It gets a little more complicated depending on how you are scaffolding your app but its a great starting point:
http://docs.sqlalchemy.org/en/latest/orm/extensions/declarative/basic_use.html
I'm using Python Sqlalchemy for MYSQL db. I wrote the following script to create the class object and then added some rows in the table.
from sqlalchemy import create_engine, MetaData, Table, Column, ForeignKey
from sqlalchemy.dialects.mysql.base import VARCHAR, LONGTEXT, INTEGER
from sqlalchemy.orm import sessionmaker
from sqlalchemy.ext.declarative import declarative_base
engine = create_engine("mysql+mysqldb://root:#localhost/mydb")
connection = engine.connect()
Session = sessionmaker(bind=engine)
session = Session()
Base = declarative_base()
metadata = MetaData()
class User(Base):
__tablename__ = 'User'
id = Column('id', INTEGER(display_width=11), primary_key=True, nullable=False)
email = Column('email', VARCHAR(charset='utf8mb4', collation='utf8mb4_0900_ai_ci', length=100), unique=True)
password = Column('password', VARCHAR(charset='utf8mb4', collation='utf8mb4_0900_ai_ci', length=45))
name = Column('name', VARCHAR(charset='utf8mb4', collation='utf8mb4_0900_ai_ci', length=100))
Now, I need to get all the rows from the table "User" so I am doing this:
user = session.query(User).all()
print(user)
but the output I am getting is not the table data but this:
[<__main__.User object at 0x7f10b0c6ebe0>, <__main__.User object at 0x7f10b0c6ec50>]
How would I get the actual data from the table? Any help would be appreciated
The output you will get is a tuple of records.
So, use a loop
users = session.query(User).all()
for user in users:
print (user)
print (user.id, user.email, user.password, user.name)
you should write __str__ method in User class something like:
class User(Base):
...
def __str__(self):
str_out = 'id={} email={} password={} name={}'
str_formated = str_out.format(self.id,self.email,self.password,self.name)
return str_formated
I'm trying to set up a secondary many-to-many relationship from one table to two others, via a third in the middle that links to all three. I have two files - one for ORM objects (model.py) and one for schema objects (schema.py) They look like this:
model.py
import schema
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import *
class AbstractBase(object):
def __repr__(self):
macros = ["%s=%s" % (key, getattr(self, key, None)) for key in self.__table__.columns.keys()]
rep = "<%s(%s)>" % (self.__class__.__name__, str.join(', ', macros))
return rep
Base = declarative_base(cls=AbstractBase)
class A(Base):
__table__ = schema.a_table
dees = relationship("D",
secondary=schema.b_table,
primaryjoin="A.a_id==b_table.c.a_id",
secondaryjoin="b_table.c.c_id==D.d_id")
cees = relationship("C",
secondary=schema.b_table,
primaryjoin="A.a_id==schema.b_table.c.a_id",
secondaryjoin="b_table.c.d_id==C.c_id",
backref="a_collection")
class C(Base):
__table__ = schema.c_table
class D(Base):
__table__ = schema.d_table
schema.py
from sqlalchemy import *
from sqlalchemy.dialects.mysql import *
metadata = MetaData()
a_table = Table(
'a',
metadata,
Column("a_id", INTEGER(), primary_key=True, nullable=False),
Column("date", DATETIME(timezone=True)),
)
b_table = Table(
'shipment_runs',
metadata,
Column("a_id", ForeignKey("a.a_id"), primary_key=True,),
Column("c_id", ForeignKey("c.c_id"), primary_key=True),
Column("d_id", ForeignKey("d.d_id")),
)
c_table = Table(
'c',
metadata,
Column('c_id', INTEGER(), primary_key=True, nullable=False),
Column('name', VARCHAR(64), unique=True),
)
d_table = Table(
'd',
metadata,
Column('d_id', INTEGER(), primary_key=True, nullable=False)
)
Unfortunately, instantiating this results in the following error:
sqlalchemy.exc.InvalidRequestError: When initializing mapper Mapper|A|a, expression 'A.a_id==b_table.c.a_id' failed to locate a name ("name 'b_table' is not defined"). If this is a class name, consider adding this relationship() to the class after both dependent classes have been defined.
Is there a way I can change my imports or make the mapper be aware of the objects in the schema module somehow?
Was able to get it by doing the following:
class B(Base):
__table__ = schema.b_table
class A(Base):
__table__ = schema.a_table
dees = relationship("D",
secondary=b.__table__,
primaryjoin="A.a_id==B.a_id",
secondaryjoin="B.c_id==D.d_id")
cees = relationship("C",
secondary=B.__table__,
primaryjoin="A.a_id==B.a_id",
secondaryjoin="B.d_id==C.c_id",
backref="a_collection")
All credit goes to this question:
SQLAlchemy Relationship Error: object has no attribute 'c'
In general, you can use the table name in a string, or drop the string and use your actual references.
primaryjoin="A.a_id==shipment_runs.c.a_id",
primaryjoin=schema.a_table.c.a_id==schema.b_table.c.a_id,
That being said, given that you have the ForeignKeys set up in your tables, SQLAlchemy is smart enough that you don't even need the joins for a simple relationship, just secondary.
c_list = relationship("C", secondary=schema.b_table, backref="a_list")
(I think the "C" and "D" are swapped in your example?)
I would like to have a 'relationship' in an inherited (mixin) class.
However, when I create the inherited object, the relationship object is None. I cannot append to it.
How do I resolve this?
Here is code based upon the documentation
from sqlalchemy import Column, Integer, String, DateTime, Boolean, BigInteger, Float
from sqlalchemy import ForeignKey
from sqlalchemy.orm import relationship, backref
from sqlalchemy.ext.declarative import declared_attr
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class Target(Base):
__tablename__ = "target"
id = Column(Integer, primary_key=True)
class RefTargetMixin(object):
#declared_attr
def target_id(cls):
return Column('target_id', ForeignKey('target.id'))
#declared_attr
def target(cls):
return relationship("Target",
primaryjoin="Target.id==%s.target_id" % cls.__name__
)
class Foo(RefTargetMixin, Base):
__tablename__ = 'foo'
id = Column(Integer, primary_key=True)
print repr(RefTargetMixin.target)
print repr(Foo.target)
print repr(Foo().target)
The output is:
<sqlalchemy.orm.properties.RelationshipProperty object at 0x24e7890>
<sqlalchemy.orm.attributes.InstrumentedAttribute object at 0x24e7690>
None
In general, I should be able to append to the relationship object (target), but here I cannot because it is None. Why?
the reason the value is None is because you've defined this as a many-to-one relationship. Many-to-one, from parent-to-child, means there is a foreign key on the parent, which can only refer to one and only one child. If you'd like something of class RefTargetMixin to refer to a collection of items, then foreign keys must be on the remote side.
So then the goal here is to make any object that is a subclass of RefTargetMixin be a potential parent for a Target. This pattern is called the polymorphic association pattern. While it is common in many ORM toolkits to provide this by declaring a "polymorphic foreign key" on Target, this is not a good practice relationally, so the answer is to use multiple tables in some way. There are three scenarios for this provided in SQLAlchemy core in the examples/generic_association folder, including "single association table with discriminator", "table per association", and "table per related". Each pattern provides the identical declarative pattern for RefTargetMixin here but the structure of the tables changes.
For example, here is your model using "table per association", which in my view tends to scale the best provided you don't need to query multiple types of RefTargetMixin objects at once (note I literally used the example as is, just changed the names):
from sqlalchemy.ext.declarative import declarative_base, declared_attr
from sqlalchemy import create_engine, Integer, Column, \
String, ForeignKey, Table
from sqlalchemy.orm import Session, relationship
class Base(object):
"""Base class which provides automated table name
and surrogate primary key column.
"""
#declared_attr
def __tablename__(cls):
return cls.__name__.lower()
id = Column(Integer, primary_key=True)
Base = declarative_base(cls=Base)
class Target(Base):
pass
class RefTargetMixin(object):
#declared_attr
def targets(cls):
target_association = Table(
"%s_targets" % cls.__tablename__,
cls.metadata,
Column("target_id", ForeignKey("target.id"),
primary_key=True),
Column("%s_id" % cls.__tablename__,
ForeignKey("%s.id" % cls.__tablename__),
primary_key=True),
)
return relationship(Target, secondary=target_association)
class Customer(RefTargetMixin, Base):
name = Column(String)
class Supplier(RefTargetMixin, Base):
company_name = Column(String)
engine = create_engine('sqlite://', echo=True)
Base.metadata.create_all(engine)
session = Session(engine)
session.add_all([
Customer(
name='customer 1',
targets=[
Target(),
Target()
]
),
Supplier(
company_name="Ace Hammers",
targets=[
Target(),
]
),
])
session.commit()
for customer in session.query(Customer):
for target in customer.targets:
print target
This is the normal behaviour : Foo has one Target. When you create the Foo object, it has no Target yet, so the value of Foo().target is None.
If you want Foo to have multiple Targets, you should put a foo_id in Target, and not a target_id in Foo, and use a backref.
Also, in that case, it is not needed to specify the primary join.