We Keep Coding

null value for primary key when trying to insert list of dictionaries

I'm trying to do a large number of inserts with one call, and the way someone here recommended was by giving .insert a list of dictionaries. This is using SQLAlchemy Core.
As an example:
try:
engine = db.create_engine(f"postgres://user:pass#myip/addressbook", connect_args={'connect_timeout': 5})
connection = engine.connect()
metadata = db.MetaData()
except exc.OperationalError:
print_error(f":: Could not connect to myip!")
sys.exit()
table_addressbook = db.Table('addressbook', metadata, autoload=True, autoload_with=engine)
list = []
list.append({'firstname': "John", 'lastname': "Doe"})
list.append({'firstname': "Jane", 'lastname': "Doe"})
query = db.insert(table_addressbook).values(list)
connection.execute(query)
But I'm getting an error saying the column id violates a non-null constraint. This is because insert normally auto-generates the primary-key id. How do I use this method but specify that id should be auto-generated? Or is there a different method I should use?
edit
Table name is addressbook.
Column id is type integer with default sequence 'untitled_table_id_seq', constraints are PRIMARY_KEY. This was autogenerated by Postico for Mac, but I've always been able to insert without including id and it auto increments from the last inserted ID.
Columns firstname and lastname are type text, no default, no constraints.

Without any information on your model and/or connection it is a bit difficult to answer your question. Please find below a piece of code which uses insert without throwing non-null constraint errors. Hopefully it helps you.
from sqlalchemy import create_engine, Column, Integer, String, Table
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
from sqlalchemy.sql import insert
engine = create_engine('sqlite:///:memory:', echo=True)
Base = declarative_base()
class User(Base):
__tablename__ = 'users'
id = Column(Integer, primary_key=True)
firstname = Column(String)
lastname = Column(String)
Base.metadata.create_all(engine)
Session = sessionmaker(bind=engine)
Session.configure(bind=engine) # once engine is available
session = Session()
new_users = []
new_users.append({'firstname': "John", 'lastname': "Doe"})
new_users.append({'firstname': "Jane", 'lastname': "Doe"})
i = insert(User).values(new_users)
session.execute(i)
PS: most of this is coming from the tutorial on: https://docs.sqlalchemy.org/en/13/orm/tutorial.html

from sqlalchemy import Column
from sqlalchemy import create_engine
from sqlalchemy import Integer
from sqlalchemy import String
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import Session
engine = create_engine('sqlite:///:memory:', echo=True)
Base = declarative_base()
# Example Model definition for the illustration
class Customer(Base):
__tablename__ = "customer"
id = Column(Integer, primary_key=True)
name = Column(String(255))
description = Column(String(255))
Base.metadata.create_all(engine)
######################################################
# Bulk insert using dictionaries.
######################################################
# Insert test records into `customer`table.
def bulk_insert_customers(n):
session = Session(bind=engine)
session.bulk_insert_mappings(
Customer,
[
dict(
name="customer name %d" % i,
description="customer description %d" % i,
)
for i in range(n)
],
)
session.commit()
Refer these for more examples of how to do bulk inserts in different ways:
https://docs.sqlalchemy.org/en/13/_modules/examples/performance/bulk_inserts.html

SQLAlchemy: relationship collection lazy loading

As per the SQLAlchemy documentation on relationship loading:
When the given collection or reference is first accessed on a particular object, an additional SELECT statement is emitted such that the requested collection is loaded.
How do I achieve loading behavior such that only the single elements of a relationship collection that I access are loaded, rather than the entire collection all at once?
I have heard of deferred column loading; this would be more like "deferred row loading". Rather than deferring loading of attributes, I'd like to defer loading of relationship collection elements.
Desired use case:
# Persist instance.
coln = Collection([1, 2, 3])
session.add(coln)
session.commit()
# Test lazy loading.
print('data' in coln.__dict__)
# Lazy loads the entire collection. I'd like only one element.
print(coln.data[1])
# Will output: "True 3". I'd like: "True 1".
print('data' in coln.__dict__, len(coln.__dict__['data']))
Class definitions and other backwork:
from sqlalchemy import Column, Integer, ForeignKey
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import relationship
Base = declarative_base()
engine = create_engine('sqlite:///:memory:')
# Define classes.
class Collection(Base):
__tablename__ = 'collection'
id = Column(Integer, primary_key=True)
data = relationship('Element')
def __init__(self, list_):
self.data = [Element(e) for e in list_]
class Element(Base):
__tablename__ = 'element'
id = Column(Integer, primary_key=True)
parent_id = Column(Integer, ForeignKey('collection.id'))
value = Column(Integer)
def __init__(self, value):
self.value = value
def __repr__(self):
return 'Element({})'.format(self.value)
# Create schema.
Base.metadata.create_all(engine)
# Create session.
from sqlalchemy.orm import sessionmaker
Session = sessionmaker(bind=engine)
session = Session()

Use the lazy parameter with dynamic value:
data = relationship('Element', lazy='dynamic')
https://docs.sqlalchemy.org/en/13/orm/collections.html#dynamic-relationship

Change SQLAlchemy Primary Key after it has been defined

Problem: Simply put, I am trying to redefine a SQLAlchemy ORM table's primary key after it has already been defined.
Example:
class Base:
#declared_attr
def __tablename__(cls):
return f"{cls.__name__}"
#declared_attr
def id(cls):
return Column(Integer, cls.seq, unique=True,
autoincrement=True, primary_key=True)
Base = declarative_base(cls=Base)
class A_Table(Base):
newPrimaryKeyColumnsDerivedFromAnotherFunction = []
# Please Note: as the variable name tries to say,
# these columns are auto-generated and not known until after all
# ORM classes (models) are defined
# OTHER CLASSES
def changePriKeyFunc(model):
pass # DO STUFF
# Then do
Base.metadata.create_all(bind=arbitraryEngine)
# After everything has been altered and tied into a little bow
*Please note, this is a simplification of the true problem I am trying to solve.
Possible Solution: Your first thought might have been to do something like this:
def possibleSolution(model):
for pricol in model.__table__.primary_key:
pricol.primary_key = False
model.__table__.primary_key = PrimaryKeyConstraint(
*model.newPrimaryKeyColumnsDerivedFromAnotherFunction,
# TODO: ADD all the columns that are in the model that are also a primary key
# *[col for col in model.__table__.c if col.primary_key]
)
But, this doesn't work, because when trying to add, flush, and commit, an error gets thrown:
InvalidRequestError: Instance <B_Table at 0x104aa1d68> cannot be refreshed -
it's not persistent and does not contain a full primary key.
Even though this:
In [2]: B_Table.__table__.primary_key
Out[2]: PrimaryKeyConstraint(Column('a_TableId', Integer(),
ForeignKey('A_Table.id'), table=<B_Table>,
primary_key=True, nullable=False))
as well as this:
In [3]: B_Table.__table__
Out[3]: Table('B_Table', MetaData(bind=None),
Column('id', Integer(), table=<B_Table>, nullable=False,
default=Sequence('test_1', start=1, increment=1,
metadata=MetaData(bind=None))),
Column('a_TableId', Integer(),
ForeignKey('A_Table.id'), table=<B_Table>,
primary_key=True, nullable=False),
schema=None)
and finally:
In [5]: b.a_TableId
Out[5]: 1
Also note that the database actually reflects the changed (and true) primary key, so I know that there's something going on with the ORM/SQLAlchemy.
Question: In summary, how can I change the model's primary key after the model has already been defined?
edit: See below for full code (same type of error, just in SQLite)
from sqlalchemy import Column, Integer, ForeignKey
from sqlalchemy.orm import relationship, sessionmaker
from sqlalchemy.ext.declarative import declared_attr, declarative_base
from sqlalchemy.schema import PrimaryKeyConstraint
from sqlalchemy import Sequence, create_engine
class Base:
#declared_attr
def __tablename__(cls):
return f"{cls.__name__}"
#declared_attr
def seq(cls):
return Sequence("test_1", start=1, increment=1)
#declared_attr
def id(cls):
return Column(Integer, cls.seq, unique=True, autoincrement=True, primary_key=True)
Base = declarative_base(cls=Base)
def relate(model, x):
"""Model is the original class, x is what class needs to be as
an attribute for model"""
attributeName = x.__tablename__
idAttributeName = "{}Id".format(attributeName)
setattr(model, idAttributeName,
Column(ForeignKey(x.id)))
setattr(model, attributeName,
relationship(x,
foreign_keys=getattr(model, idAttributeName),
primaryjoin=getattr(
model, idAttributeName) == x.id,
remote_side=x.id
)
)
return model.__table__.c[idAttributeName]
def possibleSolution(model):
if len(model.defined):
newPriCols = []
for x in model.defined:
newPriCols.append(relate(model, x))
for priCol in model.__table__.primary_key:
priCol.primary_key = False
priCol.nullable = True
model.__table__.primary_key = PrimaryKeyConstraint(
*newPriCols
# TODO: ADD all the columns that are in the model that are also a primary key
# *[col for col in model.__table__.c if col.primary_key]
)
class A_Table(Base):
pass
class B_Table(Base):
defined = [A_Table]
possibleSolution(B_Table)
engine = create_engine('sqlite://')
Base.metadata.create_all(bind=engine)
Session = sessionmaker(bind=engine)
session = Session()
a = A_Table()
b = B_Table(A_TableId=a.id)
print(B_Table.__table__.primary_key)
session.add(a)
session.commit()
session.add(b)
session.commit()

Originally, the error you say the PK reassignment is causing is:
InvalidRequestError: Instance <B_Table at 0x104aa1d68> cannot be refreshed -
it's not persistent and does not contain a full primary key.
I don't get that running you MCVE, instead I get a pretty helpful warning first:
SAWarning: Column 'B_Table.A_TableId' is marked as a member of the
primary key for table 'B_Table', but has no Python-side or server-side
default generator indicated, nor does it indicate 'autoincrement=True'
or 'nullable=True', and no explicit value is passed. Primary key
columns typically may not store NULL.
And a very detailed exception message when the script fails:
sqlalchemy.orm.exc.FlushError: Instance has
a NULL identity key. If this is an auto-generated value, check that
the database table allows generation of new primary key values, and
that the mapped Column object is configured to expect these generated
values. Ensure also that this flush() is not occurring at an
inappropriate time, such as within a load() event.
So assuming that the example accurately describes your problem, the answer is straightforward. A primary key cannot be null.
A_Table inherits off Base:
class A_Table(Base):
pass
Base gives A_Table an autoincrement PK through declared_attr id():
#declared_attr
def id(cls):
return Column(Integer, cls.seq, unique=True, autoincrement=True, primary_key=True)
Similarly, B_Table is defined off Base but the PK is overwritten in possibleSolution() such that it becomes a ForeignKey to A_Table:
PrimaryKeyConstraint(Column('A_TableId', Integer(), ForeignKey('A_Table.id'), table=<B_Table>, primary_key=True, nullable=False))
Then, we instantiate an instance of A_Table without any kwargs and immediately allocate the id attribute of instance a to field A_TableId when constructing b:
a = A_Table()
b = B_Table(A_TableId=a.id)
At this point we can stop and inspect the attribute values of each:
print(a.id, b.A_TableId)
# None None
a.id is None because it's an autoincrement which needs to be populated by the database, not the ORM. So SQLAlchemy doesn't know it's value until after the instance is flushed to the database.
So what happens if we include a flush() operation after adding instance a to the session:
a = A_Table()
session.add(a)
session.flush()
b = B_Table(A_TableId=a.id)
print(a.id, b.A_TableId)
# 1 1
So by issuing the flush first, we've got a value for a.id, meaning that we also have a value for b.A_TableId.
session.add(b)
session.commit()
# no error

SQLAlchemy inheritance with relationship is None in instantiated object

I would like to have a 'relationship' in an inherited (mixin) class.
However, when I create the inherited object, the relationship object is None. I cannot append to it.
How do I resolve this?
Here is code based upon the documentation
from sqlalchemy import Column, Integer, String, DateTime, Boolean, BigInteger, Float
from sqlalchemy import ForeignKey
from sqlalchemy.orm import relationship, backref
from sqlalchemy.ext.declarative import declared_attr
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class Target(Base):
__tablename__ = "target"
id = Column(Integer, primary_key=True)
class RefTargetMixin(object):
#declared_attr
def target_id(cls):
return Column('target_id', ForeignKey('target.id'))
#declared_attr
def target(cls):
return relationship("Target",
primaryjoin="Target.id==%s.target_id" % cls.__name__
)
class Foo(RefTargetMixin, Base):
__tablename__ = 'foo'
id = Column(Integer, primary_key=True)
print repr(RefTargetMixin.target)
print repr(Foo.target)
print repr(Foo().target)
The output is:
<sqlalchemy.orm.properties.RelationshipProperty object at 0x24e7890>
<sqlalchemy.orm.attributes.InstrumentedAttribute object at 0x24e7690>
None
In general, I should be able to append to the relationship object (target), but here I cannot because it is None. Why?

the reason the value is None is because you've defined this as a many-to-one relationship. Many-to-one, from parent-to-child, means there is a foreign key on the parent, which can only refer to one and only one child. If you'd like something of class RefTargetMixin to refer to a collection of items, then foreign keys must be on the remote side.
So then the goal here is to make any object that is a subclass of RefTargetMixin be a potential parent for a Target. This pattern is called the polymorphic association pattern. While it is common in many ORM toolkits to provide this by declaring a "polymorphic foreign key" on Target, this is not a good practice relationally, so the answer is to use multiple tables in some way. There are three scenarios for this provided in SQLAlchemy core in the examples/generic_association folder, including "single association table with discriminator", "table per association", and "table per related". Each pattern provides the identical declarative pattern for RefTargetMixin here but the structure of the tables changes.
For example, here is your model using "table per association", which in my view tends to scale the best provided you don't need to query multiple types of RefTargetMixin objects at once (note I literally used the example as is, just changed the names):
from sqlalchemy.ext.declarative import declarative_base, declared_attr
from sqlalchemy import create_engine, Integer, Column, \
String, ForeignKey, Table
from sqlalchemy.orm import Session, relationship
class Base(object):
"""Base class which provides automated table name
and surrogate primary key column.
"""
#declared_attr
def __tablename__(cls):
return cls.__name__.lower()
id = Column(Integer, primary_key=True)
Base = declarative_base(cls=Base)
class Target(Base):
pass
class RefTargetMixin(object):
#declared_attr
def targets(cls):
target_association = Table(
"%s_targets" % cls.__tablename__,
cls.metadata,
Column("target_id", ForeignKey("target.id"),
primary_key=True),
Column("%s_id" % cls.__tablename__,
ForeignKey("%s.id" % cls.__tablename__),
primary_key=True),
)
return relationship(Target, secondary=target_association)
class Customer(RefTargetMixin, Base):
name = Column(String)
class Supplier(RefTargetMixin, Base):
company_name = Column(String)
engine = create_engine('sqlite://', echo=True)
Base.metadata.create_all(engine)
session = Session(engine)
session.add_all([
Customer(
name='customer 1',
targets=[
Target(),
Target()
]
),
Supplier(
company_name="Ace Hammers",
targets=[
Target(),
]
),
])
session.commit()
for customer in session.query(Customer):
for target in customer.targets:
print target

This is the normal behaviour : Foo has one Target. When you create the Foo object, it has no Target yet, so the value of Foo().target is None.
If you want Foo to have multiple Targets, you should put a foo_id in Target, and not a target_id in Foo, and use a backref.
Also, in that case, it is not needed to specify the primary join.

How to automatically add a SQLAlchemy object to the session?

I have a SQLAlchemy table class created using the Declarative method:
mysqlengine = create_engine(dsn)
session = scoped_session(sessionmaker(bind=mysqlengine))
Base = declarative_base()
Base.metadata.bind = mysqlengine
class MyTable(Base):
__table_args__ = {'autoload' : True}
Now, when using this table within the code I would like to not have to use the session.add method in order to add each new record to the active session so instead of:
row = MyTable(1, 2, 3)
session.add(row)
session.commit()
I would like to have:
row = MyTable(1, 2, 3)
session.commit()
Now, I know of this question already: Possible to add an object to SQLAlchemy session without explicit session.add()?
And, I realize you can force this behavior by doing the following:
class MyTable(Base):
def __init__(self, *args, **kw):
super(MyTable, self).__init__(*args, **kw)
session.add(self)
However, I do not want to bloat my code containing 30 tables with this method. I also know that Elixir ( http://elixir.ematia.de/trac/wiki ) does this so it must be possible in some sense.

Super simple. Use an event:
from sqlalchemy import event, Integer, Column, String
from sqlalchemy.orm import scoped_session, sessionmaker, mapper
from sqlalchemy.ext.declarative import declarative_base
Session = scoped_session(sessionmaker())
#event.listens_for(mapper, 'init')
def auto_add(target, args, kwargs):
Session.add(target)
Base = declarative_base()
class A(Base):
__tablename__ = "a"
id = Column(Integer, primary_key=True)
data = Column(String)
a1 = A(data="foo")
assert a1 in Session()