We Keep Coding

Is it possible to use session.insert for one to main relationships in SQLAlchemy?

I have read in the following link:
Sqlalchemy adding multiple records and potential constraint violation
That using SQLAlchemy core library to perform the inserts is much faster option, rather than the ORM's session.add() method:
i.e:
session.add()
should be replaced with:
session.execute(Entry.__table__.insert(), params=inserts)
In the following code I have tried to replace .add with .insert:
from sqlalchemy import Column, DateTime, String, Integer, ForeignKey, func
from sqlalchemy.orm import relationship, backref
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class Department(Base):
__tablename__ = 'department'
id = Column(Integer, primary_key=True)
name = Column(String)
class Employee(Base):
__tablename__ = 'employee'
id = Column(Integer, primary_key=True)
name = Column(String)
# Use default=func.now() to set the default hiring time
# of an Employee to be the current time when an
# Employee record was created
hired_on = Column(DateTime, default=func.now())
department_id = Column(Integer, ForeignKey('department.id'))
# Use cascade='delete,all' to propagate the deletion of a Department onto its Employees
department = relationship(
Department,
backref=backref('employees',
uselist=True,
cascade='delete,all'))
from sqlalchemy import create_engine
engine = create_engine('postgres://blah:blah#blah:blah/blah')
from sqlalchemy.orm import sessionmaker
session = sessionmaker()
session.configure(bind=engine)
Base.metadata.create_all(engine)
d = Department(name="IT")
emp1 = Employee(name="John", department=d)
s = session()
s.add(d)
s.add(emp1)
s.commit()
s.delete(d) # Deleting the department also deletes all of its employees.
s.commit()
s.query(Employee).all()
# Insert Option Attempt
from sqlalchemy.dialects.postgresql import insert
d = insert(Department).values(name="IT")
d1 = d.on_conflict_do_nothing()
s.execute(d1)
emp1 = insert(Employee).values(name="John", department=d1)
emp1 = emp1.on_conflict_do_nothing()
s.execute(emp1)
The error I receive:
sqlalchemy.exc.CompileError: Unconsumed column names: department
I can't quite understand the syntax and how to do it in the right way, I'm new to the SQLAlchemy.
It looks my question is similar to How to get primary key columns in pd.DataFrame.to_sql insertion method for PostgreSQL "upsert"
, so potentially by answering either of our questions, you could help two people at the same time ;-)

I am new to SQLAlchemy as well, but this is what I found :
Using your exact code, adding department only didn't work using "s.execute(d1)", so I changed it to the below and it does work :
with engine.connect() as conn:
d = insert(Department).values(name="IT")
d1 = d.on_conflict_do_nothing()
conn.execute(d1)
I found on SQLAlchemy documentation that in the past it was just a warning when you try to use a virtual column that doesn't really exist. But from version 0.8, it has been changed to an exception.
As a result, I am not sure if you can do that using the insert. I think that SQLAlchemy does it behind the scene in some other way when using session.add(). Maybe some experts can elaborate here.
I hope that will help.

SQLAlchemy: how to create dynamic columns multiple times

I succeeded to create a sql table with columns defined dynamically, thanks to python class reflexion.
But I cannot run the code more than one time.
For instance, the following import_file , should create a static table and a dynamic table with specific columns.
It works if I run it one time. But the second time it crashs and returns the following error :
Table 'dynamic' is already defined for this MetaData instance
Code example :
from sqlalchemy import Column, Integer, String, Float, Boolean, Table
from sqlalchemy import create_engine, MetaData
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker, relationship
from sqlalchemy.orm import clear_mappers
import os
def import_file(filename, columns):
path = filename
if os.path.exists(path):
os.remove(path)
engine = create_engine(f"sqlite:///{path}", echo=True)
clear_mappers()
Base = declarative_base()
class StaticTable(Base):
__tablename__ = "static"
id = Column(Integer, primary_key=True)
name = Column(String)
class DynamicTable(Base):
__tablename__ = "dynamic"
id = Column(Integer, primary_key=True)
for c in columns:
setattr(DynamicTable,c,Column(String))
Base.metadata.create_all(engine)
import_file("test.db", columns = ["age","test"]) # WORKS
import_file("test2.db", columns= ["id","age","foo","bar"]) # NOT WORKING
I try to use sqlalchemy.orm.clear_mappers, but unsucessfully.
Any idea how can I resolve it ?

I think that it is not full you code, because conflicts on Base.metadata
Mappers for link on orm classes and tables, you have defined tables.
You can try somethink like that
import_file("test.db", columns = ["age","test"])
Base.metadata.clear()
sqlalchemy.orm.clear_mappers()
import_file("test2.db", columns= ["id","age","foo","bar"])

How to merge an object into the session based on ID using SQLAlchemy while keeping correct discriminator of child classes?

I am passing object IDs between threads, and would like to be able to merge an object into the session by ID. I could of course query objects across threads to accomplish this but would rather avoid the overhead of a database trip if the object is in the session already.
My code is working, but it is not correctly setting the discriminator value when I merge by the parent class. How can I ensure the discriminator is set correctly?
from sqlalchemy import create_engine, __version__
from sqlalchemy.orm import sessionmaker, scoped_session
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column
from sqlalchemy.types import Integer, String
Session = scoped_session(sessionmaker())
Base = declarative_base()
class Person(Base):
__tablename__ = 'person'
id = Column(Integer, primary_key=True)
name = Column(String, nullable=False)
discriminator = Column(String, nullable=False)
__mapper_args__ = {'polymorphic_identity': 'person',
'polymorphic_on': discriminator}
class Programmer(Person):
__mapper_args__ = {'polymorphic_identity': 'programmer'}
fav_language = Column(String)
engine = create_engine('postgresql+zxjdbc://mnaber:test123#localhost:5432/ajtest2', echo=True)
Session.configure(bind=engine)
Base.metadata.bind = engine
Base.metadata.drop_all(checkfirst=True)
Base.metadata.create_all()
s = Session()
michael = Programmer(name='Michael', fav_language='Python')
s.add(michael)
s.commit()
print "Sqlalchemy Version: %s" % __version__
print "INITIAL: %s %s" % (type(michael), michael.discriminator)
michael_merged = s.merge(Person(id=michael.id)) #Merge by parent class
print "MERGED: %s %s" % (type(michael_merged), michael_merged.discriminator)
print "FINAL: %s %s" % (type(michael), michael.discriminator)
michael_merged.fav_language = 'Jython'
s.add(michael_merged)
s.commit()
The interesting output of this is:
Sqlalchemy Version: 0.8.7
INITIAL: <class '__main__.Programmer'> programmer
MERGED: <class '__main__.Programmer'> person
FINAL: <class '__main__.Programmer'> person
site-packages/sqlalchemy/orm/persistence.py:154: SAWarning: Flushing object <Programmer at 0x18> with incompatible polymorphic identity 'person'; the object may not refresh and/or load correctly
mapper._validate_polymorphic_identity(mapper, state, dict_)
How should I ensure that MERGED and FINAL have a discriminator of programmer?

merge() assumes an object coming in looks the way you want it to look so you'd need to pass a Programmer here. Passing in Person(id=1) when there's really a Programmer in the database for that identity is not a supported pattern, the behavior is undefined. What happens at the moment is that your Person object has the "discriminator" of "person" set up front, so that's the value that gets flushed into the database; it overrides what's already in the session.
You can trick it into working like this:
p1 = Person(id=michael.id)
del p1.discriminator
michael_merged = s.merge(p1) #Merge by parent class
however, I can't guarantee that code like the above will always work for future SQLAlchemy versions. It would not, for example, work for earlier versions where the "discriminator" was chosen at flush time.
Your code example is such that Programmer is already present in the Session; you can get this object in a class-agnostic way like this:
obj = session.query(Person).get(michael.id)
then you have your Programmer object and you can modify it freely.

SQLAlchemy association table (association object pattern) raises IntegrityError

I'm using SQLAlchemy release 0.8.2 (tried python 2.7.5 and 3.3.2)
I've had to use the association object pattern (for a many-to-many relationship) in my code, but whenever I've been adding an association, it has been raising an IntegrityError exception. This is because instead of executing "INSERT INTO association (left_id, right_id, extra_data) [...]", it instead executes "INSERT INTO association (right_id, extra_data) [...]", which is going to raise an IntegrityError exception since it's missing a primary key.
After trying to narrow down the problem for a while and simplifying the code as much as possible, I found the culprit(s?), but I don't understand why it's behaving this way.
I included my complete code so the reader can test it as is. The class declarations are exactly the same as in the documentation (with backrefs).
#!/usr/bin/env python2
import sqlalchemy
from sqlalchemy.orm import sessionmaker
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, Integer, String
from sqlalchemy import ForeignKey
from sqlalchemy.orm import relationship, backref
Base = declarative_base()
class Association(Base):
__tablename__ = 'association'
left_id = Column(Integer, ForeignKey('left.id'), primary_key=True)
right_id = Column(Integer, ForeignKey('right.id'), primary_key=True)
extra_data = Column(String(50))
child = relationship("Child", backref="parent_assocs")
class Parent(Base):
__tablename__ = 'left'
id = Column(Integer, primary_key=True)
children = relationship("Association", backref="parent")
class Child(Base):
__tablename__ = 'right'
id = Column(Integer, primary_key=True)
def main():
engine = sqlalchemy.create_engine('sqlite:///:memory:', echo=True)
Base.metadata.create_all(engine)
Session = sessionmaker(bind=engine)
session = Session()
# populate old data
session.add(Child())
# new data
p = Parent()
session.add(p) # Commenting this fixes the error.
session.flush()
# rest of new data
a = Association(extra_data="some data")
a.child = session.query(Child).one()
# a.child = Child() # Using this instead of the above line avoids the error - but that's not what I want.
p.children.append(a)
# a.parent = p # Using this instead of the above line fixes the error! They're logically equivalent.
session.add(p)
session.commit()
if __name__ == '__main__':
main()
So, as mentioned in the comments in the code above, there are three ways to fix/avoid the problem.
Don't add the parent to the session before declaring the association
Create a new child for the association instead of selecting an already existing child.
Use the backref on the association
I don't understand the behaviour of all three cases.
The second case does something different, so it's not a possible solution. I don't understand the behaviour however, and would appreciate an explanation of why the problem is avoided in this case.
I'm thinking the first case may have something to do with "Object States", but I don't know exactly what's causing it either. Oh, and adding session.autoflush=False just before the first occurrence of session.add(p) also fixes the problem which adds to my confusion.
For the third case, I'm drawing a complete blank since they should be logically equivalent.
Thanks for any insight!

what happens here is that when you call upon p.children.append(), SQLAlchemy can't append to a plain collection without loading it first. As it goes to load, autoflush kicks in - you know this because in your stack trace you will see a line like this:
File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/orm/session.py", line 1183, in _autoflush
self.flush()
Your Association object is then flushed here in an incomplete state; it's in the session in the first place because when you say a.child = some_persistent_child, an event appends a to the parent_assocs collection of Child which then cascades the Association object into the session (see Controlling Cascade on Backrefs for some background on this, and one possible solution).
But without affecting any relationships, the easiest solution when you have this chicken/egg sort of problem is to temporarily disable autoflush using no_autoflush:
with session.no_autoflush:
p.children.append(a)
by disabling the autoflush when p.children is loaded, your pending object a is not flushed; it is then associated with the already persistent Parent (because you've added and flushed that already) and is ready for INSERT.
this allows your test program to succeed.

How to set up global connection to database?

I have problem with setting up database connection. I want to set connection, where I can see this connection in all my controllers.
Now I use something like this in my controller:
db = create_engine('mysql://root:password#localhost/python')
metadata = MetaData(db)
email_list = Table('email',metadata,autoload=True)
In development.ini I have:
sqlalchemy.url = mysql://root#password#localhost/python
sqlalchemy.pool_recycle = 3600
How do I set _____init_____.py?

I hope you got pylons working; for anyone else that may later read question I'll present some pointers in the right direction.
First of all, you are only creating a engine and a metadata object. While you can use the engine to create connections directly you would almost always use a Session to manage querying and updating your database.
Pylons automatically setups this for you by creating a engine from your configuration file, then passing it to yourproject.model.__init__.py:init_model() which binds it to a scoped_session object.
This scoped_session object is available from yourproject.model.meta and is the object you would use to query your database. For example:
record = meta.Session.query(model.MyTable).filter(id=42)
Because it is a scoped_session it automatically creates a Session object and associates it with the current thread if it doesn't already exists. Scoped_session passes all action (.query(), .add(), .delete()) down into the real Session object and thus allows you a simple way to interact the database with having to manage the non-thread-safe Session object explicitly.
The scoped_session, Session, object from yourproject.model.meta is automatically associated with a metadata object created as either yourproject.model.meta:metadata (in pylons 0.9.7 and below) or yourproject.model.meta:Base.metadata (in pylons 1.0). Use this metadata object to define your tables. As you can see in newer versions of pylons a metadata is associated with a declarative_base() object named Base, which allows you to use SqlAlchemy's declarative style.
Using this from the controller
from yourproject import model
from yourproject.model import Session
class MyController(..):
def resource(self):
result = Session.query(model.email_list).\
filter(model.email_list.c.id=42).one()
return str(result)
Use real connections
If you really want to get a connection object simply use
from yourproject.model import Session
connection = Session.connection()
result = connection.execute("select 3+4;")
// more connection executions
Session.commit()
However this is all good, but what you should be doing is...
This leaves out that you are not really using SqlAlchemy much. The power of SqlAlchemy really shines when you start mapping your database tables to python classes. So anyone looking into using pylons with a database should take a serious look at what you can do with SqlAlchemy. If SqlAlchemy starts out intimidating simply start out with using its declarative approach, which should be enough for almost all pylons apps.
In your model instead of defining Table constructs, do this:
from sqlalchemy import Column, Integer, Unicode, ForeignKey
from sqlalchemy.orm import relation
from yourproject.model.meta import Base
class User(Base):
__tablename__ = 'users'
# primary_key implies nullable=False
id = Column(Integer, primary_key=True, index=True)
# nullable defaults to True
name = Column(Unicode, nullable=False)
notes = relation("UserNote", backref="user")
query = Session.query_property()
class UserNote(Base):
__tablename__ = 'usernotess'
# primary_key implies nullable=False
id = Column(Integer, primary_key=True, index=True)
userid = Column(Integer, index=True, ForeignKey("User.id"))
# nullable defaults to True
text = Column(Unicode, nullable=False)
query = Session.query_property()
Note the query objects. These are smart object that live on the class and associates your classes with the scoped_session(), Session. This allows you to event more easily extract data from your database.
from sqlalchemy.orm import eagerload
def resource(self):
user = User.query.filter(User.id==42).options(eagerload("notes")).one()
return "\n".join([ x.text for x in user.notes ])

1.0 version of Pylons use declarative syntax. More about this, you can see here .
In mode/init.py you can write somthing like this:
from your_programm.model.meta import Session, Base
from sqlalchemy import *
from sqlalchemy.types import *
def init_model(engine):
Session.configure(bind=engine)
class Foo(Base) :
__tablename__ = "foo"
id = Column(Integer, primary_key=True)
name = Column(String)
...

What you want to do is modify the Globals class in your app_globals.py file to include a .engine (or whatever) attribute. Then, in your controllers, you use from pylons import app_globals and app_globals.engine to access the engine (or metadata, session, scoped_session, etc...).