When I try to reinitialize my database, I keep getting an IntegrityError:
sqlalchemy.exc.IntegrityError: (IntegrityError) (1217, 'Cannot delete or update a parent row: a foreign key constraint fails') '\nDROP TABLE users' ()
In my init_db script, I have:
def main(argv=sys.argv):
if len(argv) != 2:
usage(argv)
config_uri = argv[1]
setup_logging(config_uri)
settings = get_appsettings(config_uri)
engine = create_engine('...')
Session.configure(bind=engine)
session = Session()
Base.metadata.bind=engine
Base.metadata.drop_all(engine)
Base.metadata.create_all(engine)
It would let me drop the Users table, which is:
class User(Base):
__tablename__='users'
__table_args__={
'mysql_engine':'InnoDB',
'mysql_charset':'utf8',
}
user_id = Column(Integer, primary_key=True, unique=True)
username = Column(VARCHAR(16), primary_key=True, unique=True)
name = Column(VARCHAR(16))
password = Column(VARCHAR(20))
email = Column(VARCHAR(30))
creation = Column(DateTime, default = datetime.datetime.now)
def __init__(self, un, name, pw, email):
self.username = un
self.name = name
self.email = email
self.password = pw
I see two possible reasons:
Unmapped table: You have another table in the database which has a ForeignKey to the users table, but which is not mapped to any sqlalchemy model and therefore is not dropped during drop_all operation.
Missing ForeignKey mapping: Some relationship is missing in the sqlalchemy model. For example, the addresses table on the DB has a ForeignKey which points to the users table, but your model Address (or to put it properly, the mapper for the Address) does not configure this ForeignKey, and therefore during drop_all sqlalchemy has no way of knowing that this table should be dropped before the referenced one (users).
Related
I can return the name of the table of which a foreign key references, but I want to do the opposite.
Suppose I have two tables, Users and Address, and Address has a foreign key to Users (one-to-many). I also made this bidirectional so that I can get the User from the address table.
class User(Base):
__tablename__ = 'user'
id = Column(Integer, primary_key=True)
name = Column(String)
address = relationship("Address", back_populates="user")
class Address(Base):
__tablename__ = 'address'
id = Column(Integer, primary_key=True, autoincrement=True)
company_name = Column(String)
user_id = Column(Integer, ForeignKey('user.id'))
user = relationship("User", back_populates="address")
If I wanted to figure out the name of the parent table, in a way that works for any other table in a similar relationship, I can to this:
for i in Address.__table__.foreign_key_constraints:
print(i.referred_table)
tab = Table(Address.__tablename__, Base.metadata, autoload_with=engine, extend_existing=True)
for i in tab.foreign_key_constraints:
print(i.referred_table)
#both output "user" which is the __tablename__ of User
This was me accessing the User table using the foreign key constraint attribute. But how do I do the opposite, using an attribute from User to access Address? I mainly want to know so I can handle relationships in a bulk core insert.
tab = Table(User.__tablename__, Base.metadata, autoload_with=engine, extend_existing=True)
records = df.to_dict(orient="records")
insert_stmt = sqlalchemy.dialects.sqlite.insert(tab).values(records)
#list of primary keys
pks = [pk.name for pk in tab.primary_key]
update_columns = {col.name: col for col in insert_stmt.excluded if col.name not in pks}
update_statement = insert_stmt.on_conflict_do_update(index_elements=pks, set_=update_columns)
The above works once I execute the on_conflict_do_update statement, but it does not generate any Address rows.
I found one way of doing it, but it's not as clean as I wanted.
tab = list(inspect(User).relationships)[0].entity.local_table
For any declarative table (classes), you can inspect them to get a Mapper Object. The Mapper Object has relationships as an attribute, which returns a collection of relationships.
inspect(User).relationships
#Get the relationship, note this will return the relationship attribute from the class
list(inspect(User).relationships)[0]
So I then get the mapper/class this relationship belongs to with .entity. Then from the declarative table, a table object is returned with the local_table attribute
entity = list(inspect(User).relationships)[0].entity
tab = entity.local_table
This statement should evaluate to True
tab == Address.__table__
I'm new to the SQLAlchemy and I wrote a simple CRUD database layout like this: I have three tables, Customer, Product, and Template. The idea is this: Each customer can have a template of the products he usually orders. When I fetch a particular customer from the database, his template along with all the products should be fetched as well. So I have a one-to-one relationship (customer-template) and one many-to-many relationship (template-product). Now, a template should contain fields such as quantity of a particular product, along with its net, gross and tax values. I'd like to have a listener on the quantity column, such that when the quantity of a particular product is changed, other attributes I mentioned will be changed too. So the code I wrote is as follows (please, if you can, also verify whether all the relationships are written appropriately)
from sqlalchemy import event
from sqlalchemy.orm import relationship, exc, column_property, sessionmaker
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm.query import Query as _Query
from sqlalchemy import Table, Column, Integer, String, Boolean, ForeignKey, UniqueConstraint, create_engine, Numeric
from decimal import *
# decimal operations settings
getcontext().prec = 6
getcontext().rounding = ROUND_HALF_UP
engine = create_engine('sqlite:///test.db')
Base = declarative_base()
# Initalize the database if it is not already.
Base.metadata.create_all(engine)
# Create a session to handle updates.
Session = sessionmaker(bind=engine)
# customer has a one-to-one relationship with template, where customer is the parent
class Customer(Base):
__tablename__ = "customers"
id = Column(Integer, primary_key=True)
alias = Column(String)
firm_name = Column(String)
last_name = Column(String)
first_name = Column(String)
tax_id = Column(String, nullable=False)
address = Column(String)
postal_code = Column(String)
city = Column(String)
payment = Column(Boolean)
template = relationship("Template", uselist=False, back_populates="customer")
# product has many-to-many relationship with template, where template is the parent
class Product(Base):
__tablename__ = "products"
id = Column(Integer, primary_key=True)
product_name = Column(String, nullable=False)
symbol = Column(String)
unit = Column(String, nullable=False)
unit_net_price = Column(Numeric, nullable=False)
vat_rate = Column(Numeric, nullable=False)
per_month = Column(Boolean, nullable=False)
# association table for the products-template many-to-many relationship
association_table = Table('association', Base.metadata,
Column('product_id', Integer, ForeignKey('product.id')),
Column('template_id', Integer, ForeignKey('template.id'))
)
# template implements one-to-one relationship with customer and many-to-many relationship with product
class Template(Base):
id = Column(Integer, primary_key=True)
customer_id = Column(Integer, ForeignKey("customer.id"))
customer = relationship("Customer", back_populates="template")
products = relationship("Product", secondary=association_table)
quantity = Column(Numeric)
net_val = Column(Numeric)
tax_val = Column(Numeric)
gross_val = Column(Numeric)
# TODO: implement constructor
def __init__(self, **kwargs):
self.quantity = Decimal(0.0)
self.net_val = Decimal(0.0)
self.tax_val = Decimal(0.0)
self.gross_val = Decimal(0.0)
#event.listens_for(Template.quantity, "set")
def quantity_listener(target, value, oldvalue, initiator):
print(target)
print(initiator)
# target.net_val =
# target.tax_val =
# target.gross_val =
Now, I'm unsure how should I get a particular initiator and set its values, since products in Template table is a list (?)
I'd probably do it like this. It's hard to use Sqlalchemy orm in such cases as you can't access the Session object we all got used to use.
#event.listens_for(Template, "after_update")
def quantity_listener(mapper, connection, target):
field = 'quantity'
added, _, deleted = get_history(target, field)
# added is a new values of the specified field
# deleted is the old values
# so we basically check if 'quantity' has changed
# its a tuple btw
if added and deleted and added[0] != deleted[0]:
stmt = Template.update(). \
values(Template.something=Template.something * Template.other_something). \
where(Template.id == target.id)
conn.execute(stmt)
# here goes another updates for products or you can have another
# listener for them
There might be a better way to accomplish this. I can't debug this right now and I can't manage to get your examples working. Hope it helps.
I have two databases (src and dst) and each database uses the same SQLAlchemy model. I would like to query src and insert the results of that query into dst.
When I attempt to insert the values from src into dst, my code throws a InvalidRequestError exception because the queried objects are attached to the src session.
#!/usr/bin/env python
from sqlalchemy import create_engine, Column, Integer, String
from sqlalchemy.exc import InvalidRequestError
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
Base = declarative_base()
class User(Base):
__tablename__ = 'users'
uid = Column(Integer, primary_key=True)
username = Column(String, unique=True, nullable=False)
gid = Column(Integer, nullable=False)
password = Column(String)
gecos = Column(String)
home_directory = Column(String)
shell = Column(String)
def __init__(self, passwd_string):
passwd_fields = passwd_string.split(':')
self.username = passwd_fields[0]
self.password = passwd_fields[1]
self.uid = passwd_fields[2]
self.gid = passwd_fields[3]
self.gecos = passwd_fields[4]
self.home_directory = passwd_fields[5]
self.shell = passwd_fields[6]
def get_session(db_uri):
engine = create_engine(db_uri)
Base.metadata.create_all(engine)
return sessionmaker(bind=engine)()
def add_users(session, user_list):
session.add_all(user_list)
session.commit()
def main():
users = [
User('root:x:0:0:Root User:/:/bin/bash'),
User('genuser:x:100:100:Generic User:/home/user:/bin/bash')
]
# insert users to src database
src = get_session('sqlite:///src.sqlite')
add_users(src, users)
# query src database for users
src_users = src.query(User).all()
# insert users found in src database into dst database
try:
dst = get_session('sqlite:///dst.sqlite')
add_users(dst, src_users)
except InvalidRequestError as error:
print "InvalidRequestError: {}".format(error)
if __name__ == "__main__":
main()
Running above code generates the following exception/error:
InvalidRequestError: Object '<User at 0x10bc10250>' is already attached to session '1' (this is '2')
In response to #Deepak's answer, I implemented a copy() method in User:
def copy(self):
return User(
'{}:{}:{}:{}:{}:{}:{}'.format(
self.username,
self.password,
self.uid,
self.gid,
self.gecos,
self.home_directory,
self.shell
)
)
Which means I can now do:
try:
dst = get_session('sqlite:///dst.sqlite')
dst.add(user.copy()) for user in src_users
except InvalidRequestError as error:
print "InvalidRequestError: {}".format(error)
You are trying to query via Session of Database 1 and inserting the attached user object of this session to another session.
A better way would be inserting the values in 2nd table in the same way you are inserting in 1st table.
So get the values from table 1:
src_users = src.query(User).all()
duplicate_records = []
for row in src.query(User).all():
duplicate_user.append(User(row.username +":"+ row.password +":"+str(row.uid)+":"+str(row.gid)+":"+row.gecos+":"+row.home_directory+":"+row.shell))
And then insert this list in your second table.
Hello I am new with SQLalchemy and have some problems with inserting data in a column.
import sqlalchemy
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, Integer, String, ForeignKey, Boolean
from sqlalchemy.orm import relationship, backref
from sqlalchemy.orm import sessionmaker
engine = create_engine('sqlite:///test.db')
Base = declarative_base()
class Users(Base):
__tablename__ = 'users'
id = Column(Integer, primary_key=True)
name = Column(String, unique=True)
password = Column(String)
email = Column(String)
def __init__(self, name, password, email):
self._name = name
self._password = password
self._email = email
Base.metadata.create_all(engine)
Session = sessionmaker(bind=engine)
Session.configure(bind=engine)
session = Session()
dm_user = Users("Dungeonmaster", "123", "email")
print dm_user.name
print dm_user.password
print dm_user.email
session.add(dm_user)
our_user = session.query(Users).filter_by(name='Dungeonmaster').first()
session.commit()
I used sqlite studio to see if the data is added, and I see that a new row is being made (with a new id.) but the data name, password and email is not inserted.
With the print I see even before I try to add something goes wrong, but I don't know what. please help me out a bit
( dm_user = Users("Dungeonmaster", "123", "email") tried with single quotes as well (helped me before using sqlalchemy, but no difference here)
Your init method does not initialize the persistent columns. Try instead (remove underscores):
def __init__(self, name, password, email):
self.name = name
self.password = password
self.email = email
I have a Test model/table and a TestAuditLog model/table, using SQLAlchemy and SQL Server 2008. The relationship between the two is Test.id == TestAuditLog.entityId, with one test having many audit logs. TestAuditLog is intended to keep a history of changes to rows in the Test table. I want to track when a Test is deleted, also, but I'm having trouble with this. In SQL Server Management Studio, I set the FK_TEST_AUDIT_LOG_TEST relationship's "Enforce Foreign Key Constraint" property to "No", thinking that would allow a TestAuditLog row to exist with an entityId that no longer connects to any Test.id because the Test has been deleted. However, when I try to create a TestAuditLog with SQLAlchemy and then delete the Test, I get an error:
(IntegrityError) ('23000', "[23000] [Microsoft][ODBC SQL Server Driver][SQL Server]Cannot insert the value NULL into column 'AL_TEST_ID', table 'TEST_AUDIT_LOG'; column does not allow nulls. UPDATE fails. (515) (SQLExecDirectW); [01000] [Microsoft][ODBC SQL Server Driver][SQL Server]The statement has been terminated. (3621)") u'UPDATE [TEST_AUDIT_LOG] SET [AL_TEST_ID]=? WHERE [TEST_AUDIT_LOG].[AL_ID] = ?' (None, 8)
I think because of the foreign-key relationship between Test and TestAuditLog, after I delete the Test row, SQLAlchemy is trying to update all that test's audit logs to have a NULL entityId. I don't want it to do this; I want SQLAlchemy to leave the audit logs alone. How can I tell SQLAlchemy to allow audit logs to exist whose entityId does not connect with any Test.id?
I tried just removing the ForeignKey from my tables, but I'd like to still be able to say myTest.audits and get all of a test's audit logs, and SQLAlchemy complained about not knowing how to join Test and TestAuditLog. When I then specified a primaryjoin on the relationship, it grumbled about not having a ForeignKey or ForeignKeyConstraint with the columns.
Here are my models:
class TestAuditLog(Base, Common):
__tablename__ = u'TEST_AUDIT_LOG'
entityId = Column(u'AL_TEST_ID', INTEGER(), ForeignKey(u'TEST.TS_TEST_ID'),
nullable=False)
...
class Test(Base, Common):
__tablename__ = u'TEST'
id = Column(u'TS_TEST_ID', INTEGER(), primary_key=True, nullable=False)
audits = relationship(TestAuditLog, backref="test")
...
And here's how I'm trying to delete a test while keeping its audit logs, their entityId intact:
test = Session.query(Test).first()
Session.begin()
try:
Session.add(TestAuditLog(entityId=test.id))
Session.flush()
Session.delete(test)
Session.commit()
except:
Session.rollback()
raise
You can solve this by:
POINT-1: not having a ForeignKey neither on the RDBMS level nor on the SA level
POINT-2: explicitly specify join conditions for the relationship
POINT-3: mark relationship cascades to rely on passive_deletes flag
Fully working code snippet below should give you an idea (points are highlighted in the code):
from sqlalchemy import create_engine, Column, Integer, String, ForeignKey
from sqlalchemy.orm import scoped_session, sessionmaker, relationship
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
engine = create_engine('sqlite:///:memory:', echo=False)
Session = sessionmaker(bind=engine)
class TestAuditLog(Base):
__tablename__ = 'TEST_AUDIT_LOG'
id = Column(Integer, primary_key=True)
comment = Column(String)
entityId = Column('TEST_AUDIT_LOG', Integer, nullable=False,
# POINT-1
#ForeignKey('TEST.TS_TEST_ID', ondelete="CASCADE"),
)
def __init__(self, comment):
self.comment = comment
def __repr__(self):
return "<TestAuditLog(id=%s entityId=%s, comment=%s)>" % (self.id, self.entityId, self.comment)
class Test(Base):
__tablename__ = 'TEST'
id = Column('TS_TEST_ID', Integer, primary_key=True)
name = Column(String)
audits = relationship(TestAuditLog, backref='test',
# POINT-2
primaryjoin="Test.id==TestAuditLog.entityId",
foreign_keys=[TestAuditLog.__table__.c.TEST_AUDIT_LOG],
# POINT-3
passive_deletes='all',
)
def __init__(self, name):
self.name = name
def __repr__(self):
return "<Test(id=%s, name=%s)>" % (self.id, self.name)
Base.metadata.create_all(engine)
###################
## tests
session = Session()
# create test data
tests = [Test("test-" + str(i)) for i in range(3)]
_cnt = 0
for _t in tests:
for __ in range(2):
_t.audits.append(TestAuditLog("comment-" + str(_cnt)))
_cnt += 1
session.add_all(tests)
session.commit()
session.expunge_all()
print '-'*80
# check test data, delete one Test
t1 = session.query(Test).get(1)
print "t: ", t1
print "t.a: ", t1.audits
session.delete(t1)
session.commit()
session.expunge_all()
print '-'*80
# check that audits are still in the DB for deleted Test
t1 = session.query(Test).get(1)
assert t1 is None
_q = session.query(TestAuditLog).filter(TestAuditLog.entityId == 1)
_r = _q.all()
assert len(_r) == 2
for _a in _r:
print _a
Another option would be to duplicate the column used in the FK, and make the FK column nullable with ON CASCADE SET NULL option. In this way you can still check the audit trail of deleted objects using this column.