How can I avoid this exception in sqlalchemy orm when I try to create a table which already exists in a database:
sqlalchemy.exc.InvalidRequestError: Table 'col1' is already defined for this MetaData instance. Specify 'extend_existing=True' to redefine options and columns on an existing Table object.
Base = automap_base()
Base.prepare(engine, reflect=True)
class Col1(Base):
__tablename__ = 'col1'
id = Column(Integer(), primary_key=True)
name = Column(String())
Base.metadata.create_all(engine)
I needed to add {'useexisting': True} (in SQLAlchemy 1.4+, use {'extend_existing': True}).
class Col1(Base):
__tablename__ = 'col1'
__table_args__ = {'useexisting': True}
id = Column(Integer(), primary_key=True)
name = Column(String())
Related
I am new to sqlalchemy. I can create database tables by declarative mapping like this:
engine = create_engine("--engine works---")
Base = declarative_base()
class Customer(Base):
__tablename__ = 'customer'
customer_id = Column(Integer, primary_key=True)
name = Column(String(30))
email = Column(String(30))
invoices = relationship(
'Invoice',
order_by="Invoice.inv_id",
back_populates='customer',
cascade="all, delete, delete-orphan"
)
class Invoice(Base):
__tablename__ = 'invoice'
inv_id = Column(Integer, primary_key=True)
name = Column(String(30))
created = Column(Date)
customer_id = Column(ForeignKey('customer.customer_id'))
customer = relationship('Customer', back_populates='invoices')
Base.metadata.create_all(engine)
This is fine. I added some data into both customer and invoice tables.
So far so good. Next, I would try out automap_base on this existing database like this:
from sqlalchemy import select, text
from sqlalchemy.orm import declarative_base, sessionmaker
from sqlalchemy.ext.automap import automap_base
engine = create_engine('--engine works---')
Base = automap_base()
# reflect
Base.prepare(engine, reflect=True)
Customer = Base.classes.customer
Invoice = Base.classes.invoice
Session = sessionmaker(bind=engine, future=True)
session = Session()
# query invoice
stmt = select(Customer, Invoice).join(Customer.invoices).order_by(Customer.customer_id, Invoice.inv_id)
res = session.execute(stmt)
for c in res:
print(c.customer_id)
When I ran the code, I got:
AttributeError: type object 'customer' has no attribute 'invoices'
What did I miss for the relationship on the Customer (one side) or Invoice (many side) in this case so that when I query for customers with its invoices attibute and for invoices with customer attribute? Thanks for any help.
By default, automap will create the relation in the parent by appending "_collection" the lower-cased classname, so the name will be Customer.invoice_collection.
While answering this, I found that the join would raise an AttributeError on Customer.invoice_collection unless I performed a query on Customer beforehand, for example
session.execute(sa.select(Customer).where(False))
I'm not sure why that happens, however you don't necessarily need the join as you can iterate over Customer.invoice_collection directly, or join against the invoice table:
stmt = sa.select(Customer, Invoice).join(Invoice)
res = session.execute(stmt)
for c, i in res:
print(c.customer_id, i.inv_id)
I have the following simplified database access layer and two tables:
class DataAccessLayer():
def __init__(self):
conn_string = "mysql+mysqlconnector://root:root#localhost/"
self.engine = create_engine(conn_string)
Base.metadata.create_all(self.engine)
Session = sessionmaker()
Session.configure(bind=self.engine)
self.session = Session()
class MatchesATP(Base):
__tablename__ = "matches_atp"
__table_args__ = {"schema": "belgarath", "extend_existing": True}
ID_M = Column(Integer, primary_key=True)
ID_T_M = Column(Integer, ForeignKey("oncourt.tours_atp.ID_T"))
class TournamentsATP(Base):
__tablename__ = "tours_atp"
__table_args__ = {"schema": "oncourt", "extend_existing": True}
ID_T = Column(Integer, primary_key=True)
NAME_T = Column(String(255))
I want to be able to switch the schema names for the two tables to test databases as follows:
belgarath to belgarath_test
oncourt to oncourt_test
I've tried adding:
self.session.connection(execution_options={"schema_translate_map": {"belgarath": belgarath, "oncourt": oncourt}})
To the bottom of DataAccessLayer and then initialising the class with two variables as follows:
def __init__(self, belgarath, oncourt):
However, when I build the following query:
dal = DataAccessLayer("belgarath_test", "oncourt_test")
query = dal.session.query(MatchesATP)
print(query)
I get the following SQL:
SELECT belgarath.matches_atp.`ID_M` AS `belgarath_matches_atp_ID_M`, belgarath.matches_atp.`ID_T_M` AS `belgarath_matches_atp_ID_T_M`
FROM belgarath.matches_atp
This is still referencing the belgarath table.
I also can't figure out a way of changing the schema of the foreign key of oncourt.tours_atp.ID_T at the same time as the tables.
Are there individual solutions or a combined solution to my issues?
You might wanna decorate your subclassed Base declarative model with the #declared_attr decorator.
Try this--
In a base class for your models, say __init__.py...
from sqlalchemy.ext.declarative import declarative_base, declared_attr
SCHEMA_MAIN = 'belgarath' # figure out how you want to retrieve this
SCHEMA_TEST = 'belgarath_test'
class _Base(object):
#declared_attr
def __table_args__(cls):
return {'schema': SCHEMA_MAIN}
...
Base = declarative_base(cls=_Base)
Base.metadata.schema = SCHEMA_MAIN
Now that you have a Base that subclasses _Base with the main schema already defined, all your other models will subclass Base and do the following:
from . import Base, declared_attr, SCHEMA_TEST
class TestModel(Base):
#declared_attr
def __table_args__(cls):
return {'schema': SCHEMA_TEST}
Changing a schema for a foreign key could look like this:
class TournamentsATP(Base):
__tablename__ = "tours_atp"
__table_args__ = {"schema": "oncourt", "extend_existing": True}
ID_T = Column(Integer, primary_key=True)
NAME_T = Column(String(255))
match_id = Column('match_id', Integer, ForeignKey(f'{__table_args__.get("schema")}.matches_atp.id'))
Where match_id is a foreign key to matches_atp.id by using the __table_args[schema] element defined at the class level via #declared_attr.
It only took me 18 months to figure this out. Turns out I needed to add the schema_translate_map to an engine and then create the session with this engine:
from sqlalchemy import create_engine
engine = create_engine(conn_str, echo=False)
schema_engine = engine.execution_options(schema_translate_map={<old_schema_name>: <new_schema_name>})
NewSession = sessionmaker(bind=schema_engine)
session = NewSession()
All ready to roll...
Assuming your goal is to:
have dev/test/prod schemas on a single mysql host
allow your ORM classes to be flexible enough to be used in three different environments without modification
Then John has you most of the way to one type of solution. You could use #declared_attr to dynamically generate __table_args__ as he has suggested.
You could also consider using something like flask-sqlalchemy that comes with a built-in solution for this:
import os
DB_ENV = os.getenv(DB_ENV)
SQLALCHEMY_BINDS = {
'belgarath': 'mysql+mysqlconnector://root:root#localhost/belgarath{}'.format(DB_ENV),
'oncourt': 'mysql+mysqlconnector://root:root#localhost/oncourt{}'.format(DB_ENV)
}
class MatchesATP(Base):
__bind_key__ = "belgarath"
ID_M = Column(Integer, primary_key=True)
ID_T_M = Column(Integer, ForeignKey("oncourt.tours_atp.ID_T"))
class TournamentsATP(Base):
__bind_key__ = "oncourt"
ID_T = Column(Integer, primary_key=True)
NAME_T = Column(String(255))
Basically this method allows you to create a link to a schema (a bind key), and that schema is defined at run-time via the connection string. More information at the flask-sqlalchemy link.
I am playing with a toy example to see the back populates in action but hitting an error that I can't understand. Below I have two 'models' that back populate each other. When I try to create a User object it throws an error. What am I missing?
"sqlalchemy.exc.NoForeignKeysError: Could not determine join condition between parent/child tables on relationship Child.user - there are no foreign keys linking these tables. Ensure that referencing columns are associated with a ForeignKey or ForeignKeyConstraint, or specify a 'primaryjoin' expression."
engine = create_engine('sqlite:///:memory:', echo=False)
Session = sessionmaker(bind=engine)
session = Session()
Base = declarative_base(bind=engine)
class User(Base):
__tablename__ = 'user'
user_id = Column("UserID", Integer, primary_key=True)
name = Column("Name", String(50))
age = Column("Age", SmallInteger)
child = relationship("Child", back_populates="user")
class Child(Base):
__tablename__ = 'child'
child_id = Column("ChildID", Integer, primary_key=True)
school = Column("School", String)
grade = Column("Grade", String)
user_id = Column("UserID", Integer, ForeignKey('User.UserID'), index=True, nullable=True)
user = relationship("User", back_populates="child")
ForeignKey requires the table and column name, not model and attribute name, so it should be:
user_id = Column("UserID", Integer, ForeignKey('user.UserID'), ...)
because your User model has a table name of user.
I have a problem with SqlAlchemy. When I define the schema in my Oracle database, foreign keys are not recognized in other tables. Tested without using layout and it worked. But I need to define the schema of this application because the user is not the owner of the tables.
I would like your help to solve this problem.
code:
Base = declarative_base()
SCHEMA = {'schema' : 'SIATDESV'}
class Pgdasd(Base):
__tablename__ = 'PGDASD'
__table_args__ = SCHEMA
PGDASD_00000_ID_DECLARACAO = Column(String(17), primary_key = True)
class Pgdasd_01000(Base):
__tablename__ = 'pgdasd_01000'
__table_args__ = SCHEMA
PGDASD_00000_ID_DECLARACAO = Column(String, ForeignKey('PGDASD.PGDASD_00000_ID_DECLARACAO'))
PGDASD_01000_NRPAGTO = Column(String, primary_key = True)
error:
*Foreign key associated with column 'pgdasd_01000.PGDASD_00000_ID_DECLARACAO' could not find table 'PGDASD' with which to generate a foreign key to target column 'PGDASD_00000_ID_DECLARACAO'*
thanks!
Step 1: check that table(s) are created in db. BTW How did you create these tables? Have you run metadata.create_all() method ?
Step 2:
Try to add the name of the schema to the ForeignKey defintion:
Base = declarative_base()
SCHEMA = {'schema' : 'SIATDESV'}
class Pgdasd(Base):
__tablename__ = 'PGDASD'
__table_args__ = SCHEMA
PGDASD_00000_ID_DECLARACAO = Column(String(17), primary_key = True)
class Pgdasd_01000(Base):
__tablename__ = 'pgdasd_01000'
__table_args__ = SCHEMA
PGDASD_00000_ID_DECLARACAO = Column(String, ForeignKey('SIATDESV.PGDASD.PGDASD_00000_ID_DECLARACAO'))
PGDASD_01000_NRPAGTO = Column(String, primary_key = True)
If this will help you may also make code to use schema value to avoid hardcoding the schema.
PGDASD_00000_ID_DECLARACAO = Column(String, ForeignKey(SCHEMA['schema']+'.PGDASD.PGDASD_00000_ID_DECLARACAO'))
PS: Anyway it is always useful to check SQLAlchemy log what SQL queries it generates.
SqlAlchemy newbie question:
Base = declarative_base()
class A(Base):
__tablename__ = 'as'
id = Column(Integer, primary_key=True)
class B(Base):
__tablename__ = 'bs'
id = Column(Integer, primary_key=True)
a = relation(A)
When I create my database schema, I have two tables, as and bs, which have one column (id) but there is no a column in table bs that points to A.
What can I be doing wrong? My database is mysql, if it matters.
relation() only tells the mapper how are the two tables related. You still need to add a column with the foreign key information. For example:
class B(Base):
__tablename__ = 'bs'
id = Column(Integer, primary_key=True)
a_id = Column(Integer, ForeignKey('as.id'), name="a")
a = relation(A)