We Keep Coding

How do I change the schema for both a table and a foreign key?

I have the following simplified database access layer and two tables:
class DataAccessLayer():
def __init__(self):
conn_string = "mysql+mysqlconnector://root:root#localhost/"
self.engine = create_engine(conn_string)
Base.metadata.create_all(self.engine)
Session = sessionmaker()
Session.configure(bind=self.engine)
self.session = Session()
class MatchesATP(Base):
__tablename__ = "matches_atp"
__table_args__ = {"schema": "belgarath", "extend_existing": True}
ID_M = Column(Integer, primary_key=True)
ID_T_M = Column(Integer, ForeignKey("oncourt.tours_atp.ID_T"))
class TournamentsATP(Base):
__tablename__ = "tours_atp"
__table_args__ = {"schema": "oncourt", "extend_existing": True}
ID_T = Column(Integer, primary_key=True)
NAME_T = Column(String(255))
I want to be able to switch the schema names for the two tables to test databases as follows:
belgarath to belgarath_test
oncourt to oncourt_test
I've tried adding:
self.session.connection(execution_options={"schema_translate_map": {"belgarath": belgarath, "oncourt": oncourt}})
To the bottom of DataAccessLayer and then initialising the class with two variables as follows:
def __init__(self, belgarath, oncourt):
However, when I build the following query:
dal = DataAccessLayer("belgarath_test", "oncourt_test")
query = dal.session.query(MatchesATP)
print(query)
I get the following SQL:
SELECT belgarath.matches_atp.`ID_M` AS `belgarath_matches_atp_ID_M`, belgarath.matches_atp.`ID_T_M` AS `belgarath_matches_atp_ID_T_M`
FROM belgarath.matches_atp
This is still referencing the belgarath table.
I also can't figure out a way of changing the schema of the foreign key of oncourt.tours_atp.ID_T at the same time as the tables.
Are there individual solutions or a combined solution to my issues?

You might wanna decorate your subclassed Base declarative model with the #declared_attr decorator.
Try this--
In a base class for your models, say __init__.py...
from sqlalchemy.ext.declarative import declarative_base, declared_attr
SCHEMA_MAIN = 'belgarath' # figure out how you want to retrieve this
SCHEMA_TEST = 'belgarath_test'
class _Base(object):
#declared_attr
def __table_args__(cls):
return {'schema': SCHEMA_MAIN}
...
Base = declarative_base(cls=_Base)
Base.metadata.schema = SCHEMA_MAIN
Now that you have a Base that subclasses _Base with the main schema already defined, all your other models will subclass Base and do the following:
from . import Base, declared_attr, SCHEMA_TEST
class TestModel(Base):
#declared_attr
def __table_args__(cls):
return {'schema': SCHEMA_TEST}
Changing a schema for a foreign key could look like this:
class TournamentsATP(Base):
__tablename__ = "tours_atp"
__table_args__ = {"schema": "oncourt", "extend_existing": True}
ID_T = Column(Integer, primary_key=True)
NAME_T = Column(String(255))
match_id = Column('match_id', Integer, ForeignKey(f'{__table_args__.get("schema")}.matches_atp.id'))
Where match_id is a foreign key to matches_atp.id by using the __table_args[schema] element defined at the class level via #declared_attr.

It only took me 18 months to figure this out. Turns out I needed to add the schema_translate_map to an engine and then create the session with this engine:
from sqlalchemy import create_engine
engine = create_engine(conn_str, echo=False)
schema_engine = engine.execution_options(schema_translate_map={<old_schema_name>: <new_schema_name>})
NewSession = sessionmaker(bind=schema_engine)
session = NewSession()
All ready to roll...

Assuming your goal is to:
have dev/test/prod schemas on a single mysql host
allow your ORM classes to be flexible enough to be used in three different environments without modification
Then John has you most of the way to one type of solution. You could use #declared_attr to dynamically generate __table_args__ as he has suggested.
You could also consider using something like flask-sqlalchemy that comes with a built-in solution for this:
import os
DB_ENV = os.getenv(DB_ENV)
SQLALCHEMY_BINDS = {
'belgarath': 'mysql+mysqlconnector://root:root#localhost/belgarath{}'.format(DB_ENV),
'oncourt': 'mysql+mysqlconnector://root:root#localhost/oncourt{}'.format(DB_ENV)
}
class MatchesATP(Base):
__bind_key__ = "belgarath"
ID_M = Column(Integer, primary_key=True)
ID_T_M = Column(Integer, ForeignKey("oncourt.tours_atp.ID_T"))
class TournamentsATP(Base):
__bind_key__ = "oncourt"
ID_T = Column(Integer, primary_key=True)
NAME_T = Column(String(255))
Basically this method allows you to create a link to a schema (a bind key), and that schema is defined at run-time via the connection string. More information at the flask-sqlalchemy link.

sqlalchemy orm skip already-existing tables on create

How can I avoid this exception in sqlalchemy orm when I try to create a table which already exists in a database:
sqlalchemy.exc.InvalidRequestError: Table 'col1' is already defined for this MetaData instance. Specify 'extend_existing=True' to redefine options and columns on an existing Table object.
Base = automap_base()
Base.prepare(engine, reflect=True)
class Col1(Base):
__tablename__ = 'col1'
id = Column(Integer(), primary_key=True)
name = Column(String())
Base.metadata.create_all(engine)

I needed to add {'useexisting': True} (in SQLAlchemy 1.4+, use {'extend_existing': True}).
class Col1(Base):
__tablename__ = 'col1'
__table_args__ = {'useexisting': True}
id = Column(Integer(), primary_key=True)
name = Column(String())

SqlAlchemy issues with foreign keys

I am getting the error
Could not parse rfc1738 URL from string 'MACHINE_IE'
When I attempt to import the following
class MACHINE(declarative_base()):
__tablename__ = 'MACHINE'
MACHINE_UID = Column(Integer, primary_key=True)
MACHINE_IP = Column(String)
MACHINE_NAME = Column(String)
MACHINE_INFO = Column(String)
class IE(declarative_base()):
__tablename__ = 'IE'
IE_UID = Column(Integer, primary_key=True)
IE_VERSION = Column(String)
IE_MAJOR = Column(String)
class MACHINE_IE(declarative_base().metadata):
__tablename__ = 'MACHINE_IE'
IE_UID = Column(Integer, ForeignKey("IE.IE_UID"), primary_key=True)
MACHINE_UID = Column(Integer, ForeignKey("MACHINE.MACHINE_UID"))
EFFECTIVE_DATE = Column(DateTime)
If I remove
metadata
then I can import the module and perform a query on the "MACHINE" and "IE" tables and display the data from the rows. However when I query "MACHINE_IE" and then try to display some data from the query I get the following
sqlalchemy.exc.NoReferencedTableError: Foreign key associated with column 'MACHINE_IE.IE_UID' could not find table 'IE' with which to generate a foreign key to target column 'IE_UID'
I am guessing I need "metadata" in there but the error seems to be complaining about the db connection string when I use it. However I am able to connect to the db w/o issue if "metadata" is removed.
Here is the connection data
connect_string = "oracle+cx_oracle://{0}:{1}#{2}:{3}/{4}".format(user, pw, server, port, sid)
Any assistance is appreciated.

You are using multiple instances of Base. You should be doing:
Base = declarative_base()
class MACHINE(Base):
...
class IE(Base):
...
...

Overriding SQLAlchemy Columns for Testing

I'm trying to override a SQLAlchemy column to have different attributes in my test DB. Basically, I have a composite primary key in all of my MySQL tables: an autoncrement BIGINT id and a DATETIME create_time. For unit testing, I want to use SQLite, but my research has led me to believe that autoincrement is not possible for this use case. Since I don't use my create_time column for anything other than creating a rolling partition in MySQL, I want to override the two columns in my unit tests so they are compatible with SQLite. Note: I am using the Base = declarative_base() route, and all my models inherit Base and PrimaryKeyMixin.
In my models.py, I have created a mixin PrimaryKeyMixin that all tables inherit. All tables also inherit TABLE_ARGS for the __table_args__ component.
TABLE_ARGS = {
'mysql_engine': 'InnoDB',
'mysql_charset': 'utf8'
}
class PrimaryKeyMixin:
id = Column(BIGINT(unsigned=True), primary_key=True, autoincrement=True)
create_time = Column(DATETIME, primary_key=True, default=func.now())
In my unit tests, I do the following:
from myapp.models import TABLE_ARGS, PrimaryKeyMixin, Column, INT, DATETIME, func
TABLE_ARGS['sqlite_autoincrement'] = True
PrimaryKeyMixin.id = Column(INT(unsigned=True), primary_key=True, autoincrement=True)
PrimaryKeyMixin.create_date = Column(DATETIME, default=func.now())
TEST_DB = '/tmp/myapp_test.db'
from myapp import db
from myapp.models import Base
db.ENGINE = create_engine(
'sqlite:///%s' % TEST_DB,
encoding='utf-8',
)
Base.metadata.drop_all(db.ENGINE)
Base.metadata.create_all(db.ENGINE)
However, when I add echo=True to the ENGINE, it still is creating a PRIMARY_KEY (id, create_time) for each table.

Did it like this:
Base.metadata = MetaData()
def copy_table(model):
Table(model.__tablename__, Base.metadata, *change_columns(model.__table__.columns), sqlite_autoincrement=True)
def change_columns(columns):
new_cols = [column.copy() for column in columns if column.name != 'id' and column.name !='create_time']
new_cols.append(Column('id', INT(unsigned=True), primary_key=True, autoincrement=True))
new_cols.append(Column('create_time', DATETIME, default=func.now(), autoincrement=True))
return new_cols
for model in PrimaryKeyMixin.__subclasses__():
copy_table(model)

How to create relationships from existing relationships in SQLAlchemy?

I'm getting a bit stuck with relationships within relationships in sqlalchemy. I have a model like this:
engine = create_engine('sqlite:///tvdb.db', echo=True)
Base = declarative_base(bind=engine)
episodes_writers = Table('episodes_writers_assoc', Base.metadata,
Column('episode_id', Integer, ForeignKey('episodes.id')),
Column('writer_id', Integer, ForeignKey('writers.id')),
Column('series_id', Integer, ForeignKey('episodes.series_id'))
class Show(Base):
__tablename__ = 'shows'
id = Column(Integer, primary_key = True)
episodes = relationship('Episode', backref = 'shows')
class Episode(Base):
__tablename__ = 'episodes'
id = Column(Integer, primary_key = True)
series_id = Column(Integer, ForeignKey('shows.id'))
writers = relationship('Writer', secondary = 'episodes_writers',
backref = 'episodes')
class Writer(Base):
__tablename__ = 'writers'
id = Column(Integer, primary_key = True)
name = Column(Unicode)
So there is one-to-many between shows and episodes, and many-to-many between episodes and writers, but I now want to create a many-to-many relationship between shows and writers, based on the fact that a writer is associated with an episode of a show. I tried to add another column to the assoc table but that results in the following exception:
sqlalchemy.exc.ArgumentError: Could not determine join condition between parent/child tables on relationship Episode.writers. Specify a 'primaryjoin' expression. If 'secondary' is present, 'secondaryjoin' is needed as well.
So I'm obviously doing something wrong with my relationship declarations. How do I create the right primaryjoins to achieve what I'm trying to do here?

The fix is suggested right in the error message. Anyway, you have no need for a column that denormalises your schema; writer.shows can be an associationproxy.