I'm trying to override a SQLAlchemy column to have different attributes in my test DB. Basically, I have a composite primary key in all of my MySQL tables: an autoncrement BIGINT id and a DATETIME create_time. For unit testing, I want to use SQLite, but my research has led me to believe that autoincrement is not possible for this use case. Since I don't use my create_time column for anything other than creating a rolling partition in MySQL, I want to override the two columns in my unit tests so they are compatible with SQLite. Note: I am using the Base = declarative_base() route, and all my models inherit Base and PrimaryKeyMixin.
In my models.py, I have created a mixin PrimaryKeyMixin that all tables inherit. All tables also inherit TABLE_ARGS for the __table_args__ component.
TABLE_ARGS = {
'mysql_engine': 'InnoDB',
'mysql_charset': 'utf8'
}
class PrimaryKeyMixin:
id = Column(BIGINT(unsigned=True), primary_key=True, autoincrement=True)
create_time = Column(DATETIME, primary_key=True, default=func.now())
In my unit tests, I do the following:
from myapp.models import TABLE_ARGS, PrimaryKeyMixin, Column, INT, DATETIME, func
TABLE_ARGS['sqlite_autoincrement'] = True
PrimaryKeyMixin.id = Column(INT(unsigned=True), primary_key=True, autoincrement=True)
PrimaryKeyMixin.create_date = Column(DATETIME, default=func.now())
TEST_DB = '/tmp/myapp_test.db'
from myapp import db
from myapp.models import Base
db.ENGINE = create_engine(
'sqlite:///%s' % TEST_DB,
encoding='utf-8',
)
Base.metadata.drop_all(db.ENGINE)
Base.metadata.create_all(db.ENGINE)
However, when I add echo=True to the ENGINE, it still is creating a PRIMARY_KEY (id, create_time) for each table.
Did it like this:
Base.metadata = MetaData()
def copy_table(model):
Table(model.__tablename__, Base.metadata, *change_columns(model.__table__.columns), sqlite_autoincrement=True)
def change_columns(columns):
new_cols = [column.copy() for column in columns if column.name != 'id' and column.name !='create_time']
new_cols.append(Column('id', INT(unsigned=True), primary_key=True, autoincrement=True))
new_cols.append(Column('create_time', DATETIME, default=func.now(), autoincrement=True))
return new_cols
for model in PrimaryKeyMixin.__subclasses__():
copy_table(model)
Related
I have the following simplified database access layer and two tables:
class DataAccessLayer():
def __init__(self):
conn_string = "mysql+mysqlconnector://root:root#localhost/"
self.engine = create_engine(conn_string)
Base.metadata.create_all(self.engine)
Session = sessionmaker()
Session.configure(bind=self.engine)
self.session = Session()
class MatchesATP(Base):
__tablename__ = "matches_atp"
__table_args__ = {"schema": "belgarath", "extend_existing": True}
ID_M = Column(Integer, primary_key=True)
ID_T_M = Column(Integer, ForeignKey("oncourt.tours_atp.ID_T"))
class TournamentsATP(Base):
__tablename__ = "tours_atp"
__table_args__ = {"schema": "oncourt", "extend_existing": True}
ID_T = Column(Integer, primary_key=True)
NAME_T = Column(String(255))
I want to be able to switch the schema names for the two tables to test databases as follows:
belgarath to belgarath_test
oncourt to oncourt_test
I've tried adding:
self.session.connection(execution_options={"schema_translate_map": {"belgarath": belgarath, "oncourt": oncourt}})
To the bottom of DataAccessLayer and then initialising the class with two variables as follows:
def __init__(self, belgarath, oncourt):
However, when I build the following query:
dal = DataAccessLayer("belgarath_test", "oncourt_test")
query = dal.session.query(MatchesATP)
print(query)
I get the following SQL:
SELECT belgarath.matches_atp.`ID_M` AS `belgarath_matches_atp_ID_M`, belgarath.matches_atp.`ID_T_M` AS `belgarath_matches_atp_ID_T_M`
FROM belgarath.matches_atp
This is still referencing the belgarath table.
I also can't figure out a way of changing the schema of the foreign key of oncourt.tours_atp.ID_T at the same time as the tables.
Are there individual solutions or a combined solution to my issues?
You might wanna decorate your subclassed Base declarative model with the #declared_attr decorator.
Try this--
In a base class for your models, say __init__.py...
from sqlalchemy.ext.declarative import declarative_base, declared_attr
SCHEMA_MAIN = 'belgarath' # figure out how you want to retrieve this
SCHEMA_TEST = 'belgarath_test'
class _Base(object):
#declared_attr
def __table_args__(cls):
return {'schema': SCHEMA_MAIN}
...
Base = declarative_base(cls=_Base)
Base.metadata.schema = SCHEMA_MAIN
Now that you have a Base that subclasses _Base with the main schema already defined, all your other models will subclass Base and do the following:
from . import Base, declared_attr, SCHEMA_TEST
class TestModel(Base):
#declared_attr
def __table_args__(cls):
return {'schema': SCHEMA_TEST}
Changing a schema for a foreign key could look like this:
class TournamentsATP(Base):
__tablename__ = "tours_atp"
__table_args__ = {"schema": "oncourt", "extend_existing": True}
ID_T = Column(Integer, primary_key=True)
NAME_T = Column(String(255))
match_id = Column('match_id', Integer, ForeignKey(f'{__table_args__.get("schema")}.matches_atp.id'))
Where match_id is a foreign key to matches_atp.id by using the __table_args[schema] element defined at the class level via #declared_attr.
It only took me 18 months to figure this out. Turns out I needed to add the schema_translate_map to an engine and then create the session with this engine:
from sqlalchemy import create_engine
engine = create_engine(conn_str, echo=False)
schema_engine = engine.execution_options(schema_translate_map={<old_schema_name>: <new_schema_name>})
NewSession = sessionmaker(bind=schema_engine)
session = NewSession()
All ready to roll...
Assuming your goal is to:
have dev/test/prod schemas on a single mysql host
allow your ORM classes to be flexible enough to be used in three different environments without modification
Then John has you most of the way to one type of solution. You could use #declared_attr to dynamically generate __table_args__ as he has suggested.
You could also consider using something like flask-sqlalchemy that comes with a built-in solution for this:
import os
DB_ENV = os.getenv(DB_ENV)
SQLALCHEMY_BINDS = {
'belgarath': 'mysql+mysqlconnector://root:root#localhost/belgarath{}'.format(DB_ENV),
'oncourt': 'mysql+mysqlconnector://root:root#localhost/oncourt{}'.format(DB_ENV)
}
class MatchesATP(Base):
__bind_key__ = "belgarath"
ID_M = Column(Integer, primary_key=True)
ID_T_M = Column(Integer, ForeignKey("oncourt.tours_atp.ID_T"))
class TournamentsATP(Base):
__bind_key__ = "oncourt"
ID_T = Column(Integer, primary_key=True)
NAME_T = Column(String(255))
Basically this method allows you to create a link to a schema (a bind key), and that schema is defined at run-time via the connection string. More information at the flask-sqlalchemy link.
I am using Eve-SQLAlchemy==0.5.0
I would like to perform a nested query using Postman on my users such that I find all users that are within a specified organization.
Using SQL I would write my query such that:
select * from app_user
left join user_organization on user_organization.user_id = app_user.id
left join organization on organization.id = user_organization.organization_id
where organization.id = 2
I have a user model, an organization model, and a relational model linking the two user_organization.
from sqlalchemy import Column, DateTime, func, String, Integer
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class BaseModel(Base):
id = Column(Integer, primary_key=True, autoincrement=True)
__abstract__ = True
_created = Column(DateTime, default=func.now())
_updated = Column(DateTime, default=func.now(), onupdate=func.now())
_etag = Column(String(40))
class User(BaseModel):
__tablename__ = 'app_user'
organizations = relationship("Organization", secondary=UserOrganization.__tablename__)
class Organization(BaseModel):
__tablename__ = 'organization'
name = Column(String)
class UserOrganization(BaseModel):
__tablename__ = 'user_organization'
user_id = Column(Integer,
ForeignKey('app_user.id', ondelete='CASCADE'))
organization_id = Column(Integer,
ForeignKey('organization.id', ondelete='CASCADE'))
In my settings.py I have the resources registered:
# Resource Registration
DOMAIN = DomainConfig({
'organization': ResourceConfig(Organization),
'user': ResourceConfig(User)
}).render()
I have a series of postman collections setup, and using a GET request I can easily query any attribute... GET localhost:5000/user?where={"id":1}
I have tried (amongst many other things):
GET user?where={"organizations": {"organization_id" :2 }}
GET user?where={"organizations": 2}
It seems it's not possible at the moment due to a bug. I will try to fix it within the next week.
The code in https://github.com/pyeve/eve-sqlalchemy/blob/master/eve_sqlalchemy/parser.py#L73 is causing a GET ?where={"organizations": 2} to result in a SQL expression like user_id = 42 AND organization_id = 42 is generated. Which rarely makes any sense.
How can I avoid this exception in sqlalchemy orm when I try to create a table which already exists in a database:
sqlalchemy.exc.InvalidRequestError: Table 'col1' is already defined for this MetaData instance. Specify 'extend_existing=True' to redefine options and columns on an existing Table object.
Base = automap_base()
Base.prepare(engine, reflect=True)
class Col1(Base):
__tablename__ = 'col1'
id = Column(Integer(), primary_key=True)
name = Column(String())
Base.metadata.create_all(engine)
I needed to add {'useexisting': True} (in SQLAlchemy 1.4+, use {'extend_existing': True}).
class Col1(Base):
__tablename__ = 'col1'
__table_args__ = {'useexisting': True}
id = Column(Integer(), primary_key=True)
name = Column(String())
I created the following ORM:
from sqlalchemy import Column, Integer, String, UniqueConstraint
from sqlalchemy.dialects.postgresql import JSONB
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class TableA(Base):
__tablename__ = 'table_a'
id = Column(Integer, primary_key=True, nullable=False)
identifier = Column(String(320))
internal_id = Column(Integer)
type = Column(String(32))
time = Column(DateTime(timezone=True))
success = Column(Boolean())
parameters = Column(JSONB())
class TableB(Base):
__tablename__ = 'table_b'
__table_args__ = (UniqueConstraint('generate_action',
'print_action',
name='my_action_key'),)
id = Column(Integer, primary_key=True, autoincrement=True, nullable=False)
generate_action = Column(Integer)
print_action = Column(Integer)
generate_action = Column(Integer)
coupon_code = Column(String(300))
number_of_rebought_items = Column(Integer)
seconds_between_rebuy = Column(Integer)
I'm trying to figure out how to convert the following raw SQL view to ORM syntax with sqlalchemy.
CREATE VIEW my_view AS
SELECT table_b.id as table_b_id,
tb.coupon_code as coupon_code,
tb.number_of_rebought_items as number_of_rebought_items,
ta.id as table_a_action_id,
ta.time as time,
ta.parameters as parameters,
FROM table_b tb
LEFT JOIN table_a ta on
ta.id = tb.generate_action;
Couldn't find any good examples out there of how to do it with ORM.
So far, my solution is to just run raw sql to create this view.
can anyone point me to the right direction, or give an example of how to create views with sqlalchemy orm?
Is it possible to create the views with metadata.create_all()
the library sqlalchemy-utils now includes functionality for creating views, and it associates the view with sqlalchemy's metadata so that it is possible to create the view using Base.metadata.create_all
example:
# installation: pip install sqlalchemy-utils
from sqlalchemy_utils import create_view
from sqlalchemy import select, func
# engine Base & Table declaration elided for brevity
stmt = select([
TableB.id.label('table_b_id'),
TableB.coupon_code,
TableB.number_of_rebought_items,
TableA.id.label('table_a_action_id'),
TableA.time,
TableA.parameters
]).select_from(TableB.__table__.outerjoin(TableA, TableB.generate_action == TableA.id))
# attaches the view to the metadata using the select statement
view = create_view('my_view', stmt, Base.metadata)
# provides an ORM interface to the view
class MyView(Base):
__table__ = view
# will create all tables & views defined with ``create_view``
Base.metadata.create_all()
# At this point running the following yields 0, as expected,
# indicating that the view has been constructed on the server
engine.execute(select([func.count('*')], from_obj=MyView)).scalar()
I have a problem with SqlAlchemy. When I define the schema in my Oracle database, foreign keys are not recognized in other tables. Tested without using layout and it worked. But I need to define the schema of this application because the user is not the owner of the tables.
I would like your help to solve this problem.
code:
Base = declarative_base()
SCHEMA = {'schema' : 'SIATDESV'}
class Pgdasd(Base):
__tablename__ = 'PGDASD'
__table_args__ = SCHEMA
PGDASD_00000_ID_DECLARACAO = Column(String(17), primary_key = True)
class Pgdasd_01000(Base):
__tablename__ = 'pgdasd_01000'
__table_args__ = SCHEMA
PGDASD_00000_ID_DECLARACAO = Column(String, ForeignKey('PGDASD.PGDASD_00000_ID_DECLARACAO'))
PGDASD_01000_NRPAGTO = Column(String, primary_key = True)
error:
*Foreign key associated with column 'pgdasd_01000.PGDASD_00000_ID_DECLARACAO' could not find table 'PGDASD' with which to generate a foreign key to target column 'PGDASD_00000_ID_DECLARACAO'*
thanks!
Step 1: check that table(s) are created in db. BTW How did you create these tables? Have you run metadata.create_all() method ?
Step 2:
Try to add the name of the schema to the ForeignKey defintion:
Base = declarative_base()
SCHEMA = {'schema' : 'SIATDESV'}
class Pgdasd(Base):
__tablename__ = 'PGDASD'
__table_args__ = SCHEMA
PGDASD_00000_ID_DECLARACAO = Column(String(17), primary_key = True)
class Pgdasd_01000(Base):
__tablename__ = 'pgdasd_01000'
__table_args__ = SCHEMA
PGDASD_00000_ID_DECLARACAO = Column(String, ForeignKey('SIATDESV.PGDASD.PGDASD_00000_ID_DECLARACAO'))
PGDASD_01000_NRPAGTO = Column(String, primary_key = True)
If this will help you may also make code to use schema value to avoid hardcoding the schema.
PGDASD_00000_ID_DECLARACAO = Column(String, ForeignKey(SCHEMA['schema']+'.PGDASD.PGDASD_00000_ID_DECLARACAO'))
PS: Anyway it is always useful to check SQLAlchemy log what SQL queries it generates.