We Keep Coding

How to map table relationships with flask-sqlalchemy, before tables exist?

I have a problem creating relationships between my tables in flask-sqlalchemy. I have a table with project overview, and from there on out I want to dynamically create new experiment tables with a relationship to my project overview. However, when I try to define the relationship, sqlalchemy throws the following error:
sqlalchemy.exc.InvalidRequestError: When initializing mapper mapped class Projects->projects, expression 'Experiment_Overview' failed to locate a name ('Experiment_Overview'). If this is a c
lass name, consider adding this relationship() to the <class 'app.Projects'> class after both dependent classes have been defined.
This seems to be the case because the class Experiment_Overview(db.Model) does not exist yet, which is correct since it will be dynamically generated later on through user input. How can I mitigate this error?
import os
from flask import Flask, render_template, redirect, request, url_for
from flask_sqlalchemy import SQLAlchemy
from datetime import datetime
app = Flask(__name__)
Bootstrap(app)
app.config["SQLALCHEMY_DATABASE_URI"] = "sqlite:///DATA/DB.db"
db = SQLAlchemy(app)
def TableCreator(tablename):
class Experiment_Overview(db.Model):
__tablename__ = tablename
creation_date = db.Column(db.DateTime, nullable=False, default=datetime.utcnow)
experiment_name = db.Column(db.String(30), unique=False, nullable=False, primary_key=True)
projectname = db.Column(db.String(150), db.ForeignKey('projects.projectname'), nullable=False, unique=True)
return MyTable
class Projects(db.Model):
projectname = db.Column(db.String(150), unique=True, nullable=False, primary_key=True)
creation_date = db.Column(db.DateTime, nullable=False, default=datetime.utcnow)
experiments = db.relationship('Experiment_Overview', backref="experiments", lazy=True, uselist=False)
def __init__(self, owner, projectname, status, created_at):
self.owner=owner
self.projectname=projectname
self.status=status
self.created_at=created_at
db.create_all()

Generally speaking, you don't dynamically create tables; you usually shouldn't create or drop tables while your program is running.
Additionally, I believe it's impossible to create a true relationship that links back to an entire table. Relationships/Foreign Keys are for linking between rows in tables.
Don't worry, there are easier ways to achieve the behavior that you are looking for here.
From your question it sounds like you can have multiple Projects, and each project can have multiple Experiments within the project.
This would make the relationship between a Project and its Experiments a One-To-Many relationship.
If this is the case, you would need one Projects table (which you have already in your code), and you would also have one Experiments table.
Each row in the Projects table represents one Project.
Each row in the Experiments table represents one Experiment. The Experiments table will have a column containing a foreign key linking back to the Project the Experiment is linked to.
I've modified your code according to the One-To-Many example code given in the SQLAlchemy documentation that I linked above.
Note the addition of the back_populates option to the relationship(), this allows bi-directional knowledge of the relationship: the Experiment know what project it belongs to, and the Project know what Experiments it has.
import os
from flask import Flask, render_template, redirect, request, url_for
from flask_sqlalchemy import SQLAlchemy
from datetime import datetime
app = Flask(__name__)
Bootstrap(app)
app.config["SQLALCHEMY_DATABASE_URI"] = "sqlite:///DATA/DB.db"
db = SQLAlchemy(app)
class Projects(db.Model):
__tablename__ = "projects"
projectname = db.Column(db.String(150), unique=True, nullable=False, primary_key=True)
creation_date = db.Column(db.DateTime, nullable=False, default=datetime.utcnow)
experiments = db.relationship("Experiment", back_populates="project")
def __init__(self, owner, projectname, status, created_at):
self.owner=owner
self.projectname=projectname
self.status=status
self.created_at=created_at
class Experiment(db.Model):
__tablename__ = "experiments"
creation_date = db.Column(db.DateTime, nullable=False, default=datetime.utcnow)
experiment_name = db.Column(db.String(30), unique=False, nullable=False, primary_key=True)
projectname = db.Column(db.String(150), db.ForeignKey('projects.projectname'), nullable=False)
project = relationship("Project", back_populates="experiments")
db.create_all()

Alembic recreates foreign keys every time I migrate causing duplicate foreign keys on my tables

I have a flask app using sql-alchemy and flask migrate to handle database changes. Every time I run a flask migrate to create a script for Alembic to update the database, the script contains commands to create foreign keys that already exist in my database.
The table definition in my models.py is
class Airline(db.Model):
__tablename__ = 'Airlines'
AirlineID = db.Column(db.Integer, primary_key=True)
AirlineShortCode = db.Column(db.String(3), index=True, unique=True, nullable=False)
FullName = db.Column(db.String(256), unique=False, nullable=True)
ShortName = db.Column(db.String(64), unique=False, nullable=True)
class CabinClass(db.Model):
__tablename__ = 'CabinClasses'
CabinClassID = db.Column(db.Integer, primary_key=True)
AirlineShortCode = db.Column(db.ForeignKey("Airlines.AirlineShortCode"), nullable=True)
CabinClassShortCode = db.Column(db.String(32), unique=False, nullable=False)
CabinClassName = db.Column(db.String(64), unique=False, nullable=True)
The line in the migration database update script that is generated to create the foreign key is
op.create_foreign_key(None, 'CabinClasses', 'Airlines', ['AirlineShortCode'], ['AirlineShortCode'])
This line is generated every time I create the migration script, resulting in multiple foreign key entries in the CabinClasses table:
I see that the name of each foreign key created is different and that the create_foreign_key command in the database migration script states the name as None. I believe this is correct if you are using an automated naming scheme, which I believe is what happens by default
For setups that use an automated naming scheme such as that described
at Configuring Constraint Naming Conventions, name here can be None,
as the event listener will apply the name to the constraint object
when it is associated with the table
https://alembic.sqlalchemy.org/en/latest/naming.html
Can anyone identify what would cause these foreign keys to be created every time I update the database?

The names of the constraints that you are getting look like they come from your database, not SQLAlchemy. You need to add the constraint naming templates for all types of constraints to the SQLAlchemy metadata, and then I think you will get consistent names. See how to do this in the Flask-SQLAlchemy documentation. I'm copying the code example from the docs below for your convenience:
from sqlalchemy import MetaData
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
convention = {
"ix": 'ix_%(column_0_label)s',
"uq": "uq_%(table_name)s_%(column_0_name)s",
"ck": "ck_%(table_name)s_%(constraint_name)s",
"fk": "fk_%(table_name)s_%(column_0_name)s_%(referred_table_name)s",
"pk": "pk_%(table_name)s"
}
metadata = MetaData(naming_convention=convention)
db = SQLAlchemy(app, metadata=metadata)

Python SQLAlchemy: Reflecting the database breaks default/onupdate methods?

I have two separate SQLAlchemy interfaces to a Postgres database. The first interface, in the context of a Flask App, contains this model:
app = create_app() # sets the SQLAlchemy Database URI, etc.
db = SQLAlchemy(app)
class User(db.Model):
id = db.Column(db.Integer, primary_key=True)
created_at = db.Column(db.DateTime, default=datetime.datetime.utcnow)
updated_at = db.Column(db.DateTime, onupdate=datetime.datetime.utcnow)
name = db.Column(db.String, nullable=False)
The second interface is not through Flask -- rather, it's a script that listens for a particular event, in which case it is meant to perform some computations and update a row in the database. To accomplish this, I have SQLAlchemy reflect the existent database:
from sqlalchemy import create_engine, MetaData, Table
from sqlalchemy.orm import mapper, sessionmaker
from sqlalchemy.ext.automap import automap_base
from os import environ
dbPath = "postgresql://" + ...
engine = create_engine(dbPath)
Base = automap_base()
Base.prepare(engine, reflect=True)
metadata = MetaData(engine)
class User(object):
pass
users = Table('user', metadata, autoload=True, autoload_with=engine)
mapper(User, users)
Session = sessionmaker(bind=engine)
session = Session()
The issue I'm now running into is this: when I'm using the first interface to create a new entry or update one, things work as expected, and the created_at and updated_at fields are updated appropriately.
However, when I'm using the second interface -- importing the code and using session.query(User) to get an entry and to update it, the updated_at field doesn't change. Moreover, when I'm using this interface to create a new User, while it creates the new row as expected, it populates neither the created_at nor updated_at fields.
My questions:
Why is this happening? Why does the reflection seemingly break the default/onupdate methods?
How can I fix this?

default and onupdate are handled entirely client side in Python and so cannot be reflected from the DB. See "Limitations of Reflection". In case of default you could use server_default:
class User(db.Model):
...
created_at = db.Column(db.DateTime,
server_default=text("now() at time zone 'UTC'"))
and for onupdate you'd have to write a DB trigger and use server_onupdate=FetchedValue().
On the other hand you could avoid all that and just separate your models from your application code to a module, used by both your Flask application and your script. This would of course be a bit more involved as you'd have to use vanilla SQLAlchemy declarative instead of the customized db.Model base of Flask-SQLAlchemy. Or, you could use custom commands with Flask to implement your scripts, which would allow using the Flask-SQLAlchemy extensions.

SQLAlchemy create_all without flask

I have a model being used by my flask server as well as a command-line tool. I want both the script and flask app to be able to create tables programmatically using this model.
In the flask app, I do this:
from models import db
db.app = app
db.init_app(app)
db.create_all()
However I can't use this code in the command-line tool as there is no 'app', and without the app I get
RuntimeError: application not registered on db instance and no application bound to current context
Which makes sense. So how do I make it work with the command-line tool without creating another version of my model file?
My model file (roughly):
class User(db.Model):
__tablename__ = "users"
id = Column(Integer, primary_key=True)
account_id = Column(BigInteger, nullable=False)
team_name = Column(String, nullable=False)
external_id = Column(String(36), nullable=False)
# should not be filled with web app, but for misc notes we take later
notes = Column(String)
def __str__(self):
return "User (account_id={}, team_name='{}', external_id='{}')".format(
self.account_id, self.team_name, self.external_id)
__table_args__ = (
UniqueConstraint("account_id", name="unique_account_id"),
UniqueConstraint("external_id", name="unique_external_id"),
UniqueConstraint("team_name", name="unique_team_name")
)

Migrate Flask models.py to MySQL

I'm new. Bear with me.
I'm developing a Flask application using SQLAlchemy as an ORM and up until today I have been using SQLite for convenience. I'm now putting the application live on Digital Ocean and want to use MySQL instead of SQLite.
I have MySQL installed on my ubuntu VPS on Digital Ocean and it seems like it is configured properly. However, obviously I have to create the database tables, so I can save the data inside.
Question: Is there a way for me to migrate my models.py, so the database tables are created from what I have written in models.py or do I have to create all the database tables myself manually in MySQL?
You can see the application live here: http://workflowforum.dk/ and I have made a small test to see if there is a database connection here: http://workflowforum.dk/testdb
Models.py (Only user model):
from flask import Flask
from flask.ext.sqlalchemy import SQLAlchemy
from flask.ext.script import Manager
from flask.ext.migrate import Migrate, MigrateCommand
from datetime import datetime, date
from hashlib import md5
from bcrypt import hashpw, gensalt
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'mysql://root:password#localhost/database'
db = SQLAlchemy(app)
migrate = Migrate(app, db)
manager = Manager(app)
manager.add_command('db', MigrateCommand)
class User(db.Model):
id = db.Column(db.Integer, primary_key=True)
slug = db.Column(db.String(80))
username = db.Column(db.String(80), unique=True)
email = db.Column(db.String(80))
password = db.Column(db.String(80), unique=False)
admin = db.Column(db.Boolean(), default=False)
join_date = db.Column(db.DateTime)
last_seen = db.Column(db.DateTime)
topics = db.relationship('Topic')
posts = db.relationship('Post')
picture = db.Column(db.Boolean(), default=False)
title = db.Column(db.String(80))
company = db.Column(db.String(80))
summary = db.Column(db.String(80))
class Category(db.Model):
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(80), unique=True)
description = db.Column(db.String(180), unique=False)
topics = db.relationship('Topic', backref="category")
class Topic(db.Model):
id = db.Column(db.Integer, primary_key=True)
slug = db.Column(db.String(255), unique=True)
title = db.Column(db.String(80), unique=False)
description = db.Column(db.Text, unique=False)
pub_date = db.Column(db.DateTime)
last_update = db.Column(db.DateTime)
user_id = db.Column(db.String(80), db.ForeignKey('user.id'))
category_id = db.Column(db.Integer, db.ForeignKey('category.id'))
views = db.Column(db.Integer, default=0)
locked = db.Column(db.Boolean(), default=False)
pinned = db.Column(db.Boolean(), default=False)
user = db.relationship('User')
posts = db.relationship('Post')
Views.py (Only database test):
#app.route('/testdb')
def testdb():
if db.session.query("1").from_statement("SELECT 1").all():
return 'It works.'
else:
return 'Something is broken.'
UPDATE after Lukas comment:
When trying to db.create_all() I get this traceback:
sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1005, "Can't create table 'pwforum.topic' (errno: 150)") [SQL: u'\nCREATE TABLE topic (\n\tid INTEGER NOT NULL AUTO_INCREMENT, \n\tslug VARCHAR(255), \n\ttitle VARCHAR(80), \n\tdescription TEXT, \n\tpub_date DATETIME, \n\tlast_update DATETIME, \n\tuser_id VARCHAR(80), \n\tcategory_id INTEGER, \n\tviews INTEGER, \n\tlocked BOOL, \n\tpinned BOOL, \n\tPRIMARY KEY (id), \n\tUNIQUE (slug), \n\tFOREIGN KEY(user_id) REFERENCES user (id), \n\tFOREIGN KEY(category_id) REFERENCES category (id), \n\tCHECK (locked IN (0, 1)), \n\tCHECK (pinned IN (0, 1))\n)\n\n']

The db.create_all suggestion in the comments is usually what you do when you don't use a migration framework. But it looks like you are using Flask-Migrate as a database migration framework here. The problem with doing create_all is that your migration scripts are skipped, so any fixes that you need to make to convert your sqlite models to MySQL will not be reflected in the migration scripts.
My recommendation is that you create an empty MySQL database, configure it in your Flask app, and then generate the MySQL tables simply by calling:
$ ./manage.py db upgrade
When you do this, Flask-Migrate will start running the migration scripts one by one and in order.
If you get failures, that it is probably because some of your migration scripts have changes that are compatible with sqlite but not with MySQL. You will need to fix all these problems in your migration scripts, until you get all of them to run cleanly, at which point you will have a complete MySQL database to use in your application.