We Keep Coding

Backup and restore a sqlalchemy db using sqlite as an engine

Hi!
I'm working on a flask-sqlalchemy application, and as you can imagine I'm changing database models and other thing through the process, every time that I made changes to the models I have to populate de DB again, and at the same time, as a safety measure I need to have some sort of backup restore process prepared in case of something goes wrong.
main.py
# You can image the other code...
app.config["SQLALCHEMY_DATABASE_URI"] = f"sqlite:///{DB_FILE}"
db.init_app(app)
db.py
from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy()
models.py
class User_device(db.Model):
__tablename__ = "user_devices"
id = db.Column(db.String(32), primary_key=True)
user_id = db.Column(db.String(32), db.ForeignKey('users.id', ondelete="CASCADE"))
ip = db.Column(db.String(32))
agent = db.Column(db.String(100))
I'll like to have a route where to do this process

I found a solution that works for me, if you have a better alternative don't hesitate to share it:
from the terminal:
back up your current db
sqlite3 instance/pre_backup.db .dump > back.sql
delete your db
rm instance/pre_backup.db
init your project in order to create the instances of your db
python3 main.py
or with the flask run etc.
4. Restore your db
sqlite3 instance/new.db < back.sql
In my case I've got one error msg for every table that I had before, but it didn't give me any trouble to populate the db.
PD:
I'm new in Stack Overflow and I don't see the problem with starting with Hi!
thanks to nothing #PChemGuy -> This is not a letter, so you do not start your question with Hi!!!!!!!!!

alembic autogenerates creates empty migration but generates table

I followed the documentation from Alembic to auto-generate migrations. My project structure looks like this:
alembic/
versions/
env.py
README
script.py.mako
data/
__init__.py
db.py
models.py
alembic.ini
app.db
I made changes to env.py by exactly following the document:
from logging.config import fileConfig
from sqlalchemy import engine_from_config
from sqlalchemy import pool
from alembic import context
from data.models import Base
# this is the Alembic Config object, which provides
# access to the values within the .ini file in use.
config = context.config
# Interpret the config file for Python logging.
# This line sets up loggers basically.
fileConfig(config.config_file_name)
# add your model's MetaData object here
# for 'autogenerate' support
# from myapp import mymodel
# target_metadata = mymodel.Base.metadata
target_metadata = Base.metadata
# other values from the config, defined by the needs of env.py,
# can be acquired:
# my_important_option = config.get_main_option("my_important_option")
# ... etc.
def run_migrations_offline():
"""Run migrations in 'offline' mode.
This configures the context with just a URL
and not an Engine, though an Engine is acceptable
here as well. By skipping the Engine creation
we don't even need a DBAPI to be available.
Calls to context.execute() here emit the given string to the
script output.
"""
url = config.get_main_option("sqlalchemy.url")
context.configure(
url=url,
target_metadata=target_metadata,
literal_binds=True,
dialect_opts={"paramstyle": "named"},
)
with context.begin_transaction():
context.run_migrations()
def run_migrations_online():
"""Run migrations in 'online' mode.
In this scenario we need to create an Engine
and associate a connection with the context.
"""
connectable = engine_from_config(
config.get_section(config.config_ini_section),
prefix="sqlalchemy.",
poolclass=pool.NullPool,
)
with connectable.connect() as connection:
context.configure(
connection=connection, target_metadata=target_metadata
)
with context.begin_transaction():
context.run_migrations()
if context.is_offline_mode():
run_migrations_offline()
else:
run_migrations_online()
This is my __init__.py:
import os
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from .models import Base
basedir = os.path.abspath(os.path.dirname(os.path.dirname(__file__)))
db_url = os.environ.get('DATABASE_URL') or \
'sqlite:///' + os.path.join(basedir, 'app.db')
engine = create_engine(db_url, echo=False)
Base.metadata.create_all(engine)
session = sessionmaker(bind=engine)
I created a User class in models.py like this:
from sqlalchemy import Column, Sequence, Integer, String
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class User(Base):
__tablename__ = 'user'
id = Column(Integer, Sequence('id_seq'), primary_key=True)
first_name = Column(String, nullable=False)
last_name = Column(String, nullable=False)
def __repr__(self):
return "<%s('%s', '%s')>" \
% (self.__class__.__qualname__, self.first_name,
self.last_name)
After that, I run migration by:
alembic revision --autogenerate -m "Added user table"
However, I got empty upgrade() and downgrade() in migration file:
"""Added user table
Revision ID: 279933caec54
Revises:
Create Date: 2021-05-12 16:21:05.772568
"""
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = '279933caec54'
down_revision = None
branch_labels = None
depends_on = None
def upgrade():
# ### commands auto generated by Alembic - please adjust! ###
pass
# ### end Alembic commands ###
def downgrade():
# ### commands auto generated by Alembic - please adjust! ###
pass
# ### end Alembic commands ###
But when I checked the database, the user table was created there (I didn't do it myself). How did it happen? How did alembic create empty migration but generating the table?

I found the answer in this question. The problem described in this question is not the same as mine (the alembic didn't create an empty migration file in this case) though. That's why I asked the question here (may be seen as duplicated).
So as the answer suggested, I commented the Base.metadata.create_all(engine) in the __init__.py. It is this line of code that creates the user table (if it not exist). So when alembic checks my database, the table is already there. Since no differences are found, the upgrade() and downgrade() are empty.

in this situation don't import your Base model from base file, import your Base from some where you have extend this Base class
for example I have a structure like below:
also i imported Base class in alembic/.env from models.trade.
in this form your base class metadata will detect your models and your auto generation migrations will work fine.

Flask-Migrate No Changes Detected to Schema on first migration

I'm using Flask with Flask-SQLAlchemy and Flask-Migrate to create an application, however when I try to create a migration nothing happens.
I've created two tables in app/models.py:
from flask import current_app
from . import db
class Student(db.Model):
__tablename__ = 'students'
id = db.Column(db.Integer, primary_key=True)
email = db.Column(db.String(64), unique=True, nullable=False)
password_hash = db.Column(db.String(128))
def __init__(self, **kwargs):
super(Student, self).__init__(**kwargs)
def __repr__(self):
return '<Tutor {}>' % self.id
class Tutor(db.Model):
__tablename__ = 'tutors'
id = db.Column(db.Integer, primary_key=True)
email = db.Column(db.String(64), unique=True, index=True)
password_hash = db.Column(db.String(128))
def __init__(self, **kwargs):
super(Tutor, self).__init__(**kwargs)
def __repr__(self):
return '<Student %r>' % self.id
Then I also have app/__init__.py with the following code:
from flask import Flask
from flask_bootstrap import Bootstrap
from flask_sqlalchemy import SQLAlchemy
from flask_migrate import Migrate
#from .models import User, Task, Project, UserProject
from config import config
bootstrap = Bootstrap()
db = SQLAlchemy()
migrate = Migrate()
def create_app(config_name='default'):
#print config_name.name
app = Flask(__name__)
app.config.from_object(config[config_name])
config[config_name].init_app(app)
bootstrap.init_app(app)
db.init_app(app)
migrate.init_app(app, db)
## Register the main blueprint for main app functionality
from .main import main as main_blueprint
app.register_blueprint(main_blueprint)
return app
and app.py:
import os
from app import create_app, db
from app.models import Tutor, Student
app = create_app('default')
#app.shell_context_processor
def make_shell_context():
return dict(db=db, Tutor=Tutor, Student=Student)
I can run flask db init with no problem and it creates the migrations directory and all necessary files with the following output:
Creating directory /Users/Jasmine/projects/flask/flask-tutoring/migrations ... done
Creating directory /Users/Jasmine/projects/flask/flask-tutoring/migrations/versions ... done
Generating /Users/Jasmine/projects/flask/flask-tutoring/migrations/script.py.mako ... done
Generating /Users/Jasmine/projects/flask/flask-tutoring/migrations/env.py ... done
Generating /Users/Jasmine/projects/flask/flask-tutoring/migrations/README ... done
Generating /Users/Jasmine/projects/flask/flask-tutoring/migrations/alembic.ini ... done
Please edit configuration/connection/logging settings in '/Users/Jasmine/projects/flask/flask-tutoring/migrations/alembic.ini' before proceeding.
but when I try and run flask db migrate alembic can't detect that I've got tables in app/models.py. I get the following output:
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.env] No changes in schema detected.
There is no migration script created, its as though models.py doesn't exist.
Apologies if this is a repeated question, but I can't find another example where its the first migration that fails and no migration script at all is created.
I've tried checking if there is already a table created somewhere by running db.drop_all() in the shell but that doesn't seem to be the problem.
UPDATE
I figured out a way to solve this on my own but would like a better understanding of why this worked.
I re-named app.py to flasktutor.py and re-ran export FLASK_APP='flasktutor.py'. Subsequently the migration worked perfectly.
Please could someone explain why when the file was called app.py and I used export FLASK_APP='app.py' the migration did not register changes to the schema.

I encountered this problem and solved it by importing my models at env.py in the migrations folder right after the following comments
# add your model's MetaData object here
# for 'autogenerate' support
# from myapp import mymodel
# target_metadata = mymodel.Base.metadata
from app.models import Student, Tutor

Pls make sure that you already import your models (Tutor, Student, ...) before your migrate.init_app(app, db)

I just ran into the same issue, following Miguel's great tutorial, and the hints here were very helpful!
My solution - which doesn't require "hacking" env.py - is to simply add
from app import models
to __init__.py.
If you have a different project layout you might need to adapt the import but it seems like you need to make sure that models gets imported properly for flask db migrate to work.

This is how I solved the problem in my case:
Import the Migrate model: from flask_migrate import Migrate
Initiate Migrate class: migrate = Migrate(app, db)
Comment db.create_all()
Drop your database now => DROP DATABASE db_name;
Create it again => CREATE DATABSE db_name OWNER owner_name;
Export you flask entry file => export FLASK_APP=name_app.py
Run flask db migrate
Note: The 6th step should be used in case you get this error:
Error: Could not locate a Flask application
Hope this will help someone.

Well, I encountered the same problem following Miguel Grinberg tutorial.
Previously I created the tables using the shell calling
db.create_all()
So, I thougth to drop the tables
db.drop_all()
and trying the migrate command again, it worked as expected:
Roberto#MyPC MINGW64 /e/Projects/Flask/flasky ((5c))
$ flask db migrate -m "initial migration - Role Users"
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.autogenerate.compare] Detected added table 'roles'
INFO [alembic.autogenerate.compare] Detected added table 'users'
INFO [alembic.autogenerate.compare] Detected added index 'ix_users_username' on '['username']'
Generating E:\Projects\Flask\flasky\migrations\versions\4de323c9c089_initial_migration_role_users.py ... done
After that, I used flask-migrate to re-create the tables
$ flask db upgrade
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.runtime.migration] Running upgrade -> b641ee80a60d, initial migration - Role Users

I encountered the same problem following Miguel Grinberg tutorial.
I solved this my adding
from app.models import User, Post
to migrations/env.py

The flask-migrate works after you change your table schema.
Such as, before:
from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy(app)
class Test(db.Model):
content = db.Column(db.String(60))
and then you change your Test schema, like:
class Test(db.Model):
content = db.Column(db.String(60))
add = db.Column(db.String(60))
Now, you can use flask db migrate -m "migrate test" to work.
You will get migrate version information.

This worked for me:
In migrations/env.py, import your models
# add your model's MetaData object here
# for 'autogenerate' support
# from myapp import mymodel
# target_metadata = mymodel.Base.metadata
from app.models import * # <= Add this line
from flask import current_app

You can point to empty database only to create migration file, then return back to upgrade your real database.

After some time trying to fix it and reading about it, I just deleted the migrations directory (folder) and the app.db.
Then ran again:
"flask db init"
"flask db migrate"
"flask db upgrade"
This re-generate the directories and all working fine now.
If you want to improve your migration script, this link to an upgrade from Miguel, can be useful: https://www.youtube.com/watch?v=wpRVZFwsD70&feature=emb_logo

Flask SQLAlchemy column "id" referenced in foreign key constraint does not exist

Minimal example:
models.py
from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy()
class Patient(db.Model):
id = db.Column(db.Integer, primary_key=True)
notes = db.relationship("Note", backref=db.backref("patient", lazy=True))
class Note(db.Model):
id = db.Column(db.Integer, primary_key=True)
patient_id = db.Column(db.Integer, db.ForeignKey("patient.id"), nullable=False)
app.py
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
app = Flask(__name__)
app.secret_key = "super secret"
POSTGRES = {
"user": "postgres",
"pw": "password",
"db": "test_db",
"host": "localhost",
"port": "5432",
}
app.config["SQLALCHEMY_DATABASE_URI"] = "postgresql://%(user)s:%(pw)s#%(host)s:%(port)s/%(db)s" % POSTGRES
app.config["SQLALCHEMY_TRACK_MODIFICATIONS"] = False
app.config["DEBUG"] = True
from models import db
with app.app_context():
db.init_app(app)
run.py
from app import app, db
if __name__ == "__main__":
with app.app_context():
db.create_all()
app.run()
However, I get the following error:
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) column "id" referenced in foreign key constraint does not exist
[SQL: '\nCREATE TABLE note (\n\tid SERIAL NOT NULL, \n\tpatient_id INTEGER NOT NULL, \n\tPRIMARY KEY (id), \n\tFOREIGN KEY(patient_id) REFERENCES patient (id)\n)\n\n'] (Background on this error at: http://sqlalche.me/e/f405)
It seems to work when I define the table with the foreign key in a psql console. What's going wrong?

I tried your sample code (I had to add the app initialisation to app.py so your code would run as-is). It worked as expected and both the note and patient tables were created.
This tells me that your issue is environmental. I'm willing to bet that if you created a brand new test database in your Postgres instance and ran your example code it would work for you too.
So let's focus on the state of the database you're connecting to.
The ProgrammingError exception you're getting shows an error coming from Postgres itself. It's saying that it can't create the notes table because there's no such foreign key as patient.id. This is probably throwing you off because you know you are defining a patient.id key in models.py. Unfortunately I don't have enough information from what you've posted to give you a definitive answer, but my guess is this:
The patient table in Postgres may have already been created from a previous run, but with a different schema (e.g. maybe it was first defined without an id column). the create_all() function will create tables that don't exist in the target database, but will not update existing tables with a modified schema.
Go check your Postgres DB and take a look at the patient table. Does it actually have an id column that is properly defined as a primary key?
If there's no data in these tables that you need, try dropping them and running your app again. My guess is that it will create both tables correctly and throw no errors.

Alembic Migrations on Multiple Models

I am attempting to create a revision with --autogenerate using Alembic for two Models, but am receiving a duplicate table keys error. Does, a schema need to be specified? If so, how can it be set? The documentation I've read says to use __table_args__ = {'schema': 'somename'}, but that hasn't helped. Any tips or suggestions are greatly appreciated.
My current setup is:
base.py
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
workspace.py
from sqlalchemy import Column, Integer, String
from base import Base
class WorkspaceModel(Base):
__tablename__ = 'workspaces'
id = Column(Integer, primary_key=True)
name = Column(String)
host.py
from sqlalchemy import Column, Integer, String
from base import Base
class HostModel(Base):
__tablename__ = 'hosts'
id = Column(Integer, primary_key=true)
ip = Column(String)
alembic/env.py
from host import HostModel
from workspace import WorkspaceModel
target_metadata = [HostModel.metadata, WorkspaceModel.metadata]
Error
ValueError: Duplicate table keys across multiple MetaData objects: "hosts", "workspaces"

To make it clear from what #esdotzed and #univerio said, you have to use a single Base.metadata - but still import the individual models.
In the original question, this is how the alembic/env.py should look like:
from base import Base
# This two won't be referenced, but *have* to be imported to populate `Base.metadata`
from host import HostModel
from workspace import WorkspaceModel
target_metadata = Base.metadata
If you didn't import both models, the autogenerated migration would end up deleting your whole database - because Base.metadata doesn't know about any model by itself.

quoting univerio's answer from the comment section:
target_metadata should just be target_metadata = Base.metadata
Using Base.metadata doesn't mean you can remove the imports from host import HostModel and from workspace import WorkspaceModel
It worked for me.

I just want to add to #mgarciaisaia answer, it will work but the thing is when I tried changing, for example, the max length of username field of User model and running alembic revision --autogenerate -m "test migration", alembic output a migration file with empty upgrade() and downgrade() functions!
Note: the following operations will erase your data from the database so please back them up beforehand!
In order to update the changes made to the original User model, I had to
Delete the first migration file
Rerun alembic revision --autogenerate -m "update user model" and alembic upgrade head again for the changes to appear inside upgrade() and downgrade() functions of the migration file.