We Keep Coding

How to preserve Flask app context across Celery and SQLAlchemy

I'm building trying to learn Flask with a proof of concept Flask app, that takes a JSON payload, and uses SQLAlchemy to write it to a DB. I'm using celery to manage the write tasks.
The app is structured
|-app.py
|-project
|-__init__.py
|-celery_utils.py
|-config.py
|-users
|-__init_.py
|-models.py
|-tasks.py
app.py builds the flask app and celery instance.
app.py
from project import create_app, ext_celery
app = create_app()
celery = ext_celery.celery
#app.route("/")
def alive():
return "alive"
/project/__init__.py is the application factory for the flask app. It instantiates the extensions, links everything together, and registers the blueprints.
/project/init.py
import os
from flask import Flask
from flask_celeryext import FlaskCeleryExt
from flask_migrate import Migrate
from flask_sqlalchemy import SQLAlchemy
from project.celery_utils import make_celery
from project.config import config
# instantiate extensions
db = SQLAlchemy()
migrate = Migrate()
ext_celery = FlaskCeleryExt(create_celery_app=make_celery)
def create_app(config_name=None):
if config_name is None:
config_name = os.environ.get("FLASK_CONFIG", "development")
# instantiate the app
app = Flask(__name__)
# set config
app.config.from_object(config[config_name])
# set up extensions
db.init_app(app)
migrate.init_app(app, db)
ext_celery.init_app(app)
# register blueprints
from project.users import users_blueprint
app.register_blueprint(users_blueprint)
# shell context for flask cli
#app.shell_context_processor
def ctx():
return {"app": app, "db": db}
return app
/project/celery_utils.py manages the creation of the celery instances
/project/celery_utils.py
from celery import current_app as current_celery_app
def make_celery(app):
celery = current_celery_app
celery.config_from_object(app.config, namespace="CELERY")
return celery
In the users dir, I'm trying to manage the creation of a basic user with celery task management.
'/project/users/init.py` is where I create the blueprints and routes.
/project/users/init.py
from flask import Blueprint, request, jsonify
from .tasks import divide, post_to_db
users_blueprint = Blueprint("users", __name__, url_prefix="/users", template_folder="templates")
from . import models, tasks
#users_blueprint.route('/users', methods=['POST'])
def users():
request_data = request.get_json()
task = post_to_db.delay(request_data)
response = {"id": task.task_id,
"status": task.status,
}
return jsonify(response)
#users_blueprint.route('/responses', methods=['GET'])
def responses():
request_data = request.get_json()
result = AsyncResult(id=request_data['id'])
response = result.get()
return jsonify(response)
/project/users/models.py is a simple User model - however, it does manage to successfully remain in the context of the flask app if created from the flask app cli.
/project/users/models.py
from project import db
class User(db.Model):
"""model for the user object"""
__tablename__ = "users"
id = db.Column(db.Integer, primary_key=True, autoincrement=True)
username = db.Column(db.String(128), unique=True, nullable=False)
email = db.Column(db.String(128), unique=True, nullable=False)
def __init__(self, username, email, *args, **kwargs):
self.username = username
self.email = email
Finally, /project/users/tasks.py is where I handle the celery tasks for this dir.
/project/users/tasks.py
from celery import shared_task
from .models import User
from project import db
#shared_task()
def post_to_db(payload):
print("made it here")
user = User(**payload)
db.session.add(user)
db.session.commit()
db.session.close()
return True
The modules work, but as soon as I wire it all up and hit the endpoint with a JSON payload, I get the error message:
RuntimeError: No application found. Either work inside a view function or push an application context. ...
I have tried to preserve the app context in tasks.py by:
...
from project import db, ext_celery
#ext_celery.shared_task()
def post_to_db(payload):
...
...
from project import db, ext_celery
#ext_celery.task()
def post_to_db(payload):
...
These error with: TypeError: exceptions must derive from BaseException
I've tried pushing the app context
...
from project import db
from app import app
#shared_task()
def post_to_db(payload):
with app.app_context():
...
This also errors with: TypeError: exceptions must derive from BaseException
I've tried importing celery from the app itself
...
from project import db
from app import celery
#celery.task()
def post_to_db(payload):
...
This also errors with: TypeError: exceptions must derive from BaseException
Any suggestions gratefully received. There's a final piece of the puzzle I'm missing, and it's very frustrating.

With thanks to snakecharmerb
I had to add ContextTask to the make_celery() function in /project/celery_utils.py
from celery import current_app as current_celery_app
def make_celery(app):
celery = current_celery_app
celery.config_from_object(app.config, namespace="CELERY")
class ContextTask(celery.Task):
def __call__(self, *args, **kwargs):
with app.app_context():
return self.run(*args, **kwargs)
celery.Task = ContextTask
return celery
And then a few tweaks in /project/users/tasks.py
from celery import shared_task
from .models import User
from project import db
#shared_task()
def post_to_db(payload):
user = User(**payload)
db.session.add(user)
db.session.commit()
db.session.close()
return True
Now I can see the user in the database, and my message queue is progressing as expected.

APScheduler RuntimeError: No application found

I have a problem. I have IP addresses and with APScheduler I try to ping them every 10 seconds and update my database. For APScheduler I understand that I need to use with app.app_context() but the problem is that I don't know where to place it and wherever I have tried to place it, it raise: RuntimeError: No application found and fails to update the database
init.py:
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from os import path
db = SQLAlchemy()
DB_NAME = "database.db"
def create_app():
app = Flask(__name__)
app.config['SECRET_KEY'] = 'hjshjhdjah kjshkjdhjs'
app.config['SQLALCHEMY_DATABASE_URI'] = f'sqlite:///{DB_NAME}'
db.init_app(app)
from .views import views
from .auth import auth
app.register_blueprint(views, url_prefix='/')
app.register_blueprint(auth, url_prefix='/')
from .models import Servers
create_database(app)
return app
def create_database(app):
if not path.exists('websiteflaskupdatedd/' + DB_NAME):
db.create_all(app=app)
print('Created Database!')
auth.py:
from flask import Blueprint, render_template, request, flash, redirect, url_for, escape, session
from .models import Servers, ping
from . import db, create_app
from apscheduler.schedulers.background import BackgroundScheduler
#auth.route('/servers', methods = ['POST', 'GET'])
def servers():
if request.method == 'POST':
server_ip = request.form.get("server_ip")
new_server = Servers(server_ip = server_ip)
try:
db.session.add(new_server)
db.session.commit()
return redirect("/servers")
except:
return "There was an error adding your server"
else:
if not Servers.query.get(1) == None:
servers = Servers.query.order_by(Servers.date_created)
return render_template('servers.html', servers=servers)
def update_up_status():
with create_app().app_context():
server_status_column = Servers.query.filter(Servers.server_status.in_(["0","1"]))
for server in server_status_column:
server.server_status = ping(server.server_ip)
db.session.commit()
scheduler = BackgroundScheduler()
if not scheduler.running:
scheduler.add_job(func=update_up_status, trigger="interval", seconds=10)
scheduler.start()
models.py:
from . import db
from datetime import datetime
import platform
import subprocess
def ping(host):
cmd = ['ping', '-w', "1", "-n", '1', host]
return subprocess.call(cmd) == 0
class Servers(db.Model):
id = db.Column(db.Integer, primary_key=True)
server_ip = db.Column(db.String(12), nullable=False, unique=True)
server_status = db.Column(db.String(12), nullable=False)
date_created = db.Column(db.DateTime, default= datetime.now)

Looks like you don't have your application running.
I use this setup regularly (flask + apscheduler) and have a main.py file with the following (including the necessary module imports from your other files):
if __name__ == "__main__":
scheduler = BackgroundScheduler()
scheduler.add_job(func=update_up_status, trigger="interval", seconds=10)
scheduler.start()
app.run(host="0.0.0.0", debug=True)
The background scheduler works within the background of another application and will not be blocked by the main thread which in this case is the flask application. As opposed to the blocking scheduler which will block the main thread and cannot be run inside of a another application.

Just like this.
def insertArticles(articles):
from app import db
from app.models.articles import Article
from app import scheduler
# 解决 flask apscheduler RuntimeError: No application found
with scheduler.app.app_context():
for article in articles:
# 字典 解包，直接插入字典
new_article = Article(**article)
db.session.add(new_article)
db.session.flush()
db.session.commit()
with scheduler.app.app_context() is the important line.

Flask, Blueprint Collisions and Pytest

I am using Flask Blueprints to create an application. I am testing it with pytest. Generating test_client in different states with pytest.fixture is causing a blueprint name collision.
This is my Flask __init__ code:
from flask import Flask
from flask_dropzone import Dropzone
from config import Config
from flask_sqlalchemy import SQLAlchemy
from flask_migrate import Migrate
from flask_user import UserManager
from flask_googlemaps import GoogleMaps
import os
from app.settings import app, db
# get extensions
dropzone = Dropzone()
google_maps = GoogleMaps()
migrate = Migrate()
def create_app(config_class=Config):
# initialise extensions
google_maps.init_app(app)
dropzone.init_app(app)
db.init_app(app)
migrate.init_app(app, db)
# FLASK-User Stuff
from app.models import User
user_manager = UserManager(app, db, User)
from app.errors import bp as errors_bp
app.register_blueprint(errors_bp)
from app.main import bp as main_bp
app.register_blueprint(main_bp)
return app
I am then trying to use multiple pytests to test different aspects of the application. For each I am creating a pytest.fixture to generate a test_client. For example I'm creating two fixtures for testing the app with the db in different states.
For example.
#pytest.fixture(scope='module')
def client_state1():
app = create_app()
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite://' + PATH_TO_DB_STATE_1
app.config['TESTING'] = True
client = app.test_client()
ctx = app.app_context()
ctx.push()
yield client
ctx.pop()
#pytest.fixture(scope='module')
def client_state2():
app = create_app()
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite://' + PATH_TO_DB_STATE_2
app.config['TESTING'] = True
client = app.test_client()
ctx = app.app_context()
ctx.push()
yield client
ctx.pop()
def test_with_state1(client_state1):
"""Test should pass"""
assert 1 == 1
def test_with_state2(client_state2):
"""Test should pass"""
assert 1 == 1
When I run the tests I get the following error.
AssertionError: A blueprint's name collision occurred between <flask.blueprints.Blueprint object at 0x1a16b5d4e0> and <flask.blueprints.Blueprint object at 0x1a1633f6a0>. Both share the same name "googlemaps". Blueprints that are created on the fly need unique names.
I have tried splitting out tests into separate files and setting the scope of the fixture to function. But to no avail. In all cases the first test passes and the second causes the collision. What am I doing wrong?

Access to Flask Global Variables in Blueprint Apps

my source code has this structure:
main.py:
from flask import Flask, g
app = Flask(__name__)
with app.app_context():
g.my_db = PostgreSQL()
app.register_blueprint(my_app, url_prefix="/my_app")
my_app.py:
from flask import Blueprint, g
my_app = Blueprint("my_app", __name__)
#my_app.route("/")
def index():
return g.my_db.fetch_all() <<< ERROR
but it shows this error:
AttributeError: '_AppCtxGlobals' object has no attribute 'my_db'
Even when I try to use g outside of app context, it shows this error:
RuntimeError: Working outside of application context.
So how to set and access to global variables in Flask?

This happens because the data are lost when the context (with app.app_context()) ends (doc).
Inside the context, everything is ok :
from flask import Flask, g
app = Flask(__name__)
with app.app_context():
g.my_db = 'database ok'
print(g.my_db)
# >>> this prints 'database ok'
But outside, you cannot access the attribute :
from flask import Flask, g
app = Flask(__name__)
with app.app_context():
g.my_db = 'database ok'
print(g.my_db)
# >>> this throws RuntimeError: Working outside of application context
even if you create a new context:
from flask import Flask, g
app = Flask(__name__)
with app.app_context():
g.my_db = 'database ok'
with app.app_context():
print(g.my_db)
>>> this throws AttributeError: '_AppCtxGlobals' object has no attribute 'my_db'
Your best call should be to declare the database object before the context, and then import it. Or maybe you can create it directly inside my_app.py where you need it ?

g isn't persistent in the way you're trying to use it. Write a function to create a connection each time you need it. Preferably use a database extension like Flask-SQLAlchemy to manage connections for you.
db.py:
import <postgresql dependencies>
def get_db():
db = PostgreSQL()
# config here
return db
main.py:
from flask import Flask
app = Flask(__name__)
app.register_blueprint(my_app, url_prefix="/my_app")
my_app.py:
from flask import Blueprint, g
from db import get_db
my_app = Blueprint("my_app", __name__)
#my_app.route("/")
def index():
db = get_db()
data = db.fetch_all()
db.close()
return data

Avoiding circular imports Python

I know this question have already been answered, but for my application, I can't resolve it. What I'm trying to do is to setup a database using MongoAlchemy instead of using MongoClient. I want to have diffrent scripts for every operation This is my main file, app.py:
import os
from flask import Flask, jsonify
from blueprints.db import insert_db
from blueprints.delete_blueprint import delete_bp
from blueprints.insert_blueprint import insert_bp
from blueprints.read_blueprint import read_bp
from blueprints.login_update_blueprint import log_update
import traceback
app = Flask(__name__)
app.register_blueprint(log_update)
app.register_blueprint(read_bp)
app.register_blueprint(insert_bp)
app.register_blueprint(delete_bp)
# database configuration
app.config['MONGOALCHEMY_DATABASE'] = 'rest_api'
# populate table with initial values
insert_db.populateTables()
# mail configuration
app.config['MAIL_SERVER'] = 'smtp.gmail.com'
app.config['MAIL_PORT'] = 465
app.config['MAIL_USERNAME'] = 'chis.simion12#gmail.com'
app.config['MAIL_PASSWORD'] = ''
app.config['MAIL_USE_TLS'] = False
app.config['MAIL_USE_SSL'] = True
#app.errorhandler(404)
def page_not_found(c):
return 'The required page does not exist'
#app.errorhandler(400)
def bad_req(c):
return 'Bad request. Please review the request'
#app.errorhandler(Exception)
def handle_Exception(error):
print("\n ______ ERROR ______\n")
print(traceback.format_exc())
print("_____________________\n")
response = dict()
response['message'] = "A PROBLEM HAS OCCURED, PLEASE TRY AGAIN LATER"
response['status_code'] = 500
response = jsonify(response)
response.status_code = 500
return response
if __name__ == '__main__':
app.secret_key = os.urandom(12)
app.run(debug=True, threaded=True)
Note that I'm modyfing on the old version, so the imports are still there. The problem with the circular imports appear when I try do import app from app.py into the script which initializes the database, described below:
from flask import current_app
from flask_mongoalchemy import MongoAlchemy
# from app import app
db = MongoAlchemy(current_app)
class People(db.Document):
Name = db.StringField()
Age = db.IntField()
Password = db.StringField()
Vms = db.AnythingField()
If I try to use current_app I get the error: RuntimeError: Working outside of application context. which means that the app is not currently initialized, and if I import the app from app.py the circular dependency appears. I'm sorry if I wasn't pretty clear, so feel free to ask for more details.

I like to place all extensions on a extensions.py file and create a function to initialize and configurate the app instance
extensions.py
from flask_mongoalchemy import MongoAlchemy
db = MongoAlchemy()
models.py
from extensions import db
class Person(db.Model):
pass
flaskr.py
from flask import Flask
from extensions import db
def create_app():
app = Flask(__name__)
db.init_app(app)
return app
app = create_app()
app.run()