We Keep Coding

How to preserve Flask app context across Celery and SQLAlchemy

I'm building trying to learn Flask with a proof of concept Flask app, that takes a JSON payload, and uses SQLAlchemy to write it to a DB. I'm using celery to manage the write tasks.
The app is structured
|-app.py
|-project
|-__init__.py
|-celery_utils.py
|-config.py
|-users
|-__init_.py
|-models.py
|-tasks.py
app.py builds the flask app and celery instance.
app.py
from project import create_app, ext_celery
app = create_app()
celery = ext_celery.celery
#app.route("/")
def alive():
return "alive"
/project/__init__.py is the application factory for the flask app. It instantiates the extensions, links everything together, and registers the blueprints.
/project/init.py
import os
from flask import Flask
from flask_celeryext import FlaskCeleryExt
from flask_migrate import Migrate
from flask_sqlalchemy import SQLAlchemy
from project.celery_utils import make_celery
from project.config import config
# instantiate extensions
db = SQLAlchemy()
migrate = Migrate()
ext_celery = FlaskCeleryExt(create_celery_app=make_celery)
def create_app(config_name=None):
if config_name is None:
config_name = os.environ.get("FLASK_CONFIG", "development")
# instantiate the app
app = Flask(__name__)
# set config
app.config.from_object(config[config_name])
# set up extensions
db.init_app(app)
migrate.init_app(app, db)
ext_celery.init_app(app)
# register blueprints
from project.users import users_blueprint
app.register_blueprint(users_blueprint)
# shell context for flask cli
#app.shell_context_processor
def ctx():
return {"app": app, "db": db}
return app
/project/celery_utils.py manages the creation of the celery instances
/project/celery_utils.py
from celery import current_app as current_celery_app
def make_celery(app):
celery = current_celery_app
celery.config_from_object(app.config, namespace="CELERY")
return celery
In the users dir, I'm trying to manage the creation of a basic user with celery task management.
'/project/users/init.py` is where I create the blueprints and routes.
/project/users/init.py
from flask import Blueprint, request, jsonify
from .tasks import divide, post_to_db
users_blueprint = Blueprint("users", __name__, url_prefix="/users", template_folder="templates")
from . import models, tasks
#users_blueprint.route('/users', methods=['POST'])
def users():
request_data = request.get_json()
task = post_to_db.delay(request_data)
response = {"id": task.task_id,
"status": task.status,
}
return jsonify(response)
#users_blueprint.route('/responses', methods=['GET'])
def responses():
request_data = request.get_json()
result = AsyncResult(id=request_data['id'])
response = result.get()
return jsonify(response)
/project/users/models.py is a simple User model - however, it does manage to successfully remain in the context of the flask app if created from the flask app cli.
/project/users/models.py
from project import db
class User(db.Model):
"""model for the user object"""
__tablename__ = "users"
id = db.Column(db.Integer, primary_key=True, autoincrement=True)
username = db.Column(db.String(128), unique=True, nullable=False)
email = db.Column(db.String(128), unique=True, nullable=False)
def __init__(self, username, email, *args, **kwargs):
self.username = username
self.email = email
Finally, /project/users/tasks.py is where I handle the celery tasks for this dir.
/project/users/tasks.py
from celery import shared_task
from .models import User
from project import db
#shared_task()
def post_to_db(payload):
print("made it here")
user = User(**payload)
db.session.add(user)
db.session.commit()
db.session.close()
return True
The modules work, but as soon as I wire it all up and hit the endpoint with a JSON payload, I get the error message:
RuntimeError: No application found. Either work inside a view function or push an application context. ...
I have tried to preserve the app context in tasks.py by:
...
from project import db, ext_celery
#ext_celery.shared_task()
def post_to_db(payload):
...
...
from project import db, ext_celery
#ext_celery.task()
def post_to_db(payload):
...
These error with: TypeError: exceptions must derive from BaseException
I've tried pushing the app context
...
from project import db
from app import app
#shared_task()
def post_to_db(payload):
with app.app_context():
...
This also errors with: TypeError: exceptions must derive from BaseException
I've tried importing celery from the app itself
...
from project import db
from app import celery
#celery.task()
def post_to_db(payload):
...
This also errors with: TypeError: exceptions must derive from BaseException
Any suggestions gratefully received. There's a final piece of the puzzle I'm missing, and it's very frustrating.

With thanks to snakecharmerb
I had to add ContextTask to the make_celery() function in /project/celery_utils.py
from celery import current_app as current_celery_app
def make_celery(app):
celery = current_celery_app
celery.config_from_object(app.config, namespace="CELERY")
class ContextTask(celery.Task):
def __call__(self, *args, **kwargs):
with app.app_context():
return self.run(*args, **kwargs)
celery.Task = ContextTask
return celery
And then a few tweaks in /project/users/tasks.py
from celery import shared_task
from .models import User
from project import db
#shared_task()
def post_to_db(payload):
user = User(**payload)
db.session.add(user)
db.session.commit()
db.session.close()
return True
Now I can see the user in the database, and my message queue is progressing as expected.

Flask + Restplus adding route('/')

I'm getting an 404 errors when trying to setup routes for a Flask + Restplus app.
#api.route('/')
class asd(Resource):
def get(self):
return "asd"
Always return 404 error/ If I add #api.route('/'something) I can access "/something" URL without an error.
What do I miss? Or it's not possible when using Flask + Restplus?
I've tried adding '/' route tu the api an dto the app with same results.
app and api defines in separate class for server
class Server(object):
def __init__(self):
self.app = Flask(__name__)
self.api = Api(self.app,
version='0.1',
title='API',
description='A simple API',
doc = environment_config["swagger-url"]
)
def run(self):
self.app.run(
debug = environment_config["debug"],
port = environment_config["port"]
)
server = Server()

How do I get the application context in a Blueprint, but not in a request?

I am attempting to convert a collection of Flask apps to a single app with several Blueprints.
In one of my apps, I have a task that runs periodically in the background, not related to a request. It looks something like this:
import apscheduler.schedulers.background
import flask
app = flask.Flask(__name__)
app.config['DATABASE']
scheduler = apscheduler.schedulers.background.BackgroundScheduler()
scheduler.start()
def db():
_db = flask.g.get('_db')
if _db is None:
_db = get_db_connection_somehow(app.config['DATABASE'])
flask.g._db = _db
return _db
#scheduler.scheduled_job('interval', hours=1)
def do_a_thing():
with app.app_context():
db().do_a_thing()
When I convert this app to a Blueprint, I lose access to the app object and I can't figure out how to create an application context when I need one. This is what I tried:
import apscheduler.schedulers.background
import flask
bp = flask.Blueprint('my_blueprint', __name__)
scheduler = apscheduler.schedulers.background.BackgroundScheduler()
scheduler.start()
def db():
_db = flask.g.get('_db')
if _db is None:
_db = get_db_connection_somehow(flask.current_app.config['DATABASE'])
flask.g._db = _db
return _db
#bp.record
def record(state):
with state.app.app_context():
flask.g._app = state.app
#scheduler.scheduled_job('interval', hours=1)
def do_a_thing():
with flask.g._app.app_context():
db().do_a_thing()
The error I get is:
RuntimeError: Working outside of application context.
So, how can I get the application context in a blueprint but outside a request?

I solved this problem with the following changes. First, I set up a scheduler object on my Flask app:
app = flask.Flask(__name__)
app.scheduler = apscheduler.schedulers.background.BackgroundScheduler()
app.scheduler.start()
Next, I changed the function that runs my background task to accept the app as an argument, so I could read the database connection information from app.config:
def do_a_thing(app: flask.Flask):
db = get_db_connection_somehow(app.config['DATABASE'])
db.do_a_thing()
Finally, I set up the scheduled job in Blueprint.record():
#bp.record
def record(state):
state.app.scheduler.add_job(do_a_thing, trigger='interval', args=[state.app], hours=1)

Access mongo instance in Flask Unit Tests

I have a simple create_app function in app/__init__.py:
from flask import Flask
from flask_bootstrap import Bootstrap
from flask_login import LoginManager
from flask_pymongo import PyMongo
from .user_management import User
from config import app_config
login_manager = LoginManager()
mongo = PyMongo()
....
def create_app(config):
app = Flask(__name__, instance_relative_config=True, static_folder='static')
login_manager.init_app(app)
login_manager.login_message = 'You must be logged in to view this page'
login_manager.login_view = 'auth.login'
Bootstrap(app)
app.config.from_object(app_config[config])
app.config.from_pyfile('config.py')
if app.testing:
mongo.init_app(app, config_prefix='MONGO2')
else:
mongo.init_app(app)
....
return app
And my config:
class Config():
DEBUG = False
MONGO_HOST = 'localhost'
MONGO_PORT = 27017
....
class DevelopmentConfig(Config):
DEBUG = True
DEVELOPMENT = True
class TestingConfig(Config):
TESTING = True
DEBUG = False
CSRF_ENABLED = False
MONGO2_DBNAME = 'test'
....
app_config = {
'testing': TestingConfig,
'development': DevelopmentConfig,
'production': ProductionConfig
}
Throughout the app, I import the mongo instance from this file make use of it throughout the app. However, I can not find a find to set up a new mongo instance that uses the 'test' database without using the app context, as I'm doing here
My unit test file looks like this.
from app import create_app
import unittest
from app import mongo
class TestCase(unittest.TestCase):
def setUp(self):
app = create_app('testing')
self.app = app
def tearDown(self):
pass
def test_mongo(self):
with self.app.app_context():
assert mongo.db.name == 'test'
if __name__ == '__main__':
unittest.main()
This does not seem like the way to go at all. It also makes it impossible to use the app.test_client(). What is the proper way to instantiate the test database in a flask test setting?

I'm not sure how to do that via config files, but I have another approach that might interest you.
When I have been creating unit tests to my flask / mongo app, I have just simply read sys.args (in my '__init__.py') and decided from there if I want to use test DB or the actual DB, like this:
if "__init__.py" == sys.argv[0]:
db = LoginManager()
else:
db = TestDB() # if running unit tests
I don't know is this the most robust way to work with unit tests, but it is most certainly is super simple way to do it, and does the job.
If you are interested about the whole setup, check out my example app.

You will need to re-create the mongo object in each different unit test that needs to access a different database defined in your config. Here is how I did it and to get a real test I wrote a value to the database and made sure I could read it back two different ways:
import uuid
from datetime import datetime
from flask_testing import TestCase
from flask_pymongo import PyMongo
from project import app
class TestMongoDev(TestCase):
def create_app(self):
app.config.from_object('project.config.DevelopmentConfig')
return app
def test_mongo_development(self):
mongo = PyMongo(app)
testval = str(uuid.uuid4())
inserted_id = mongo.db.test.insert_one({
'testedAt': datetime.now(),
'testval': testval
}).inserted_id
self.assertTrue(mongo.db.name == 'dev')
self.assertTrue(mongo.db.test.find_one({'testval': testval})['testval'] == testval)
self.assertTrue(mongo.db.test.find_one({'_id': inserted_id})['testval'] == testval)
class TestMongoTest(TestCase):
def create_app(self):
app.config.from_object('project.config.TestingConfig')
return app
def test_mongo_testing(self):
mongo = PyMongo(app)
testval = str(uuid.uuid4())
inserted_id = mongo.db.test.insert_one({
'testedAt': datetime.now(),
'testval': testval
}).inserted_id
self.assertTrue(mongo.db.name == 'test')
self.assertTrue(mongo.db.test.find_one({'testval': testval})['testval'] == testval)
self.assertTrue(mongo.db.test.find_one({'_id': inserted_id})['testval'] == testval)
class TestMongoProd(TestCase):
def create_app(self):
app.config.from_object('project.config.ProductionConfig')
return app
def test_mongo_production(self):
mongo = PyMongo(app)
testval = str(uuid.uuid4())
inserted_id = mongo.db.test.insert_one({
'testedAt': datetime.now(),
'testval': testval
}).inserted_id
self.assertTrue(mongo.db.name == 'prod')
self.assertTrue(mongo.db.test.find_one({'testval': testval})['testval'] == testval)
self.assertTrue(mongo.db.test.find_one({'_id': inserted_id})['testval'] == testval)

How could I share common hook function for multiple flask servers

Suppose I need to do before_request for each flask servers
How can I share the following snippet to each servers without COPY-PASTE
#app.before_request
def before_request(*args, **kwargs):
params = get_params()
if params.has_key('start_dt') and params.has_key('end_dt'):
g.mongo_query = Mongo.get_date_range_query(params)
else:
g.mongo_query = {}

You could use application factory for this. If you initialize your flask applications like so:
from flask import Flask
import yourdb
def create_app(config_filename):
app = Flask(__name__)
app.config.from_pyfile(config_filename)
yourdb.init_app(app)
#add_extensions
#add_blueprints/views
# ... some other configuration ...
#app.before_request
def before_request(*args, **kwargs):
#Your code
return app
From manage/run, you would then
from somewhere import create_app
app = create_app(<your_config>)