I tried to add celery to my existing flask project. After adding, I got an "working outside of application context" error while running. It seems that the celery worker lacks of my application context. But I am not sure where to pass the applicaiton context to celery worker in this case.
Here is my current structure (I tried to follow a factory pattern with blueprints and api documentions):
-run.py
-app
-module1
-controller.py
-model.py
-service.py
-__init__.py
-config.py
For the init.py
# __init__.py
import os
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from flask_cors import CORS
from app.config import Config
from flask_restplus import Api
from celery import Celery
cors = CORS()
db = SQLAlchemy()
api = Api()
celery = Celery(__name__, broker=Config.CELERY_BROKER_URL, include=["app.module1.service"])
def create_app(config_class = Config):
app = Flask(__name__, static_url_path='')
app.config.from_object(Config)
cors.init_app(app)
db.init_app(app)
api.init_app(app=app)
celery.conf.update(app.config)
from app.module1.controller import blueprint
from app.module1.controller import ns
app.register_blueprint(blueprint)
api.add_namespace(ns)
return app
For the run.py
from app import create_app
app = create_app()
if __name__ == '__main__':
app.run(threaded=True, debug=True)
For the service.py
from app import db, celery
#celery.task(bind=True)
def service1(self):
# do somethigng & return
For the controller.py
from flask import Blueprint
from flask_restplus import Api, Resouce
blueprint = Blueprint('service', __name__)
apis = Api(app = blueprint)
ns = apis.namespace('service', 'service description')
#ns.route("/")
class SomeList(Resource):
def get(self):
service1.apply_async()
# return
I think the confusion is based on the fact that you are trying to "pass" an application context to the Celery worker. In reality the Flask process cannot pass a context to the worker because they are different processes. The Celery worker process needs to create its own Flask application instance by calling create_app() so that it can push its own app contexts when needed.
So for example, in your service1 task:
from app import db, celery, create_app
#celery.task(bind=True)
def service1(self):
app = create_app()
with app.app_context():
# do somethigng & return
To make this a bit more efficient, you can create a single global app that is shared by all your tasks:
from app import db, celery, create_app
app = create_app()
#celery.task(bind=True)
def service1(self):
with app.app_context():
# do somethigng & return
Related
I am trying to find a way to structure my code such that I can view logs in the production server hosted on Heroku.
Every tutorial I found seems to do logging like this:
How to show stdout logs in Heroku using Flask?
However, I am using the application factory pattern which utilizes Blueprints. As such here are some sample:
main.py
from app import create_app
app = create_app()
if __name__ == "__main__":
app.run()
app/_ _ init _ _.py
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
...
db = SQLAlchemy()
...
def create_app():
app = Flask(__name__)
...
db.init_app(app)
from .routes.demo_blueprint import demo_blueprint
...
# Register the routes to the Flask object
app.register_blueprint(demo_blueprint)
...
app/routes/demo_blueprint.py
from app import db
...
demo_blueprint = Blueprint('demo_blueprint', __name__)
#demo_blueprint.route('/demo', methods=['GET'])
...
In order to perform logging at the blueprint level, I would need to import app from main.py. However, this would cause an import error since __init__.py imports the blueprint before app is created. I was wondering if there were any work arounds for this.
Turns out it was a simple fix. To access the application context in the Blueprint, just use current_app. Following the example:
How to show stdout logs in Heroku using Flask?
main.py
from app import create_app
app = create_app()
if __name__ == "__main__":
app.run()
app/_ _ init _ _.py
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
...
db = SQLAlchemy()
...
def create_app():
app = Flask(__name__)
if __name__ != '__main__':
gunicorn_logger = logging.getLogger('gunicorn.error')
app.logger.handlers = gunicorn_logger.handlers
app.logger.setLevel(gunicorn_logger.level)
...
db.init_app(app)
from .routes.demo_blueprint import demo_blueprint
...
# Register the routes to the Flask object
app.register_blueprint(demo_blueprint)
...
app/routes/demo_blueprint.py
from flask import ***current_user***
from app import db
...
demo_blueprint = Blueprint('demo_blueprint', __name__)
#demo_blueprint.route('/demo', methods=['GET'])
def demo():
current_app.logger.debug('debug message: %s', 'test')
...
I'm trying to run my flask app with "python app.py" instead of "flask run" command.
My goal is to launch the app on cpanel server and just about every tutorial requires the applications to be called using the "python" method.
Here is my folder structure:
project
webapp
init.py
templates
static
auth.py
main.py
app.py <-------------- I want this to be called with python instead of flask run command outside the folder
Here is my init_.py file:
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from flask_login import LoginManager
# init SQLAlchemy so we can use it later in our models
db = SQLAlchemy()
def create_app():
app = Flask(__name__)
app.config['SECRET_KEY'] = '9OLWxND4o83j4iuopO'
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///db.sqlite'
db.init_app(app)
login_manager = LoginManager()
login_manager.login_view = 'auth.login'
login_manager.init_app(app)
from .models import User
#login_manager.user_loader
def load_user(user_id):
# since the user_id is just the primary key of our user table, use it in the query for the user
return User.query.get(int(user_id))
# blueprint for auth routes in our app
from .auth import auth as auth_blueprint
app.register_blueprint(auth_blueprint)
# blueprint for non-auth parts of app
from .main import main as main_blueprint
app.register_blueprint(main_blueprint)
return app
And app.py is:
from webapp import app
I'm a newbie in flask, any help is appreciated
Insert the call to create_app at the end of init.py:
if __name__ == '__main__':
create_app().run(host='0.0.0.0', port=5000, debug=True)
The if statement avoid calling the app many times. It can only be called directly. Flask default host is 127.0.0.1 (localhost). Use 0.0.0.0 at production for better traffic monitoring. Default port is also 5000, so it's up to you to include. For better readability, you should explicit it.
Then call it
python webapp/init.py
I am trying to implement Flask-PyMongo with blueprints and an application factory and keep getting AttributeError: 'Flask' object has no attribute 'db'
My directory structure looks like
myapp/
myapp.py
config.py
/app
__init__.py
/v1
__init__.py
endpoints.py
In my python script that starts the Flask app I have:
import os
from app import create_app
app = create_app('dev')
In my top level init.py I have:
mongo = PyMongo()
def create_app(config_name):
app = Flask(__name__)
mongo.init_app(app)
app.config.from_object(config[config_name])
from app.v1 import psapi as psapi_bp
app.register_blueprint(psapi_bp, url_prefix='/api')
if not os.path.exists('logs'):
os.mkdir('logs')
In my endpoints.py I have a route that looks like
#myapp.route('/addentry', methods=['POST'])
def addentry():
username = request.json['username']
userid = current_app.db.user_entry.insert({'username':username})
return jsonify({'userid':userid})
I feel like there is something small that I am missing but I am not seeing it.
You need to call db on your mongo object, not on the app object
to those who may be facing this problem again :
you should first define mongo oustside create_app to have access to it from inside other files.
then init_app with that like the following:
from flask import Flask, current_app
from flask_pymongo import PyMongo
mongo = PyMongo()
def create_app(config_name):
app = Flask(__name__, instance_relative_config=False)
app.config.from_object(app_config[config_name])
# INIT EXTENSIONS ----------------------
mongo.init_app(app)
return app
then in any file you can import mongo from above file. for example:
from ../factory import mongo
I am trying to use celery in an app to run long tasks asynchronously.
In the top project folder I have application.py:
from flask_stormpath import StormpathManager
from app import create_app
from celery import Celery
app = create_app('DevelopmentConfig')
stormpath_manager = StormpathManager(app)
celery = Celery(app.name, broker=app.config.from_object('CELERY_BROKER_URL'))
celery.conf.update(app.config)
if __name__ == '__main__':
app.run()
The config.py looks like this:
class Config:
SECRET_KEY = 'something_very_secret'
broker_url = 'sqs://'
broker_transport_options = {'region': 'eu-west-1',
'visibility_timeout': 3600,
'polling_interval': 0.3,
'queue_name_prefix': 'celery-'}
csrf = SECRET_KEY
CELERY_BROKER_URL = 'redis://localhost:6379/0'
CELERY_RESULT_BACKEND = 'redis: //localhost:6379/0'
#staticmethod
def init_app(app):
pass
class DevelopmentConfig(Config):
JUST_SOMETHING = 'a_little_trick'
DEBUG = True
STORMPATH_API_KEY_FILE = '/.stormpath/apiKey.properties'
STORMPATH_APPLICATION = 'flask-test'
CELERY_BROKER_URL = 'redis://localhost:6379/0'
CELERY_RESULT_BACKEND = 'redis: //localhost:6379/0'
class ProductionConfig(Config):
JUST_SOMETHING = 'a_little_trick'
DEBUG = True
STORMPATH_API_KEY_FILE = '/.stormpath/apiKey.properties'
STORMPATH_APPLICATION = 'flask-test'
CELERY_BROKER_URL = 'sqs://'
config = {'development': DevelopmentConfig,
'default': DevelopmentConfig}
and in my views.py I try to run a task:
from flask import render_template, flash, request, jsonify, Response
from wtforms import Form, validators, SelectMultipleField, widgets
from flask_stormpath import login_required
from . import main
import numpy as np
class MultiCheckboxField(SelectMultipleField):
widget = widgets.ListWidget(prefix_label=False)
option_widget = widgets.CheckboxInput()
#celery.task(bin=True)
def do_something(test, training_size, testing_size):
Now, when I run it like this, I get the message that #celery.task the name celery was not defined. Fair point, so I changed it to #main.celery.task. When I do this, I get the error message "AttributeError: 'Blueprint' object has no attribute 'celery'.
Then I tried to initiate celery in the init.py file:
from flask import Flask
from celery import Celery
def create_app(config_name):
app = Flask(__name__)
configuration = "config."+config_name
app.config.from_object(configuration)
celery = Celery(app.name, broker=app.config.from_object('CELERY_BROKER_URL'))
celery.conf.update(app.config)
from .main import main as main_blueprint
app.register_blueprint(main_blueprint)
return app
When I do this, I get the error: ImportError: No module named CELERY_BROKER_URL
So, I don't know, where to import and initiate celery and how to create a blueprint so that I can use celery.task in views.py. Any help would be highly appreciated.
I use supervisor to run my app. It is structured as follows:
My app layout
my_app
__init__.py
my_app
__init__.py
startup
create_app.py
create_users.py
common_settings.py
core
__init__.py
models.py
views.py
Outer __init__.py
from my_app import app
Inner __init__.py
from flask import Flask
from flask_script import Manager
from flask_sqlalchemy import SQLAlchemy
app = Flask(__name__) # The WSGI compliant web application object
db = SQLAlchemy(app) # Setup Flask-SQLAlchemy
manager = Manager(app) # Setup Flask-Script
from my_app.startup.create_app import create_app
create_app()
create_app.py
def create_app(extra_config_settings={}):
"""
Initialize Flask applicaton
"""
# ***** Initialize app config settings *****
# Read common settings from 'app/startup/common_settings.py' file
app.config.from_object('app.startup.common_settings')
# Read environment-specific settings from file defined by OS environment variable 'ENV_SETTINGS_FILE'
app.config.from_envvar('ENV_SETTINGS_FILE')
# Load all blueprints with their manager commands, models and views
# Setup Flask-User to handle user account related forms
from my_app.core.models import User
# Setup Flask-User
db_adapter = SQLAlchemyAdapter(db, User) # Setup the SQLAlchemy DB Adapter
user_manager = UserManager(db_adapter, app) # Init Flask-User and bind to app
from my_app import core
return app
my_app/core/__init__.py
from . import models
from . import views
views.py
from my_app import db, app
'''
Register a new user
'''
#app.route('/register', methods = ['POST'])
def register_user():
user_manager = app.user_manager
db_adapter = user_manager.db_adapter
I was trying to follow an example I found online.
I'm creating the variables db_adapter and user_manager in create_app(). Are these the same ones being used in my views.py?
If anyone has any suggestions or links to examples that I can follow to structure my project, it would be greatly appreciated.
Thanks.
Assuming that that's how Flask-User works (sets the user_manager attribute on app), this is trivial to determine, just compare them in the create_app function when you still have a direct reference to the objects.
db_adapter = SQLAlchemyAdapter(db, User)
user_manager = UserManager(db_adapter, app)
assert db_adapter is user_manager.db_adapter
assert user_manager is app.user_manager
However, your entire project layout doesn't make much sense. You should be creating the entire app inside the create_app factory. You should not have an __init__.py file at the top level, that's the project folder not the package. You should use current_app within views to access the app, since it will only be created at runtime by the factory. You should create a manage.py file at the project level to use the factory.
my_project/
my_app/
__init__.py
models.py
views.py
defaults.py
instance/
config.py
manage.py
__init__.py:
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy()
def create_app():
app = Flask(__name__, instance_relative_config=True)
app.config.from_object('my_app.defaults')
app.config.from_pyfile('config.py')
db.init_app(app)
from my_app.views import bp
app.register_blueprint(bp)
return app
models.py:
from my_app import db
class User(db.Model):
...
views.py:
from flask import Blueprint, render_template
from my_app.models import User
bp = Blueprint('app', __name__)
#bp.route('/')
def index():
return render_template('index.html')
manage.py:
#!/usr/bin/env python
from flask_script import Manager
from my_app import create_app
Manager(create_app).run()