We Keep Coding

How to get flask request context in celery task?

I have a flask server running within a gunicorn.
In my flask application I want to handle large upload files (>20GB), so I plan on letting a celery task do the handling of the large file.
The problem is that retrieving the file from request.files already takes quite long, in the meantime gunicorn terminates the worker handling that request. I could increase the timeout time, but the maximum file size is currently unknown, so I don't know how much time I would need.
My plan was to make the request context available to the celery task, as it was described here: http://xion.io/post/code/celery-include-flask-request-context.html, but I cannot make it work
Q1 Is the signature right?
I set the signature with
celery.signature(handle_large_file, args={}, kwargs={})
and nothing is complaining. I get the arguments I pass from the flask request handler to the celery task, but that's it. Should I somehow get a handle to the context here?
Q2 how to use the context?
I would have thought if the flask request context was available I could just use request.files in my code, but then I get the warning that I am out of context.
Using celery 4.4.0
Code:
# in celery.py:
from flask import request
from celery import Celery
celery = Celery('celery_worker',
backend=Config.CELERY_RESULT_BACKEND,
broker=Config.CELERY_BROKER_URL)
#celery.task(bind=True)
def handle_large_file(task_object, data):
# do something with the large file...
# what I'd like to do:
files = request.files['upfile']
...
celery.signature(handle_large_file, args={}, kwargs={})
# in main.py
def create_app():
app = Flask(__name__.split('.')[0])
...
celery_worker.conf.update(app.config)
# copy from the blog
class RequestContextTask(Task):...
celery_worker.Task = RequestContextTask
# in Controller.py
#FILE.route("", methods=['POST'])
def upload():
data = dict()
...
handle_large_file.delay(data)
What am I missing?

Flask APScheduler + Gunicorn workers - Still running task twice after socket fix

I have a Flask app where i use Flask-APScheduler to run a scheduled query on my database and send an email via a cron job.
I'm running my app via Gunicorn with the following config and controlled via supervisor:
[program:myapp]
command=/home/path/to/venv/bin/gunicorn -b localhost:8000 -w 4 myapp:app --preload
directory=/home/path/here
user=myuser
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
The job details are stored in my config.py:
...config stuff
JOBS = [
{
'id': 'sched_email',
'func': 'app.tasks:sched_email',
'trigger': 'cron',
'hour': 9,
},
]
SCHEDULER_API_ENABLED = True
Originally the email was being sent 4 times due to the 4 workers initialising the app and the scheduler. I found a similar article which suggested opening a socket when the app is initialised so the other workers can't grab the job.
My init.py:
# Third-party imports
import logging
from logging.handlers import SMTPHandler, RotatingFileHandler
import os
from flask import Flask
from flask_mail import Mail, Message
from flask_sqlalchemy import SQLAlchemy
from flask_apscheduler import APScheduler
from flask_migrate import Migrate
from flask_login import LoginManager
import sys, socket
# Local imports
from config import app_config
# Create initial instances of extensions
mail = Mail()
db = SQLAlchemy()
scheduler = APScheduler()
migrate = Migrate()
login_manager = LoginManager()
# Construct the Flask app instance
def create_app(config_name):
app = Flask(__name__)
app.config.from_object(app_config[config_name])
app_config[config_name].init_app(app)
migrate.init_app(app, db)
mail.init_app(app)
db.init_app(app)
# Fix to ensure only one Gunicorn worker grabs the scheduled task
try:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(("127.0.0.1", 47200))
except socket.error:
pass
else:
scheduler.init_app(app)
scheduler.start()
login_manager.init_app(app)
login_manager.login_message = "You must be logged in to access this page."
login_manager.login_message_category = 'danger'
login_manager.login_view = "admin.login"
# Initialize blueprints
from .errors import errors as errors_blueprint
app.register_blueprint(errors_blueprint)
from .main import main as main_blueprint
app.register_blueprint(main_blueprint)
from .admin import admin as admin_blueprint
app.register_blueprint(admin_blueprint)
# Setup logging when not in debug mode
if not app.debug:
... logging email config
... log file config
return app
Now the email is being sent twice!
Can anyone suggest why this is happening? Is there any logs i can dive into to work out what's going on?
I also read about using the #app.before_first_request decorator but as i'm using the app factory pattern i'm not sure how to incorporate this.
Thanks!

So it turns out the issue was a silly mistake i made.
I didn't configure supervisor correctly so my --preload flag was not actually being applied.
After i fixed supervisor and reloaded, my task is now running correctly and i'm receiving one email.
I hope my setup above will help others as being a beginner this took me a long time to get working.

Multithreading with Flask

I'd like to call generate_async_audio_service from a view and have it asynchronously generate audio files for the list of words using a threading pool and then commit them to a database.
I keep running into an error that I'm working out of the application context even though I'm creating a new polly and s3 instance each time.
How can I generate/upload multiple audio files at once?
from flask import current_app,
from multiprocessing.pool import ThreadPool
from Server.database import db
import boto3
import io
import uuid
def upload_audio_file_to_s3(file):
app = current_app._get_current_object()
with app.app_context():
s3 = boto3.client(service_name='s3',
aws_access_key_id=app.config.get('BOTO3_ACCESS_KEY'),
aws_secret_access_key=app.config.get('BOTO3_SECRET_KEY'))
extension = file.filename.rsplit('.', 1)[1].lower()
file.filename = f"{uuid.uuid4().hex}.{extension}"
s3.upload_fileobj(file,
app.config.get('S3_BUCKET'),
f"{app.config.get('UPLOADED_AUDIO_FOLDER')}/{file.filename}",
ExtraArgs={"ACL": 'public-read', "ContentType": file.content_type})
return file.filename
def generate_polly(voice_id, text):
app = current_app._get_current_object()
with app.app_context():
polly_client = boto3.Session(
aws_access_key_id=app.config.get('BOTO3_ACCESS_KEY'),
aws_secret_access_key=app.config.get('BOTO3_SECRET_KEY'),
region_name=app.config.get('AWS_REGION')).client('polly')
response = polly_client.synthesize_speech(VoiceId=voice_id,
OutputFormat='mp3', Text=text)
return response['AudioStream'].read()
def generate_polly_from_term(vocab_term, gender='m'):
app = current_app._get_current_object()
with app.app_context():
audio = generate_polly('Celine', vocab_term.term)
file = io.BytesIO(audio)
file.filename = 'temp.mp3'
file.content_type = 'mp3'
return vocab_term.id, upload_audio_file_to_s3(file)
def generate_async_audio_service(terms):
pool = ThreadPool(processes=12)
results = pool.map(generate_polly_from_term, terms)
# do something w/ results

This is not necessarily a fleshed-out answer, but rather than putting things into comments I'll put it here.
Celery is a task manager for python. The reason you would want to use this is if you have tasks pinging Flask, but they take longer to finish than the interval of tasks coming in, then certain tasks will be blocked and you won't get all of your results. To fix this, you hand it to another process. This goes like so:
1) Client sends a request to Flask to process audio files
2) The files land in Flask to be processed, Flask will send an asyncronous task to Celery.
3) Celery is notified of the task and stores its state in some sort of messaging system (RabbitMQ and Redis are the canonical examples)
4) Flask is now unburdened from that task and can receive more
5) Celery finishes the task, including the upload to your database
Celery and Flask are then two separate python processes communicating with one another. That should satisfy your multithreaded approach. You can also retrieve the state from a task through Flask if you want the client to verify that the task was/was not completed. The route in your Flask app.py would look like:
#app.route('/my-route', methods=['POST'])
def process_audio():
# Get your files and save to common temp storage
save_my_files(target_dir, files)
response = celery_app.send_tast('celery_worker.files', args=[target_dir])
return jsonify({'task_id': response.task_id})
Where celery_app comes from another module worker.py:
import os
from celery import Celery
env = os.environ
# This is for a rabbitMQ backend
CELERY_BROKER_URL = env.get('CELERY_BROKER_URL', 'amqp://0.0.0.0:5672/0')
CELERY_RESULT_BACKEND = env.get('CELERY_RESULT_BACKEND', 'rpc://')
celery_app = Celery('tasks', broker=CELERY_BROKER_URL, backend=CELERY_RESULT_BACKEND)
Then, your celery process would have a worker configured something like:
from celery import Celery
from celery.signals import after_task_publish
env = os.environ
CELERY_BROKER_URL = env.get('CELERY_BROKER_URL')
CELERY_RESULT_BACKEND = env.get('CELERY_RESULT_BACKEND', 'rpc://')
# Set celery_app with name 'tasks' using the above broker and backend
celery_app = Celery('tasks', broker=CELERY_BROKER_URL, backend=CELERY_RESULT_BACKEND)
#celery_app.task(name='celery_worker.files')
def async_files(path):
# Get file from path
# Process
# Upload to database
# This is just if you want to return an actual result, you can fill this in with whatever
return {'task_state': "FINISHED"}
This is relatively basic, but could serve as a starting point. I will say that some of Celery's behavior and setup is not always the most intuitive, but this will leave your flask app available to whoever wants to send files to it without blocking anything else.
Hopefully that's somewhat helpful

Flask and Celery on Heroku: sqlalchemy.exc.DatabaseError: (psycopg2.DatabaseError) SSL error: decryption failed or bad record mac

I'm trying to deploy a flask app on heroku that uses background tasks in Celery. I've implemented the application factory pattern so that the celery processes are not bound to any one instance of the flask app.
This works locally, and I have yet to see an error. But when deployed to heroku, the same results always occur: the celery task (I'm only using one) succeeds the first time it is run, but any subsequent celery calls to that task fail with sqlalchemy.exc.DatabaseError: (psycopg2.DatabaseError) SSL error: decryption failed or bad record mac. If I restart the celery worker, the cycle continues.
There are multiple issues that show this same error, but none specify a proper solution. I initially believed implementing the application factory pattern would have prevented this error from manifesting, but it's not quite there.
In app/__init__.py I create the celery and db objects:
celery = Celery(__name__, broker=Config.CELERY_BROKER_URL)
db = SQLAlchemy()
def create_app(config_name):
app = Flask(__name__)
app.config.from_object(config[config_name])
db.init_app(app)
return app
My flask_celery.py file creates the actual Flask app object:
import os
from app import celery, create_app
app = create_app(os.getenv('FLASK_CONFIG', 'default'))
app.app_context().push()
And I start celery with this command:
celery worker -A app.flask_celery.celery --loglevel=info
This is what the actual celery task looks like:
#celery.task()
def task_process_stuff(stuff_id):
stuff = Stuff.query.get(stuff_id)
stuff.processed = True
db.session.add(stuff)
db.session.commit()
return stuff
Which is invoked by:
task_process_stuff.apply_async(args=[stuff.id], countdown=10)
Library Versions
Flask 0.12.2
SQLAlchemy 1.1.11
Flask-SQLAlchemy 2.2
Celery 4.0.2

The solution was to add db.engine.dispose() at the beginning of the task, disposing of all db connections before any work begins:
#celery.task()
def task_process_stuff(stuff_id):
db.engine.dispose()
stuff = Stuff.query.get(stuff_id)
stuff.processed = True
db.session.commit()
return stuff
As I need this functionality across all of my tasks, I added it to task_prerun:
#task_prerun.connect
def on_task_init(*args, **kwargs):
db.engine.dispose()

Send emails with flask and outside context / cronjobs [duplicate]

This question already has answers here:
RuntimeError: working outside of application context
(4 answers)
Closed 6 years ago.
I have a project which is structured similar to the overholt and fbone example. I can send emails from all my blueprints fine, but struggle to send outside. E.g. from within a cron script or celery task.
I keep getting the error working outside of application context
app/factory.py
from flask import Flask
from .extensions import mail
def create_app(package_name, package_path, settings_override=None,
register_security_blueprint=True):
app = Flask(package_name, instance_relative_config=True)
mail.init_app(app)
register_blueprints(app, package_name, package_path)
app.wsgi_app = HTTPMethodOverrideMiddleware(app.wsgi_app)
return app
app/extensions.py
from flask_mail import Mail
mail = Mail()
app/frontend/admin.py
bp = Blueprint('admin', __name__, url_prefix='/admin', static_folder='static')
#bp.route('/')
def admin():
msg = Message(......)
mail.send(msg)
Everything up until here works fine. Now I have a separate module in app which has some cron scripts which now fail.
app/cron/alerts.py
from ..extensions import mail
from flask.ext.mail import Message
def alert():
msg = Message('asdfasdf', sender='hello#example.com', recipients=['hello#example.com'])
msg.body = 'hello'
mail.send(msg)
Which produces the error. How can I get around this?
raise RuntimeError('working outside of application context')
RuntimeError: working outside of application context

You need use Flask-Mail:
from flask_mail import Mail
mail = Mail(app)

I would recommend going with celery. For the context problem I have found my solution in the following.
Have a look at this:
Miguel Grinberg's Blogpost on celery
Or if you are using the factory pattern in your application:
Celery with Factory Pattern
The second one is some kind of further/extended reading.
Both of them helped me a lot. (The second one also solved a context issue for me)