We Keep Coding

Celery outside of flask application context

Getting the following error running a celery task, even with a Flask application context:
raised unexpected: RuntimeError('Working outside of application context.\n\nThis typically means that you attempted to use functionality that needed\nto interface with the current application object in some way. To solve\nthis, set up an application context with app.app_context(). See the\ndocumentation for more information.',)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/celery/app/trace.py", line 382, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/celery/app/trace.py", line 641, in __protected_call__
return self.run(*args, **kwargs)
File "/app/example.py", line 172, in start_push_task
}, data=data)
File "/app/push.py", line 65, in push
if user and not g.get('in_celery_task') and 'got_user' not in g:
File "/usr/lib/python3.6/site-packages/werkzeug/local.py", line 347, in __getattr__
return getattr(self._get_current_object(), name)
File "/usr/lib/python3.6/site-packages/werkzeug/local.py", line 306, in _get_current_object
return self.__local()
File "/usr/lib/python3.6/site-packages/flask/globals.py", line 44, in _lookup_app_object
raise RuntimeError(_app_ctx_err_msg)
RuntimeError: Working outside of application context.
This typically means that you attempted to use functionality that needed
to interface with the current application object in some way. To solve
this, set up an application context with app.app_context(). See the
documentation for more information.
Any way to fix this?

For me, the issue was that I had import celery instead of from app import celery.
Here's some more of my setup code for anyone who stumbles across here in the future:
app.py
def make_celery(app):
app.config['broker_url'] = 'amqp://rabbitmq:rabbitmq#rabbit:5672/'
app.config['result_backend'] = 'rpc://rabbitmq:rabbitmq#rabbit:5672/'
celery = Celery(app.import_name, backend=app.config['result_backend'], broker=app.config['broker_url'])
celery.conf.update(app.config)
class ContextTask(Task):
abstract = True
def __call__(self, *args, **kwargs):
with app.test_request_context():
g.in_celery_task = True
res = self.run(*args, **kwargs)
return res
celery.Task = ContextTask
celery.config_from_object(__name__)
celery.conf.timezone = 'UTC'
return celery
celery = make_celery(app)
In the other file:
from app import celery

Starting celery in flask: AttributeError: 'Flask' object has no attribute 'user_options'

I try to start a Celery worker server from a command line:
celery -A server application worker --loglevel=info
The code and folder path:
server.py
application/controllers/routes.py
server.py
app = Flask(__name__)
from application.controllers import routes
app.run(host='127.0.0.1',port=5051,debug=True)
route.py
from flask import Flask,
from celery import Celery
from server import app
app.config['CELERY_BROKER_URL'] = 'redis://localhost:6379/0'
app.config['CELERY_RESULT_BACKEND'] = 'redis://localhost:6379/0'
celery = Celery(app.name, broker=app.config['CELERY_BROKER_URL'])
celery.conf.update(app.config)
#celery.task()
def add_together(self, count):
return "First success"
#app.route("/queing")
def testsfunction():
count = 1
add_together.delay(count)
return "cool"
Trace back:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.7/bin/celery", line 11, in <module>
sys.exit(main())
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/celery/__main__.py", line 30, in main
main()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/celery/bin/celery.py", line 81, in main
cmd.execute_from_commandline(argv)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/celery/bin/celery.py", line 770, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/celery/bin/base.py", line 309, in execute_from_commandline
argv = self.setup_app_from_commandline(argv)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/celery/bin/base.py", line 477, in setup_app_from_commandline
user_preload = tuple(self.app.user_options['preload'] or ())
AttributeError: 'Flask' object has no attribute 'user_options'
I got this error when I'm running a celery worker in terminal.

just run the celery with this command instead of yours:
celery -A application.controllers.routes:celery worker --loglevel=info
this will solve your current problem however your codes have a plenty of mistakes for example if you want to have a self argument inside your add_together function you you should declare a task like this:
#celery.task(bind=True)

It seems like you have typo mistake:
def add_together(self, count):
to
def add_together(count):

Using multiprocessing pool from celery task raises exception

FOR THOSE READING THIS: I have decided to use RQ instead which doesn't fail when running code that uses the multiprocessing module. I suggest you use that.
I am trying to use a multiprocessing pool from within a celery task using Python 3 and redis as the broker (running it on a Mac). However, I don't seem to be able to even create a multiprocessing Pool object from within the Celery task! Instead, I get a strange exception that I really don't know what to do with.
Can anyone tell me how to accomplish this?
The task:
from celery import Celery
from multiprocessing.pool import Pool
app = Celery('tasks', backend='redis', broker='redis://localhost:6379/0')
#app.task
def test_pool():
with Pool() as pool:
# perform some task using the pool
pool.close()
return 'Done!'
which I add to Celery using:
celery -A tasks worker --loglevel=info
and then running it via the following python script:
import tasks
tasks.test_pool.delay()
that returns the following celery output:
[2015-01-12 15:08:57,571: INFO/MainProcess] Connected to redis://localhost:6379/0
[2015-01-12 15:08:57,583: INFO/MainProcess] mingle: searching for neighbors
[2015-01-12 15:08:58,588: INFO/MainProcess] mingle: all alone
[2015-01-12 15:08:58,598: WARNING/MainProcess] celery#Simons-MacBook-Pro.local ready.
[2015-01-12 15:09:02,425: INFO/MainProcess] Received task: tasks.test_pool[38cab553-3a01-4512-8f94-174743b05369]
[2015-01-12 15:09:02,436: ERROR/MainProcess] Task tasks.test_pool[38cab553-3a01-4512-8f94-174743b05369] raised unexpected: AttributeError("'Worker' object has no attribute '_config'",)
Traceback (most recent call last):
File "/usr/local/lib/python3.4/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/local/lib/python3.4/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
File "/Users/simongray/Code/etilbudsavis/offer-sniffer/tasks.py", line 17, in test_pool
with Pool() as pool:
File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/pool.py", line 150, in __init__
self._setup_queues()
File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/pool.py", line 243, in _setup_queues
self._inqueue = self._ctx.SimpleQueue()
File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/context.py", line 111, in SimpleQueue
return SimpleQueue(ctx=self.get_context())
File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/queues.py", line 336, in __init__
self._rlock = ctx.Lock()
File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/context.py", line 66, in Lock
return Lock(ctx=self.get_context())
File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/synchronize.py", line 163, in __init__
SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/synchronize.py", line 59, in __init__
kind, value, maxvalue, self._make_name(),
File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/synchronize.py", line 117, in _make_name
return '%s-%s' % (process.current_process()._config['semprefix'],
AttributeError: 'Worker' object has no attribute '_config'

This is a known issue with celery. It stems from an issue introduced in the billiard dependency. A work-around is to manually set the _config attribute for the current process. Thanks to user #martinth for the work-around below.
from celery.signals import worker_process_init
from multiprocessing import current_process
#worker_process_init.connect
def fix_multiprocessing(**kwargs):
try:
current_process()._config
except AttributeError:
current_process()._config = {'semprefix': '/mp'}
The worker_process_init hook will execute the code upon worker process initialization. We simply check to see if _config exists, and set it if it does not.

Via a useful comment in the Celery issue report linked to in Davy's comment, I was able to solve this by importing the billiard module's Pool class instead.
Replace
from multiprocessing import Pool
with
from billiard.pool import Pool

A quick solution is to use the thread-based "dummy" multiprocessing implementation. Change
from multiprocessing import Pool # or whatever you're using
to
from multiprocessing.dummy import Pool
However since this parallelism is thread-based, the usual caveats (GIL) apply.

AttributeError: 'Flask' object has no attribute 'user_options'

I am trying to setup this basic example from the following doc:
http://flask.pocoo.org/docs/patterns/celery/
But so far I keep getting the below error:
AttributeError: 'Flask' object has no attribute 'user_options'
I am using celery 3.1.15.
from celery import Celery
def make_celery(app):
celery = Celery(app.import_name, broker=app.config['CELERY_BROKER_URL'])
celery.conf.update(app.config)
TaskBase = celery.Task
class ContextTask(TaskBase):
abstract = True
def __call__(self, *args, **kwargs):
with app.app_context():
return TaskBase.__call__(self, *args, **kwargs)
celery.Task = ContextTask
return celery
Example:
from flask import Flask
app = Flask(__name__)
app.config.update(
CELERY_BROKER_URL='redis://localhost:6379',
CELERY_RESULT_BACKEND='redis://localhost:6379'
)
celery = make_celery(app)
#celery.task()
def add_together(a, b):
return a + b
Traceback error:
Traceback (most recent call last):
File "/usr/local/bin/celery", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/celery/__main__.py", line 30, in main
main()
File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 81, in main
cmd.execute_from_commandline(argv)
File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 769, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 305, in execute_from_commandline
argv = self.setup_app_from_commandline(argv)
File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 473, in setup_app_from_commandline
user_preload = tuple(self.app.user_options['preload'] or ())
AttributeError: 'Flask' object has no attribute 'user_options'

The Flask Celery Based Background Tasks page (http://flask.pocoo.org/docs/patterns/celery/) suggests this to start celery:
celery -A your_application worker
The your_application string has to point to your application’s package or module that creates the celery object.
Assuming the code resides in application.py, explicitly pointing to the celery object (not just the module name) avoided the error:
celery -A application.celery worker

This worked for me:
celery -A my_app_module_name.celery worker

rename app flask_app
It will work

like this:
celery -A your_application worker
where your_application stands:
your_application = Flask(\__name\__)
the python file name: your_application.py, it will work
By the way, celery v4 is unsupported in Windows

Fail to run Celery tutorial

Here is the tasks.py
from celery.task import task
#task
def add(x, y):
return x + y
Here is the celeryconfig.py
print 'importing ' + __file__
BROKER_URL = "amqp://guest:guest#localhost:5672//"
CELERY_RESULT_BACKEND = "amqp"
CELERY_IMPORTS = ("tasks", )
Here is the file that i run.
tasks.py:
from tasks import add
result = add.delay(4, 4)
print result.wait()
The program just stuck in the wait method.
Celeryd prints the following error:
Did you remember to import the module containing this task?
Or maybe you are using relative imports?
Please see http://bit.ly/gLye1c for more information.
The full contents of the message body was:
{'retries': 0, 'task': 'tasks.add', 'args': (4, 4), 'expires': None, 'eta': None
, 'kwargs': {}, 'id': '8c973638-4a87-4afa-8a78-958153066215'}
Traceback (most recent call last):
File "d:\python26\lib\site-packages\celery-2.4.5-py2.6.egg\celery\worker\consu
mer.py", line 427, in receive_message
eventer=self.event_dispatcher)
File "d:\python26\lib\site-packages\celery-2.4.5-py2.6.egg\celery\worker\job.p
y", line 297, in from_message
on_ack=on_ack, delivery_info=delivery_info, **kw)
File "d:\python26\lib\site-packages\celery-2.4.5-py2.6.egg\celery\worker\job.p
y", line 261, in __init__
self.task = registry.tasks[self.task_name]
File "d:\python26\lib\site-packages\celery-2.4.5-py2.6.egg\celery\registry.py"
, line 66, in __getitem__
raise self.NotRegistered(key)
NotRegistered: 'tasks.add'
When I run celeryd.py status, I the tasks.add is not here.
D:\Python26\Lib\site-packages\celery-2.4.5-py2.6.egg\celery\bin>celeryctl.py in
pect registered
<- registered
-> onfirenbpc: OK
* celery.backend_cleanup
* celery.chord
* celery.chord_unlock
* celery.ping
I run on windows, and linux as well. There are the same problem.
Does anyone know why?

Did you set you add method as task? One of variant to do that would use decorator:
from celery.decorators import task
#task
def add():
pass