We Keep Coding

Celery doesn't work when i import a module. Before importing it works fine

My celery task is to update the mongodb database. I've created a handler for it which i'm importing to celery.py. Celery seems to work fine but when i try to import the handler module it throws an error.
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet
celery.py:
from __future__ import absolute_import, unicode_literals
from django.conf import settings
import os
from celery import Celery
# from handlers.ordercount import OrderCount
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'di_idealsteel.settings')
app = Celery('di_idealsteel')
# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
app.config_from_object('django.conf:settings')
# Load task modules from all registered Django app configs.
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
#app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))
#app.task(bind=True)
def add(self,number,customer_id):
ordercount = OrderCount()
opencount = ordercount.open_enquiries(number,customer_id)
When i try to import 5th line it throws this error
Traceback (most recent call last):
File "c:\users\ashish\envs\idealvenv\lib\site-packages\celery\app\trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "c:\users\ashish\envs\idealvenv\lib\site-packages\celery\app\trace.py", line 437, in __protected_call__
return self.run(*args, **kwargs)
File "C:\Users\Ashish\Desktop\Internship\di_idealsteel\di_idealsteel\celery.py", line 32, in add
from handlers.ordercount import OrderCount
File "C:\Users\Ashish\Desktop\Internship\di_idealsteel\handlers\ordercount.py", line 4, in <module>
from handlers.user_handler import UserHandler
File "C:\Users\Ashish\Desktop\Internship\di_idealsteel\handlers\user_handler.py", line 12, in <module>
from app.models import (AuthUser,Customers,CustomerUsers,UserHistory,CustomerUsersHistory)
File "C:\Users\Ashish\Desktop\Internship\di_idealsteel\app\models.py", line 10, in <module>
from django.contrib.auth.models import User
File "c:\users\ashish\envs\idealvenv\lib\site-packages\django\contrib\auth\models.py", line 4, in <module>
from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
File "c:\users\ashish\envs\idealvenv\lib\site-packages\django\contrib\auth\base_user.py", line 49, in <module>
class AbstractBaseUser(models.Model):
File "c:\users\ashish\envs\idealvenv\lib\site-packages\django\db\models\base.py", line 94, in __new__
app_config = apps.get_containing_app_config(module)
File "c:\users\ashish\envs\idealvenv\lib\site-packages\django\apps\registry.py", line 239, in get_containing_app_config
self.check_apps_ready()
File "c:\users\ashish\envs\idealvenv\lib\site-packages\django\apps\registry.py", line 124, in check_apps_ready
raise AppRegistryNotReady("Apps aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.
Other files:
__init__.py:
from __future__ import absolute_import, unicode_literals
# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import app as celery_app
__all__ = ('celery_app',)
settings.py:
# CELERY STUFF
BROKER_URL = 'redis://localhost:6379'
CELERY_RESULT_BACKEND = 'redis://localhost:6379'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TIMEZONE = 'Africa/Nairobi'

This line in celery.py is causing the error -->
from handlers.ordercount import OrderCount
If you look at the error stacktrace, internally the OrderCount module/class is calling AbstractBaseUser model, from django.contrib.auth.models import User / AbstractBaseUser. And till now django hasn't had the chance to load all the apps inside its registry and you are trying to find the User model from the registry.
You can create a tasks.py file in your django app and register your tasks there.
You have to just remove the add celery task from celery.py file
Alternatively, you could also import OrderCount locally in the add function, to avoid the error.

Integrating Celery with Flask using the application factory pattern: maximum recursion depth error

I am working from the cookiecutter Flask template, which uses the application factory pattern. I had Celery working for tasks that did not use the application context, but one of my tasks does need to know it; it makes a database query and updates a database object. Right now I have not a circular import error (though I've had them with other attempts) but a maximum recursion depth error.
I consulted this blog post about how to use Celery with the application factory pattern, and I'm trying to follow this Stack Overflow answer closely, since it has a structure apparently also derived from cookiecutter Flask.
Relevant portions of my project structure:
cookiecutter_mbam
│ celeryconfig.py
│
└───cookiecutter_mbam
| __init__.py
│ app.py
│ run_celery.py
│
└───utility
| celery_utils.py
|
└───derivation
| tasks.py
|
└───storage
| tasks.py
|
└───xnat
tasks.py
__init__.py:
"""Main application package."""
from celery import Celery
celery = Celery('cookiecutter_mbam', config_source='cookiecutter_mbam.celeryconfig')
Relevant portion of app.py:
from cookiecutter_mbam import celery
def create_app(config_object='cookiecutter_mbam.settings'):
"""An application factory, as explained here: http://flask.pocoo.org/docs/patterns/appfactories/.
:param config_object: The configuration object to use.
"""
app = Flask(__name__.split('.')[0])
app.config.from_object(config_object)
init_celery(app, celery=celery)
register_extensions(app)
# ...
return app
run_celery.py:
from cookiecutter_mbam.app import create_app
from cookiecutter_mbam import celery
from cookiecutter_mbam.utility.celery_utils import init_celery
app = create_app(config_object='cookiecutter_mbam.settings')
init_celery(app, celery)
celeryconfig.py:
broker_url = 'redis://localhost:6379'
result_backend = 'redis://localhost:6379'
task_serializer = 'json'
result_serializer = 'json'
accept_content = ['json']
enable_utc = True
imports = {'cookiecutter_mbam.xnat.tasks', 'cookiecutter_mbam.storage.tasks', 'cookiecutter_mbam.derivation.tasks'}
Relevant portion of celery_utils.py:
def init_celery(app, celery):
"""Add flask app context to celery.Task"""
class ContextTask(celery.Task):
def __call__(self, *args, **kwargs):
with app.app_context():
return self.run(*args, **kwargs)
celery.Task = ContextTask
return celery
When I try to start the worker using celery -A cookiecutter_mbam.run_celery:celery worker I get a RecursionError: maximum recursion depth exceeded while calling a Python object error. (I also have tried several other ways to invoke the worker, all with the same error.) Here's an excerpt from the stack trace:
Traceback (most recent call last):
File "/Users/katie/anaconda/bin/celery", line 11, in <module>
sys.exit(main())
File "/Users/katie/anaconda/lib/python3.6/site-packages/celery/__main__.py", line 16, in main
_main()
File "/Users/katie/anaconda/lib/python3.6/site-packages/celery/bin/celery.py", line 322, in main
cmd.execute_from_commandline(argv)
File "/Users/katie/anaconda/lib/python3.6/site-packages/celery/bin/celery.py", line 496, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "/Users/katie/anaconda/lib/python3.6/site-packages/celery/bin/base.py", line 275, in execute_from_commandline
return self.handle_argv(self.prog_name, argv[1:])
File "/Users/katie/anaconda/lib/python3.6/site-packages/celery/bin/celery.py", line 488, in handle_argv
return self.execute(command, argv)
File "/Users/katie/anaconda/lib/python3.6/site-packages/celery/bin/celery.py", line 420, in execute
).run_from_argv(self.prog_name, argv[1:], command=argv[0])
File "/Users/katie/anaconda/lib/python3.6/site-packages/celery/bin/worker.py", line 221, in run_from_argv
*self.parse_options(prog_name, argv, command))
File "/Users/katie/anaconda/lib/python3.6/site-packages/celery/bin/base.py", line 398, in parse_options
self.parser = self.create_parser(prog_name, command)
File "/Users/katie/anaconda/lib/python3.6/site-packages/celery/bin/base.py", line 414, in create_parser
self.add_arguments(parser)
File "/Users/katie/anaconda/lib/python3.6/site-packages/celery/bin/worker.py", line 277, in add_arguments
default=conf.worker_state_db,
File "/Users/katie/anaconda/lib/python3.6/site-packages/celery/utils/collections.py", line 126, in __getattr__
return self[k]
File "/Users/katie/anaconda/lib/python3.6/site-packages/celery/utils/collections.py", line 429, in __getitem__
return getitem(k)
File "/Users/katie/anaconda/lib/python3.6/site-packages/celery/utils/collections.py", line 278, in __getitem__
return mapping[_key]
File "/Users/katie/anaconda/lib/python3.6/collections/__init__.py", line 989, in __getitem__
if key in self.data:
File "/Users/katie/anaconda/lib/python3.6/site-packages/celery/utils/collections.py", line 126, in __getattr__
return self[k]
File "/Users/katie/anaconda/lib/python3.6/collections/__init__.py", line 989, in __getitem__
if key in self.data:
File "/Users/katie/anaconda/lib/python3.6/site-packages/celery/utils/collections.py", line 126, in __getattr__
return self[k]
I understand the basic sense of this error -- something is calling itself infinitely. Maybe create_app. But I can't see why, and I don't know how to go about debugging this.
I'm also getting this when I try to load my site:
File "~/cookiecutter_mbam/cookiecutter_mbam/xnat/tasks.py", line 14, in <module>
#celery.task
AttributeError: module 'cookiecutter_mbam.celery' has no attribute 'task'
I did not have this problem when I was using the make_celery method described here, but that method creates circular import problems when you need your tasks to access the application context. Pointers on how to do this correctly with the Cookiecutter Flask template would be much appreciated.

I'm suspicious of that bit of code that's making the Flask app available to celery. It's skipping over some essential code by going directly to run(). (See https://github.com/celery/celery/blob/master/celery/app/task.py#L387)
Try calling the inherited __call__. Here's a snippet from one of my (working) apps.
# Arrange for tasks to have access to the Flask app
TaskBase = celery.Task
class ContextTask(TaskBase):
def __call__(self, *args, **kwargs):
with app.app_context():
return TaskBase.__call__(self, *args, **kwargs) ## << here
celery.Task = ContextTask
I also don't see where you're creating an instance of Celery and configuring it. I assume you have
celery = Celery(__name__)
and then need to
celery.config_from_object(...)
from somewhere within init_celery()

This is solved. I had my configcelery.py in the wrong place. I needed to move it to the package directory, not the parent repo directory. It is incredibly unintuitive/uninformative that a misplaced config file, rather than causing an "I can't find that file"-type error, causes an infinite recursion. But at least I finally saw it and corrected it.

Starting celery in flask: AttributeError: 'Flask' object has no attribute 'user_options'

I try to start a Celery worker server from a command line:
celery -A server application worker --loglevel=info
The code and folder path:
server.py
application/controllers/routes.py
server.py
app = Flask(__name__)
from application.controllers import routes
app.run(host='127.0.0.1',port=5051,debug=True)
route.py
from flask import Flask,
from celery import Celery
from server import app
app.config['CELERY_BROKER_URL'] = 'redis://localhost:6379/0'
app.config['CELERY_RESULT_BACKEND'] = 'redis://localhost:6379/0'
celery = Celery(app.name, broker=app.config['CELERY_BROKER_URL'])
celery.conf.update(app.config)
#celery.task()
def add_together(self, count):
return "First success"
#app.route("/queing")
def testsfunction():
count = 1
add_together.delay(count)
return "cool"
Trace back:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.7/bin/celery", line 11, in <module>
sys.exit(main())
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/celery/__main__.py", line 30, in main
main()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/celery/bin/celery.py", line 81, in main
cmd.execute_from_commandline(argv)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/celery/bin/celery.py", line 770, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/celery/bin/base.py", line 309, in execute_from_commandline
argv = self.setup_app_from_commandline(argv)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/celery/bin/base.py", line 477, in setup_app_from_commandline
user_preload = tuple(self.app.user_options['preload'] or ())
AttributeError: 'Flask' object has no attribute 'user_options'
I got this error when I'm running a celery worker in terminal.

just run the celery with this command instead of yours:
celery -A application.controllers.routes:celery worker --loglevel=info
this will solve your current problem however your codes have a plenty of mistakes for example if you want to have a self argument inside your add_together function you you should declare a task like this:
#celery.task(bind=True)

It seems like you have typo mistake:
def add_together(self, count):
to
def add_together(count):

'module' object has no attribute 'celery'

I've been reading through the answers to some previous questions as well as the celery docs and I just can't quite fix this issue. My task.py, celery.py, and settings.py are all contained within the same app RBWebfiles which is contained within the project called recruiting board. I'm attempting to creating a periodic-task that will run using celery beat's scheduler.
This is the portion of my settings.py that deals with celery:
from __future__ import absolute_import
import os
from datetime import timedelta
CELERY_IMPORTS = ('RBWebfiles.tasks')
CELERYBEAT_SCHEDULE = 'djcelery.schedulers.DatabaseScheduler'
CELERYBEAT_SCHEDULE = {
'schedule-name':{
'task': 'RBWebfiles.tasks.requestRefresher',
'schedule': timedelta(seconds = 30),
},
}
BROKER_URL = 'django://'
I also added 'djcelery' to my installed apps within that file
This is my tasks.py file:
from __future__ import absolute_import
import datetime
from celery.task.base import periodic_task
from student.models import StudentAccount
from celery import Celery
from celery.utils.log import get_task_logger
import os
celery = Celery('tasks', broker='django://')
logger = get_task_logger(__name__)
os.environ['DJANGO_SETTINGS_MODULE'] = 'RBWebfiles.settings'
#periodic_task(run_every=datetime.timedelta(seconds=30))
def requestRefresher(self):
logger.info("start task")
for s in StudentAccount.objects.all():
s.requestLimit = 0
s.save()
return None
and lastly this is my celery.py file:
from __future__ import absolute_import
import os
from celery import Celery
from django.conf import settings
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'conf.settings')
app = Celery('RBWebfiles.celery')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda : settings.INSTALLED_APPS)
app.conf.update(
CELERY_RESULT_BACKEND ='djcelery.backends.database:DatabaseBackend',
)
#app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))
I've tried uninstalling and reinstalling both celery and django-celery, I'm not sure if I just don't understand something or I'm making a massive error. I attempt to run it using the command : celery beat -A RBWebfiles
this is the traceback:
C:\Users\Lexie Infantino\PycharmProjects\recruitingboard>celery beat -A RBWebfiles
Traceback (most recent call last):
File "C:\Python34\lib\site-packages\celery\app\utils.py", line 235, in find_app
found = sym.app
AttributeError: 'module' object has no attribute 'app'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Python34\lib\runpy.py", line 170, in _run_module_as_main
"__main__", mod_spec)
File "C:\Python34\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Python34\Scripts\celery.exe\__main__.py", line 9, in <module>
File "C:\Python34\lib\site-packages\celery\__main__.py", line 30, in main
main()
File "C:\Python34\lib\site-packages\celery\bin\celery.py", line 81, in main
cmd.execute_from_commandline(argv)
File "C:\Python34\lib\site-packages\celery\bin\celery.py", line 769, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "C:\Python34\lib\site-packages\celery\bin\base.py", line 309, in execute_from_commandline
argv = self.setup_app_from_commandline(argv)
File "C:\Python34\lib\site-packages\celery\bin\base.py", line 469, in setup_app_from_commandline
self.app = self.find_app(app)
File "C:\Python34\lib\site-packages\celery\bin\base.py", line 489, in find_app
return find_app(app, symbol_by_name=self.symbol_by_name)
File "C:\Python34\lib\site-packages\celery\app\utils.py", line 240, in find_app
found = sym.celery
AttributeError: 'module' object has no attribute 'celery'

Make sure djcelery is in your INSTALLED_APPS and try starting celery beat with manage.py celerybeat instead.

AttributeError: 'Flask' object has no attribute 'user_options'

I am trying to setup this basic example from the following doc:
http://flask.pocoo.org/docs/patterns/celery/
But so far I keep getting the below error:
AttributeError: 'Flask' object has no attribute 'user_options'
I am using celery 3.1.15.
from celery import Celery
def make_celery(app):
celery = Celery(app.import_name, broker=app.config['CELERY_BROKER_URL'])
celery.conf.update(app.config)
TaskBase = celery.Task
class ContextTask(TaskBase):
abstract = True
def __call__(self, *args, **kwargs):
with app.app_context():
return TaskBase.__call__(self, *args, **kwargs)
celery.Task = ContextTask
return celery
Example:
from flask import Flask
app = Flask(__name__)
app.config.update(
CELERY_BROKER_URL='redis://localhost:6379',
CELERY_RESULT_BACKEND='redis://localhost:6379'
)
celery = make_celery(app)
#celery.task()
def add_together(a, b):
return a + b
Traceback error:
Traceback (most recent call last):
File "/usr/local/bin/celery", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/celery/__main__.py", line 30, in main
main()
File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 81, in main
cmd.execute_from_commandline(argv)
File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 769, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 305, in execute_from_commandline
argv = self.setup_app_from_commandline(argv)
File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 473, in setup_app_from_commandline
user_preload = tuple(self.app.user_options['preload'] or ())
AttributeError: 'Flask' object has no attribute 'user_options'

The Flask Celery Based Background Tasks page (http://flask.pocoo.org/docs/patterns/celery/) suggests this to start celery:
celery -A your_application worker
The your_application string has to point to your application’s package or module that creates the celery object.
Assuming the code resides in application.py, explicitly pointing to the celery object (not just the module name) avoided the error:
celery -A application.celery worker

This worked for me:
celery -A my_app_module_name.celery worker

rename app flask_app
It will work

like this:
celery -A your_application worker
where your_application stands:
your_application = Flask(\__name\__)
the python file name: your_application.py, it will work
By the way, celery v4 is unsupported in Windows