Cannot start Celery Worker (Kombu.asynchronous.timer) - python

I followed the first steps with Celery (Django) and trying to run a heavy process in the background. I have RabbitMQ server installed. However, when I try,
celery -A my_app worker -l info it throws the following error
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "c:\anaconda3\lib\site-packages\celery\concurrency\prefork.py", line
18, in <module>
from celery.concurrency.base import BasePool
File "c:\anaconda3\lib\site-packages\celery\concurrency\base.py", line 15,
in <module>
from celery.utils import timer2
File "c:\anaconda3\lib\site-packages\celery\utils\timer2.py", line 16, in
<module>
from kombu.asynchronous.timer import Entry
ModuleNotFoundError: No module named 'kombu.asynchronous.timer'
I've searched a lot, but can't seem to get it working. Any help will be highly appreciated. Thank you!

I had this issue with the default Celery installation from pip (3.1.26Post2). As mentionned above, I installed instead version 3.1.25, but Celery was still not working. Thus I explicitly installed the latest version:
pip install Celery==4.3
and everything is working now!

I landed here after I tried to install django-celery while reading celery 4.4 documentation, this package forces celery version to 3.1.26.post2, so I had to:
pip uninstall django-celery
pip uninstall celery && pip install celery # Uninstall 3.1 and install latest
As documentation clearly says:
Django is supported out of the box now so this document only contains a basic way to integrate Celery and Django. You’ll use the same API as non-Django users so you’re recommended to read the First Steps with Celery tutorial first and come back to this tutorial.

TL;DR: remove the kombu directory from the root of your virtualenv (if it exists). It may only fail on Windows.
It seems to be a quirk. I found the same error and I checked out what was happening.
The wheel package that pip downloads looks fine (kombu.asynchronous.timer exists in it). The release for the last version (currently 4.2.0) also is fine. What was strange is what I found in my virtualenv installation.
I found a kombu directory at my virtualenv root which has the content of the library but it also has an "async" directory, alongside an "asynchronous" one. These directories aren't from the 4.2.0 release, as async has the timer.py file but asynchronous doesn't.
From where did it come? It appears that from the wheel's data directory.
So, the solution: I removed the kombu directory from the root of my virtualenv and celery worked.

I have the same problem, but solved it when reinstall celery with version 3.1.25
pip uninstall celery && pip install celery==3.1.25
Maybe because windows is not officially supported by celery 4, https://github.com/celery/celery/issues/3551

I tested celery on the same python version you have and it is okay. and also https://github.com/celery/kombu/blob/master/kombu/asynchronous/timer.py shows that renaming things randomly is not going to help you. Maybe you should try pip uninstall kombu && pip --no-cache-dir install -U kombu to perform a fresh install for kombu. I guess there must be something wrong with your installation. so if the kombu reinstall thing didn't work, try installing the whole thing again.

I just started with Celery.
I followed instructions and installed Celery v 4.2.0
when I was trying to run the command :
celery -A mysite worker -l info
I got the error :
ModuleNotFoundError: No module named 'kombu.asynchronous.timer
I removed Celery installation : pip uninstall celery
Afterwards installed Celery 3.1.25 as 'chuhy' recommended
but..It had some other issues, so I immediately un-installed 3.1.25 , and reinstalled celery v4.2.0 .
After this scenario the error didn't pop again.

I have faced similar type of issue this is because of older version of celery. Uninstall celery (pip uninstall celery) and install again (pip install Celery==4.3) and kaboom it will work.

So I am running windows 10, python3.9.x, using aws sqs as the broker, just finished updating some files:
settings.py
###
### For celery tasks!
###
from kombu.utils.url import safequote
import urllib.parse
AWS_ACCESS_KEY_ID = 'my aws_access_key_id for a user'
AWS_SECRET_ACCESS_KEY = 'my aws_secret_access_key for a user'
BROKER_URL = 'sqs://%s:%s#' % (urllib.parse.quote(AWS_ACCESS_KEY_ID, safe=''), urllib.parse.quote(AWS_SECRET_ACCESS_KEY, safe=''))
BROKER_TRANSPORT = 'sqs'
BROKER_TRANSPORT_OPTIONS = {
'canves-celery-queue': {
'access_key_id': safequote(AWS_ACCESS_KEY_ID),
'secret_access_key': safequote(AWS_SECRET_ACCESS_KEY),
'region': 'us-east-1'
}
}
CELERY_DEFAULT_QUEUE = 'celery<-project-queue>'
CELERY_QUEUES = {
CELERY_DEFAULT_QUEUE: {
'exchange': CELERY_DEFAULT_QUEUE,
'binding_key': CELERY_DEFAULT_QUEUE,
}
}
###
### End celery tasks
###
celery_tasks.py (referred to in the tutorial as celery.py - renamed because apparently that caused some other programmers some errors):
from __future__ import absolute_import
import os
from celery import Celery
# Set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', '<project>.settings')
# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
# should have a `CELERY_` prefix.
app = Celery('<project>', include=['<project>.tasks'])
app.config_from_object('django.conf:settings')
# Load task modules from all registered Django apps.
app.autodiscover_tasks()
#app.task(bind=True)
def debug_task(self):
print("debug_task was fired!")
print(f'Request: {self.request!r}')
if __name__ == '__main__':
app.start()
tasks.py (this is in the same directory as the settings.py - also referrenced in celery_tasks.py)
from __future__ import absolute_import
from .celery_tasks import app
import time
#app.task(ignore_result=True)
def sleep(x, y):
print("Sleeping for: " + str(x + y))
time.sleep(x + y)
print("Slept for: " + str(x + y))
When I went to run the worker (make sure you are in the same directory as manage.py), it threw this error:
from kombu.async.timer import Entry, Timer as Schedule, to_timestamp, logger
to fix it, I ran as per gogaz's answer
pip uninstall django-celery
pip uninstall celery && pip install celery
which pushed me to the latest version of celery, 4.3... celery 4+ isn't supported on windows as per this SO question (Celery raises ValueError: not enough values to unpack), which conveniently has this answer (posted by Samuel Chen):
for celery 4.2+, python3, windows 10
pip install gevent
celery -A <project> worker -l info -P gevent
for celery 4.1+, python3, windows 10
pip install eventlet
celery -A <project> worker -l info -P eventlet
The only other error I get is from django's debugger being on, which apparently causes memory leaks...
The problem (for me at least) is that I can't use the Prefork pool, which means that I can't use app.control.revoke() to terminate tasks.
---EDIT---
Also worth mentioning that after this answer was posted, I switched to a linux box. Unknown to me, due to a lack of experience, there are different modes you can run background tasks in. I don't remember all the names, but if you type into google "celery multithreading vs gevent", it will likely come back with some other modes you can run celery in, their purposes and which ones are supported for each platform. Windows couldn't run the mode that I thought made the most sense for my problem (I believe it was multithreading), and that was a real issue. However linux can run all of them... so I switched back to linux, just for celery. I had some problems with DJango in a redhat environment, so I had to fix those issues as well :|

I work on Windows so I had a bit of problem with this.
But my solution was to create new conda env with python 3.6.8 ( as i have understand celery may work on python 3.7 but have lot of problems).
Then proceed to install latest versions of celery(4.3.0) and Django(2.2.3) and after that everything worked fine.

Related

Django Black Not Installing Properly in Pipenv environment

I received an error after installing Django Black. I ran pipenv install black --pre. Then when I ran manage.py runserver, I received the error below. Note that I am running Windows 10.
from custom_storages import MediaStorage
File "C:\Users\dgold2\Documents\py\ibankai\src\custom_storages.py", line 2, in <module>
from storages.backends.s3boto3 import S3Boto3Storage
File "C:\Users\dgold2\Documents\py\ibankai\src\.venv\lib\site-packages\storages\backends\s3boto3.py", line 18, in <module>
from django.utils.six.moves.urllib import parse as urlparse
ModuleNotFoundError: No module named 'django.utils.six'
I ran into this when I was using a template for Django + Vue on Heroku, the only difference being that in my case the error was thrown by whitenoise, a static files middleware, rather than s3boto3. The root cause is that my Pipfile specified Django = "*" but whitenoise = "==4.0". So pipenv was grabbing the most recent available Django, which was 3.0, but whitenoise 4 was trying to import from django.utils.six (like your s3boto3 package), which seems to have been deprecated in Django 3. (Compare 2.2, 3.0).
See if there's a s3boto3 version specifier in your Pipfile that you can update, or alternatively specify Django ="2.2" if you're willing to hold off on Django 3.

ModuleNotFoundError: No module named 'flask_session'

I have a simple python file that I am trying to set up to utilize sessions, when i run the file I am receiving the error below:
ModuleNotFoundError: No module named 'flask_session'
I believe I am importing the module properly, is there anything else that I can check to set this up properly?
from flask import Flask, render_template, request, session
from flask_session import Session
app = Flask(__name__)
app.config["SESSION_PERMANENT"] = False
app.config["SESSION_TYPE"] = "filesystem"
Session(app)
#app.route("/", methods=["GET", "POST"])
def index():
if session.get("notes") is None:
session["notes"] = []
if request.method == "POST":
note = request.form.get("note")
session["notes"].append(note)
return render_template("index.html", notes=notes)
Here is the traceback ( most recent call last )
File "c:\python37\lib\site-packages\flask\cli.py", line 325, in __call__
Open an interactive python shell in this frameself._flush_bg_loading_exception()
File "c:\python37\lib\site-packages\flask\cli.py", line 313, in _flush_bg_loading_exception
reraise(*exc_info)
File "c:\python37\lib\site-packages\flask\_compat.py", line 35, in reraise
raise value
File "c:\python37\lib\site-packages\flask\cli.py", line 302, in _load_app
self._load_unlocked()
File "c:\python37\lib\site-packages\flask\cli.py", line 317, in _load_unlocked
self._app = rv = self.loader()
File "c:\python37\lib\site-packages\flask\cli.py", line 372, in load_app
app = locate_app(self, import_name, name)
File "c:\python37\lib\site-packages\flask\cli.py", line 242, in locate_app
'\n\n{tb}'.format(name=module_name, tb=traceback.format_exc())
I've first encountered this problem while doing Harvard's CS50x class. I'm using Linux and I've been using Python3, as it turns out I've installed Flask-Session for Python2.
I don't know whether it also applies to Mac but I used the following instead:
$ pip3 install flask-session
Then you can check whether it's installed with pip's freeze command. After doing this my pylint in VSCode no longer gave an error.
I got the flask_session module not found a number of times and it took me 2 hours to find a fix after thousands of attempts. Well, I too visited a number of sites looking for a solution but didn't get any. I got this error while I was learning the online course cs50 web development on edx.org by Harvard and when I read the forum discussion on reddit I saw comments where they mentioned Harvard used a special server to make it work and I felt the pain of not going to Harvard but two hours later I'm screaming because I just found a solution that's no where on the internet.
You should not use from flask_session import Session
Instead use from flask_session.__init__ import Session
I cannot explain the terminology as to why we do this here but you can private message me if you want to understand why we do this. Happy Coding!
Be sure to have the extension installed:
pip install Flask-Session
oficial page with instructions
If you get the error "ModuleNotFoundError: No module named 'werkzeug.contrib'":
pip install werkzeug==0.16.0
The latest version is the 1.0.1 but only the 0.16.0 worked.
(where I found this solution) (official page)
simply goto terminal and type pip install flask_session , wait for installation, once done you are good to go. ( Also many other solutions do not work, look for red highlighted part in image. Once installed it works properly.)
I ran into the same error.
The following worked for me.
Install the extension with the following command:
$ easy_install Flask-Session
or alternatively, if you have pip installed:
$ pip install Flask-Session
I got it for windows:
There's no the library "flask_session" (--is a directory--) in your directory ...venv\Lib\site-packages (virtual environment)
You just need copy it from C:\Users\your _user\AppData\Local\Programs\Python\Python37-32\Lib\site-packages
and then paste it in venv\Lib\site-packages or where you have installed your virtual environment. Thats it and sorry for my english, greetings from Medellin Colombia.
Try to do following command instead of all go to terminal and type
$ easy_install Flask-Session
If it shows an error,use sudo before it , If it shows any error please let me know also feel free to upvote
Install the extension first using the following command $ pip install Flask-Session. You can read more on flask-session via https://pythonhosted.org/Flask-Session/
For completeness, I will share my solution:
In my case installed everything with pip3 to make it work with python 3.7, only worked with vscode after uninstalling python 2 from the system and reopening vscode.
Are you using this library? https://pythonhosted.org/Flask-Session/
The docs suggest importing it this way: from flask.ext.session import Session

Forcing Celery to use Python 3

I'm sorry if this is a silly question but this is my first day using Celery and I am having a difficult time getting it to use python3. I've included this shebang in all of my files...
#!/usr/bin/env python3
The stack trace is reporting errors raise by python2.7...
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
R = retval = fun(*args, **kwargs)
The app was run with this command...
celery -A GidConsumer worker --loglevel=info
Typically in a Python project, you'll install all dependencies (including celery) into a virtualenv. Then, when you want to run celery, you'll activate your virtual environment and it will find the celery executable in your PATH. There's a lot of detail not included here, but in general, it's a bad idea to try to globally install stuff and it leads to lots of hassles.
The alternative in this case is to find the celery package that you globally installed under Python3 (did you use pip3 to install it?). Figure out which bin that celery landed in and you can use its full path to call it by including its full path in that bin.
Please uninstall celery with pip(pip uninstall celery) and reinstall celery with PIP 3(pip3 install celery).... :)

Celery use wrong python version

I have a Django project, and I want to use Celery. I've installed Celery for python3, and then I run this command : sudo celery -A myApp worker -l info
But in the log, I see that it's Celery of python2.7 which is used : File "/Library/Python/2.7/site-packages
Any idea how for me to use the Celery installed for python3 ?
Ok, thanks to #Wayne I've found the solution.
First, use this command to see where the head of celery is : head -n 10 /usr/local/bin/celery
For myself, this is what I get :
#!/usr/bin/python
# EASY-INSTALL-ENTRY-SCRIPT: 'celery==3.1.23','console_scripts','celery'
__requires__ = 'celery==3.1.23'
import sys
from pkg_resources import load_entry_point
if __name__ == '__main__':
sys.exit(
load_entry_point('celery==3.1.23', 'console_scripts', 'celery')()
)
I see that the first shebang (#!/usr/bin/python) use the wrong python version.
Then, I've changed the first shebang : #!/usr/bin/env python3 and save the file. Now celery point to python3.

How can I use Django with MySQL in MAMP stack?

I have difficulty especially in installing MySQLdb module (MySQL-python-1.2.3c1), to connect to the MySQL in MAMP stack.
I've done a number of things such as copying the mysql include directory and library (including plugin) from a fresh installation of mysql (version 5.1.47) to the one inside MAMP (version 5.1.37).
Now, the MySQLdb module build and install doesnt give me error.
The error happens when I'm calling 'import MySQLdb' from python shell (version 2.6).
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "build/bdist.macosx-10.6-universal/egg/MySQLdb/__init__.py", line 19, in <module>
File "build/bdist.macosx-10.6-universal/egg/_mysql.py", line 7, in <module>
File "build/bdist.macosx-10.6-universal/egg/_mysql.py", line 6, in __bootstrap__
ImportError: dlopen(/Users/rhenru/.python-eggs/MySQL_python-1.2.3c1-py2.6-macosx-10.6-universal.egg-tmp/_mysql.so, 2): Symbol not found: _mysql_affected_rows
Referenced from: /Users/rhenru/.python-eggs/MySQL_python-1.2.3c1-py2.6-macosx-10.6-universal.egg-tmp/_mysql.so
Expected in: flat namespace
in /Users/rhenru/.python-eggs/MySQL_python-1.2.3c1-py2.6-macosx-10.6-universal.egg-tmp/_mysql.so
Any idea, what else do I need to do to make it works?
Thanks a bunch,
Robert
=========
Add the system response after using virtualenv as suggested by Hank Gay below...
(MyDjangoProject)MyMacPro:MyDjangoProject rhenru$ which python
/Users/rhenru/Workspace/django/MyDjangoProject/bin/python
After I run python in virtualenv, importing MySQLdb:
>>> import MySQLdb
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "build/bdist.macosx-10.6-universal/egg/MySQLdb/__init__.py", line 19, in <module>
File "build/bdist.macosx-10.6-universal/egg/_mysql.py", line 7, in <module>
File "build/bdist.macosx-10.6-universal/egg/_mysql.py", line 6, in __bootstrap__
ImportError: dlopen(/Users/rhenru/.python-eggs/MySQL_python-1.2.3c1-py2.6-macosx-10.6-universal.egg-tmp/_mysql.so, 2): Symbol not found: _mysql_affected_rows
Referenced from: /Users/rhenru/.python-eggs/MySQL_python-1.2.3c1-py2.6-macosx-10.6-universal.egg-tmp/_mysql.so
Expected in: flat namespace
in /Users/rhenru/.python-eggs/MySQL_python-1.2.3c1-py2.6-macosx-10.6-universal.egg-tmp/_mysql.so
import sys and sys.path
>>> import sys
>>> print sys.path
['', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/site-packages/pip-0.7.1-py2.6.egg', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python26.zip', '/Library/Python/2.6/site-packages/PyXML-0.8.4-py2.6-macosx-10.6-universal.egg', '/Library/Python/2.6/site-packages/pydot-1.0.2-py2.6.egg', '/Library/Python/2.6/site-packages/pyparsing-1.5.2-py2.6.egg', '/Library/Python/2.6/site-packages/vobject-0.8.1c-py2.6.egg', '/Library/Python/2.6/site-packages/pytz-2010h-py2.6.egg', '/Library/Python/2.6/site-packages/MySQL_python-1.2.3c1-py2.6-macosx-10.6-universal.egg', '/Library/Python/2.6/site-packages/distribute-0.6.12-py2.6.egg', '/Library/Python/2.6/site-packages/pip-0.7.1-py2.6.egg', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/plat-darwin', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/plat-mac', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/plat-mac/lib-scriptpackages', '/Users/rhenru/Workspace/django/MyDjangoProject/Extras/lib/python', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/lib-tk', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/lib-old', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/lib-dynload', '/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6', '/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-darwin', '/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-tk', '/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac', '/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac/lib-scriptpackages', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/site-packages', '/Library/Python/2.6/site-packages', '/Library/Python/2.6/site-packages/PIL', '/Library/Python/2.6/site-packages/setuptools-0.6c11-py2.6.egg-info', '/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python', '/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/PyObjC', '/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/wx-2.8-mac-unicode']
How are you installing MySQL-Python? I just tested in a fresh virtualenv and pip install mysql-python seems to have done the trick.
UPDATE:
pip is sort of like a package manager for Python packages.
By default, pip installs to your current site-packages directory, which is on your $PYTHONPATH. This lets other libraries/applications (like Django) access it. pip also works well with virtualenv (it should; Ian Bicking wrote them both), which is a nifty library that lets you sandbox an application. This is nice because it means you can try out new things without polluting (or even needing write access to) the global site-packages directory.
It probably seems like yak-shaving right now, but I'd say it's worth the effort to get up to speed on pip and virtualenv (you may also want to look into virtualenvwrapper, but we'll skip that for now; it's just sugar for virtualenv). It will lead to a slightly more complicated deployment scenario than putting everything in the global site-packages, but for development it's really no harder, and there are lots of good guides to deploying using a virtualenv.
I'd recommend something like the following:
curl -0 http://python-distribute.org/distribute_setup.py
python distribute_setup.py
easy_install pip
pip install virtualenv
virtualenv --distribute MyDjangoProject --no-site-packages
cd MyDjangoProject
source bin/activate (this activates the sandbox that virtualenv created)
pip install django mysql-python
At this point, you should have a totally functional Django+MySQL install (if I missed any steps, just comment and I'll try to add it in). You can start your Django project like this: django-admin.py startproject MyDjangoProject. cd into your project's directory, edit your settings.py file to point to your MySQL database, and run the dev server to test it out like so: ./manage.py runserver (you may need to chmod u+x your manage.py file). Voila! You should be able to access your site on localhost:8000. When you're done working on the project, you can just use deactivate to exit the virtualenv sandbox.
Try not to hold all this against Django: a lot of it is just best practices stuff for working with Python libraries. You could get by with a lot less, but this way it's more reproducible and you're less likely to accidentally mess up one of this project's dependencies when working on a different project.
I had this problem and it turned out to be due to an errant configuration:
export VERSIONER_PYTHON_PREFER_32_BIT=yes
I can't recall what I had this enabled for (some package that required 32-bit), probably related to Google AppEngine. But Setting it to 'no' solved by issues.
Otherwise I just installed everything using homebrew and pip.

Categories