We Keep Coding

Dockerized Django tests only look at first $PATH location

I'm running a Django test suite in a Docker container, and some of these tests use a program I've had to apt-get install (wkhtmltopdf). Now I can see that it's been installed correctly:
$ wkhtmltopdf --version
wkhtmltopdf 0.12.5
but for some reason the Django test can't use it. The installation location is definitely on my $PATH (third to last element):
$ echo $PATH
/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
$ find / -name wkhtmltopdf
/usr/bin/wkhtmltopdf
However when I run tests I get a stack trace ending in:
OSError: No wkhtmltopdf executable found: "/usr/local/bin/wkhtmltopdf"
If this file exists please check that this process can read it. Otherwise please install wkhtmltopdf - https://github.com/JazzCore/python-pdfkit/wiki/Installing-wkhtmltopdf
Now it's absolutely correct that there is no /usr/local/bin/wkhtmltopdf, because it got installed elsewhere (/usr/bin/) but both those locations are on $PATH.
I've tried moving /usr/bin/ to the start of $PATH, but I then get the error:
Traceback (most recent call last):
File "./manage.py", line 8, in <module>
from django.core.management import execute_from_command_line
ImportError: No module named django.core.management
presumably because it's now looking in /usr/bin/ for django when it's actually in usr/local/bin/, which is no longer the first location on $PATH.
I'm not sure if the problem is a Docker one, a Django one, a testing one, or just me misunderstanding one or more things going on here.

So it turns out that the project is using pdfkit as a wrapper for wkhtmltopdf, which includes setting the wkhtmltopdf path directly:
config = pdfkit.configuration(wkhtmltopdf=settings.WKHTMLTOPDF_BIN)
which after a quick look at the Django settings file was set to:
WKHTMLTOPDF_BIN = '/usr/local/bin/wkhtmltopdf'
Mystery solved! The fix in this case was to simply set it to the actual path desired (/usr/bin/wkhtmltopdf).

Cannot start Celery Worker (Kombu.asynchronous.timer)

I followed the first steps with Celery (Django) and trying to run a heavy process in the background. I have RabbitMQ server installed. However, when I try,
celery -A my_app worker -l info it throws the following error
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "c:\anaconda3\lib\site-packages\celery\concurrency\prefork.py", line
18, in <module>
from celery.concurrency.base import BasePool
File "c:\anaconda3\lib\site-packages\celery\concurrency\base.py", line 15,
in <module>
from celery.utils import timer2
File "c:\anaconda3\lib\site-packages\celery\utils\timer2.py", line 16, in
<module>
from kombu.asynchronous.timer import Entry
ModuleNotFoundError: No module named 'kombu.asynchronous.timer'
I've searched a lot, but can't seem to get it working. Any help will be highly appreciated. Thank you!

I had this issue with the default Celery installation from pip (3.1.26Post2). As mentionned above, I installed instead version 3.1.25, but Celery was still not working. Thus I explicitly installed the latest version:
pip install Celery==4.3
and everything is working now!

I landed here after I tried to install django-celery while reading celery 4.4 documentation, this package forces celery version to 3.1.26.post2, so I had to:
pip uninstall django-celery
pip uninstall celery && pip install celery # Uninstall 3.1 and install latest
As documentation clearly says:
Django is supported out of the box now so this document only contains a basic way to integrate Celery and Django. You’ll use the same API as non-Django users so you’re recommended to read the First Steps with Celery tutorial first and come back to this tutorial.

TL;DR: remove the kombu directory from the root of your virtualenv (if it exists). It may only fail on Windows.
It seems to be a quirk. I found the same error and I checked out what was happening.
The wheel package that pip downloads looks fine (kombu.asynchronous.timer exists in it). The release for the last version (currently 4.2.0) also is fine. What was strange is what I found in my virtualenv installation.
I found a kombu directory at my virtualenv root which has the content of the library but it also has an "async" directory, alongside an "asynchronous" one. These directories aren't from the 4.2.0 release, as async has the timer.py file but asynchronous doesn't.
From where did it come? It appears that from the wheel's data directory.
So, the solution: I removed the kombu directory from the root of my virtualenv and celery worked.

I have the same problem, but solved it when reinstall celery with version 3.1.25
pip uninstall celery && pip install celery==3.1.25
Maybe because windows is not officially supported by celery 4， https://github.com/celery/celery/issues/3551

I tested celery on the same python version you have and it is okay. and also https://github.com/celery/kombu/blob/master/kombu/asynchronous/timer.py shows that renaming things randomly is not going to help you. Maybe you should try pip uninstall kombu && pip --no-cache-dir install -U kombu to perform a fresh install for kombu. I guess there must be something wrong with your installation. so if the kombu reinstall thing didn't work, try installing the whole thing again.

I just started with Celery.
I followed instructions and installed Celery v 4.2.0
when I was trying to run the command :
celery -A mysite worker -l info
I got the error :
ModuleNotFoundError: No module named 'kombu.asynchronous.timer
I removed Celery installation : pip uninstall celery
Afterwards installed Celery 3.1.25 as 'chuhy' recommended
but..It had some other issues, so I immediately un-installed 3.1.25 , and reinstalled celery v4.2.0 .
After this scenario the error didn't pop again.

I have faced similar type of issue this is because of older version of celery. Uninstall celery (pip uninstall celery) and install again (pip install Celery==4.3) and kaboom it will work.

So I am running windows 10, python3.9.x, using aws sqs as the broker, just finished updating some files:
settings.py
###
### For celery tasks!
###
from kombu.utils.url import safequote
import urllib.parse
AWS_ACCESS_KEY_ID = 'my aws_access_key_id for a user'
AWS_SECRET_ACCESS_KEY = 'my aws_secret_access_key for a user'
BROKER_URL = 'sqs://%s:%s#' % (urllib.parse.quote(AWS_ACCESS_KEY_ID, safe=''), urllib.parse.quote(AWS_SECRET_ACCESS_KEY, safe=''))
BROKER_TRANSPORT = 'sqs'
BROKER_TRANSPORT_OPTIONS = {
'canves-celery-queue': {
'access_key_id': safequote(AWS_ACCESS_KEY_ID),
'secret_access_key': safequote(AWS_SECRET_ACCESS_KEY),
'region': 'us-east-1'
}
}
CELERY_DEFAULT_QUEUE = 'celery<-project-queue>'
CELERY_QUEUES = {
CELERY_DEFAULT_QUEUE: {
'exchange': CELERY_DEFAULT_QUEUE,
'binding_key': CELERY_DEFAULT_QUEUE,
}
}
###
### End celery tasks
###
celery_tasks.py (referred to in the tutorial as celery.py - renamed because apparently that caused some other programmers some errors):
from __future__ import absolute_import
import os
from celery import Celery
# Set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', '<project>.settings')
# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
# should have a `CELERY_` prefix.
app = Celery('<project>', include=['<project>.tasks'])
app.config_from_object('django.conf:settings')
# Load task modules from all registered Django apps.
app.autodiscover_tasks()
#app.task(bind=True)
def debug_task(self):
print("debug_task was fired!")
print(f'Request: {self.request!r}')
if __name__ == '__main__':
app.start()
tasks.py (this is in the same directory as the settings.py - also referrenced in celery_tasks.py)
from __future__ import absolute_import
from .celery_tasks import app
import time
#app.task(ignore_result=True)
def sleep(x, y):
print("Sleeping for: " + str(x + y))
time.sleep(x + y)
print("Slept for: " + str(x + y))
When I went to run the worker (make sure you are in the same directory as manage.py), it threw this error:
from kombu.async.timer import Entry, Timer as Schedule, to_timestamp, logger
to fix it, I ran as per gogaz's answer
pip uninstall django-celery
pip uninstall celery && pip install celery
which pushed me to the latest version of celery, 4.3... celery 4+ isn't supported on windows as per this SO question (Celery raises ValueError: not enough values to unpack), which conveniently has this answer (posted by Samuel Chen):
for celery 4.2+, python3, windows 10
pip install gevent
celery -A <project> worker -l info -P gevent
for celery 4.1+, python3, windows 10
pip install eventlet
celery -A <project> worker -l info -P eventlet
The only other error I get is from django's debugger being on, which apparently causes memory leaks...
The problem (for me at least) is that I can't use the Prefork pool, which means that I can't use app.control.revoke() to terminate tasks.
---EDIT---
Also worth mentioning that after this answer was posted, I switched to a linux box. Unknown to me, due to a lack of experience, there are different modes you can run background tasks in. I don't remember all the names, but if you type into google "celery multithreading vs gevent", it will likely come back with some other modes you can run celery in, their purposes and which ones are supported for each platform. Windows couldn't run the mode that I thought made the most sense for my problem (I believe it was multithreading), and that was a real issue. However linux can run all of them... so I switched back to linux, just for celery. I had some problems with DJango in a redhat environment, so I had to fix those issues as well :|

I work on Windows so I had a bit of problem with this.
But my solution was to create new conda env with python 3.6.8 ( as i have understand celery may work on python 3.7 but have lot of problems).
Then proceed to install latest versions of celery(4.3.0) and Django(2.2.3) and after that everything worked fine.

git executable not found in python

I was trying to clone a git repo with access key, but when I am trying to run it, It throws an exception saying git executable not found.
But i have installed git and the in_it.py shows correct path "C:\Program Files\Git\bin" Also I have installed gitpython to use the library in python
here's my code...
import git
git.Git("D:/madhav/myrep/").clone("#github.com:myrepo/scripts")
========= and it throws the following exception =================
Traceback (most recent call last): File
"C:\Users\1096506\Desktop\gitclone.py", line 1, in <module>
from git import Repo File "C:\Users\1096506\AppData\Local\Programs\Python\Python36-32\lib\site-packages\git\__init__.py",
line 84, in <module>
refresh() File "C:\Users\1096506\AppData\Local\Programs\Python\Python36-32\lib\site-packages\git\__init__.py",
line 73, in refresh
if not Git.refresh(path=path): File "C:\Users\1096506\AppData\Local\Programs\Python\Python36-32\lib\site-packages\git\cmd.py",
line 293, in refresh
raise ImportError(err) ImportError: Bad git executable. The git executable must be specified in one of the following ways:
- be included in your $PATH
- be set via $GIT_PYTHON_GIT_EXECUTABLE
- explicitly set via git.refresh()
All git commands will error until this is rectified.
This initial warning can be silenced or aggravated in the future by setting the
$GIT_PYTHON_REFRESH environment variable. Use one of the following values:
- quiet|q|silence|s|none|n|0: for no warning or exception
- warn|w|warning|1: for a printed warning
- error|e|raise|r|2: for a raised exception
Example:
export GIT_PYTHON_REFRESH=quiet

I had the same issue. What I did is:
I went to: System Properties -> Environment Variables
On System Variables Section I clicked Path, Edit and Move Up.
Environment Variables
Edit and Moved Up two places from the bottom

Error occurs because git is not in the path. So its not able to import git module.
Couple of ways to resolve it.
As suggested above adding the git binary path to environment variable path.
If git is not being used directly in the module and its only a dependent module import thats throwing this exception before importing git we could add
os.environ["GIT_PYTHON_REFRESH"] = "quiet"
and import git after this line, this would suppress the error caused due to git import

Had the same problem got it to work thanks to Muthukumaran. Just make Muthukumaran answer more clear.
Follow these steps:
import os
os.environ["GIT_PYTHON_REFRESH"] = "quiet"
import git

See if you have installed Git in the OS.
If not install git first this will solve your error.
Centos
sudo yum -y install git
Ubuntu/Debian
sudo apt-get install git
Mac Os
sudo brew install git
This solved my problem.

Make sure you're not in an inaccessible directory on *nix, such as when you've just been root and then done a su username
you may still be in root's home folder and that will trigger this error (assuming you have the correct environment variables set, and have sourced the .profile or .bashrc etc with source ~/.bashrc )
which git
/usr/bin/git
I was getting this error even after setting the environment:
in ~/.bashrc
# for bench
PATH=$PATH:/usr/bin/git
export PATH
GIT_PYTHON_GIT_EXECUTABLE=/usr/bin/git
export GIT_PYTHON_GIT_EXECUTABLE
cd
and it's working
$ bench --version
WARN: Command not being executed in bench directory
5.3.0

I came across similar problem recently and installing git followed by restarting Windows Powershell CommandLine solved the problem. May it helps.

For those who are using a Lambda layer with it. It worked adding as the comment above says just adding GIT_PYTHON_REFRESH=quiet as an environment variable.

Execute GIT_PYTHON_REFRESH=quiet in your terminal and then try to run the code.

Paster requires to be run from program directory?

Im running a solaris server which uses supervisor to monitor some Python applications.
Previously, I could run the command:
paster serve /opt/pyapps/menuadmin/prod.ini
from any directory on the server. There were some recent issues and the /opt folder was restored from a previous backup. This folder contained all of the applications including supervisor.
Now we are facing issues where supervisor will not start the applications because of "version conflicts" in Pylons.
This is where it gets weird and it makes no sense why these errors would occur.
If I run the paster command from outside of the program directory, it will throw the version conflict error. eg:
cd /
paster serve /opt/pyapps/menuadmin/prod.ini
Traceback (most recent call last):
File "/opt/csw/bin/paster", line 8, in <module>
load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
File "/opt/csw/lib/python2.6/site-packages/PasteScript-1.7.5-py2.6.egg/paste/script/command.py", line 93, in run
commands = get_commands()
File "/opt/csw/lib/python2.6/site-packages/PasteScript-1.7.5-py2.6.egg/paste/script/command.py", line 135, in get_commands
plugins = pluginlib.resolve_plugins(plugins)
File "/opt/csw/lib/python2.6/site-packages/PasteScript-1.7.5-py2.6.egg/paste/script/pluginlib.py", line 82, in resolve_plugins
pkg_resources.require(plugin)
File "/opt/csw/lib/python2.6/site-packages/setuptools-0.6c9-py2.6.egg/pkg_resources.py", line 626, in require
File "/opt/csw/lib/python2.6/site-packages/setuptools-0.6c9-py2.6.egg/pkg_resources.py", line 528, in resolve
pkg_resources.VersionConflict: (Pylons 0.9.7 (/opt/csw/lib/python2.6/site-packages/Pylons-0.9.7-py2.6.egg), Requirement.parse('Pylons>=0.10'))
But if I run the command from inside the program directory, it will run fine. eg:
cd /opt/pyapps/menuadmin/
paster serve /opt/pyapps/menuadmin/prod.ini
Starting server in PID 29902.
serving on http://127.0.0.1:3002
I absolutely cannot get my head around why this would happen!
Any thoughts or comments at all are appreciated!!!!

Based upon what you have said it seems you are running two different version of paster. The first version is running the older Pylons package 0.9.7, whilst the second has the more up to date version that meets or exceeds your app's requirements.
What I would do is first check which version of paster you are running. From outside of the project just run:
which paster
Then run the same command again within the project directory and compare the results. I suspect that you will find that the paths differ. If that is the case then all you need to do is update the version of pylons for the first version, which I'm guessing is the global install.
However, as others have commented it would be better to run apps within virtualenv, especially if as you seem to indicate you have multiple virtualenv and thus multiple projects. Trust me when I say it will save you from loads of headaches later on, from someone that didn't do this originally.

How can I use Django with MySQL in MAMP stack?

I have difficulty especially in installing MySQLdb module (MySQL-python-1.2.3c1), to connect to the MySQL in MAMP stack.
I've done a number of things such as copying the mysql include directory and library (including plugin) from a fresh installation of mysql (version 5.1.47) to the one inside MAMP (version 5.1.37).
Now, the MySQLdb module build and install doesnt give me error.
The error happens when I'm calling 'import MySQLdb' from python shell (version 2.6).
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "build/bdist.macosx-10.6-universal/egg/MySQLdb/__init__.py", line 19, in <module>
File "build/bdist.macosx-10.6-universal/egg/_mysql.py", line 7, in <module>
File "build/bdist.macosx-10.6-universal/egg/_mysql.py", line 6, in __bootstrap__
ImportError: dlopen(/Users/rhenru/.python-eggs/MySQL_python-1.2.3c1-py2.6-macosx-10.6-universal.egg-tmp/_mysql.so, 2): Symbol not found: _mysql_affected_rows
Referenced from: /Users/rhenru/.python-eggs/MySQL_python-1.2.3c1-py2.6-macosx-10.6-universal.egg-tmp/_mysql.so
Expected in: flat namespace
in /Users/rhenru/.python-eggs/MySQL_python-1.2.3c1-py2.6-macosx-10.6-universal.egg-tmp/_mysql.so
Any idea, what else do I need to do to make it works?
Thanks a bunch,
Robert
=========
Add the system response after using virtualenv as suggested by Hank Gay below...
(MyDjangoProject)MyMacPro:MyDjangoProject rhenru$ which python
/Users/rhenru/Workspace/django/MyDjangoProject/bin/python
After I run python in virtualenv, importing MySQLdb:
>>> import MySQLdb
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "build/bdist.macosx-10.6-universal/egg/MySQLdb/__init__.py", line 19, in <module>
File "build/bdist.macosx-10.6-universal/egg/_mysql.py", line 7, in <module>
File "build/bdist.macosx-10.6-universal/egg/_mysql.py", line 6, in __bootstrap__
ImportError: dlopen(/Users/rhenru/.python-eggs/MySQL_python-1.2.3c1-py2.6-macosx-10.6-universal.egg-tmp/_mysql.so, 2): Symbol not found: _mysql_affected_rows
Referenced from: /Users/rhenru/.python-eggs/MySQL_python-1.2.3c1-py2.6-macosx-10.6-universal.egg-tmp/_mysql.so
Expected in: flat namespace
in /Users/rhenru/.python-eggs/MySQL_python-1.2.3c1-py2.6-macosx-10.6-universal.egg-tmp/_mysql.so
import sys and sys.path
>>> import sys
>>> print sys.path
['', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/site-packages/pip-0.7.1-py2.6.egg', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python26.zip', '/Library/Python/2.6/site-packages/PyXML-0.8.4-py2.6-macosx-10.6-universal.egg', '/Library/Python/2.6/site-packages/pydot-1.0.2-py2.6.egg', '/Library/Python/2.6/site-packages/pyparsing-1.5.2-py2.6.egg', '/Library/Python/2.6/site-packages/vobject-0.8.1c-py2.6.egg', '/Library/Python/2.6/site-packages/pytz-2010h-py2.6.egg', '/Library/Python/2.6/site-packages/MySQL_python-1.2.3c1-py2.6-macosx-10.6-universal.egg', '/Library/Python/2.6/site-packages/distribute-0.6.12-py2.6.egg', '/Library/Python/2.6/site-packages/pip-0.7.1-py2.6.egg', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/plat-darwin', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/plat-mac', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/plat-mac/lib-scriptpackages', '/Users/rhenru/Workspace/django/MyDjangoProject/Extras/lib/python', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/lib-tk', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/lib-old', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/lib-dynload', '/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6', '/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-darwin', '/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-tk', '/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac', '/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac/lib-scriptpackages', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/site-packages', '/Library/Python/2.6/site-packages', '/Library/Python/2.6/site-packages/PIL', '/Library/Python/2.6/site-packages/setuptools-0.6c11-py2.6.egg-info', '/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python', '/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/PyObjC', '/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/wx-2.8-mac-unicode']

How are you installing MySQL-Python? I just tested in a fresh virtualenv and pip install mysql-python seems to have done the trick.
UPDATE:
pip is sort of like a package manager for Python packages.
By default, pip installs to your current site-packages directory, which is on your $PYTHONPATH. This lets other libraries/applications (like Django) access it. pip also works well with virtualenv (it should; Ian Bicking wrote them both), which is a nifty library that lets you sandbox an application. This is nice because it means you can try out new things without polluting (or even needing write access to) the global site-packages directory.
It probably seems like yak-shaving right now, but I'd say it's worth the effort to get up to speed on pip and virtualenv (you may also want to look into virtualenvwrapper, but we'll skip that for now; it's just sugar for virtualenv). It will lead to a slightly more complicated deployment scenario than putting everything in the global site-packages, but for development it's really no harder, and there are lots of good guides to deploying using a virtualenv.
I'd recommend something like the following:
curl -0 http://python-distribute.org/distribute_setup.py
python distribute_setup.py
easy_install pip
pip install virtualenv
virtualenv --distribute MyDjangoProject --no-site-packages
cd MyDjangoProject
source bin/activate (this activates the sandbox that virtualenv created)
pip install django mysql-python
At this point, you should have a totally functional Django+MySQL install (if I missed any steps, just comment and I'll try to add it in). You can start your Django project like this: django-admin.py startproject MyDjangoProject. cd into your project's directory, edit your settings.py file to point to your MySQL database, and run the dev server to test it out like so: ./manage.py runserver (you may need to chmod u+x your manage.py file). Voila! You should be able to access your site on localhost:8000. When you're done working on the project, you can just use deactivate to exit the virtualenv sandbox.
Try not to hold all this against Django: a lot of it is just best practices stuff for working with Python libraries. You could get by with a lot less, but this way it's more reproducible and you're less likely to accidentally mess up one of this project's dependencies when working on a different project.

I had this problem and it turned out to be due to an errant configuration:
export VERSIONER_PYTHON_PREFER_32_BIT=yes
I can't recall what I had this enabled for (some package that required 32-bit), probably related to Google AppEngine. But Setting it to 'no' solved by issues.
Otherwise I just installed everything using homebrew and pip.