I want to have a Celery Worker (with Redis as the broker and backend) which will recieve tasks from several remote Celery producers (through the use ".send_task").
I'm trying to package the producer side into a single *.exe with pyinstaller. The idea is to run the *.exe which will send a task to the remote Celery worker.
Here's the "producer" code:
from celery import Celery
def main():
print('Sending Task')
#celery_app.send_task('add',args=(2,2))
celery=Celery('redis://:password#redis_server:6379')
celery.send_task('tasks.add',(2,2))
if __name__=="__main__":
main()
I then compile the code into a single file with pyinstaller. Unfortunately I'm running into an exception when trying to run the executable. I'm getting the following error:
Sending Task
Traceback (most recent call last):
File "sendtask.py", line 11, in <module>
File "sendtask.py", line 6, in main
File "celery\app\base.py", line 291, in __init__
File "celery\app\base.py", line 291, in <listcomp>
File "kombu\utils\imports.py", line 56, in symbol_by_name
File "importlib\__init__.py", line 127, in import_module
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'celery.fixups'
[5588] Failed to execute script 'sendtask' due to unhandled exception!
Python v.3.8.10-64
Pyinstaller v. 4.5.1
Anyone know if it's possible to have a python script compiled into an executable send tasks to a remote Celery worker?
Any help would be appreciated
Thanks
José
I had the same issue with pyinstaller==4.8, celery==5.2.3 and python 3.9.7.
On my project i solved by adding explicity hidden-import requirements on the pyinstaller command line.
pyinstaller.exe --hidden-import celery.fixups --hidden-import celery.fixups.django myprogram.py
I guess it's due to an improper dependecy analysis from Pyinstaller, it seems it's common to manually explicit some deps when deploying project with such kind of tools.
Related
I've been running a dockerized flask application that uses Celery to run tasks.
To run the app I'm using gunicorn with eventlet and It's been working fine using alpine linux distribution.
However I had to move to ubuntu due to some issues with sklearn and other libraries, and now I'm having problems to run my app.
First of all I'm getting this error:
Error: class uri 'eventlet' invalid or not found:
[Traceback (most recent call last):
File "/myapp/env/lib/python3.6/site-packages/gunicorn/workers/geventlet.py", line 11, in <module>
import eventlet
ModuleNotFoundError: No module named 'eventlet'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/myapp/env/lib/python3.6/site-packages/gunicorn/util.py", line 135, in load_class
mod = import_module('.'.join(components))
File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/myapp/env/lib/python3.6/site-packages/gunicorn/workers/geventlet.py", line 13, in <module>
raise RuntimeError("You need eventlet installed to use this worker.")
RuntimeError: You need eventlet installed to use this worker.
Then I tried by adding pip install eventlet to my app's Dockerfile, and now I have this another error:
Error: class uri 'eventlet' invalid or not found:
[Traceback (most recent call last):
File "/myapp/env/lib/python3.6/site-packages/gunicorn/util.py", line 135, in load_class
mod = import_module('.'.join(components))
File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/myapp/env/lib/python3.6/site-packages/gunicorn/workers/geventlet.py", line 11, in <module>
import eventlet
File "/myapp/env/lib/python3.6/site-packages/eventlet/__init__.py", line 10, in <module>
from eventlet import convenience
File "/myapp/env/lib/python3.6/site-packages/eventlet/convenience.py", line 7, in <module>
from eventlet.green import socket
File "/myapp/env/lib/python3.6/site-packages/eventlet/green/socket.py", line 21, in <module>
from eventlet.support import greendns
File "/myapp/env/lib/python3.6/site-packages/eventlet/support/greendns.py", line 69, in <module>
setattr(dns.rdtypes.IN, pkg, import_patched('dns.rdtypes.IN.' + pkg))
File "/myapp/env/lib/python3.6/site-packages/eventlet/support/greendns.py", line 59, in import_patched
return patcher.import_patched(module_name, **modules)
File "/myapp/env/lib/python3.6/site-packages/eventlet/patcher.py", line 126, in import_patched
*additional_modules + tuple(kw_additional_modules.items()))
File "/myapp/env/lib/python3.6/site-packages/eventlet/patcher.py", line 100, in inject
module = __import__(module_name, {}, {}, module_name.split('.')[:-1])
File "/myapp/env/lib/python3.6/site-packages/dns/rdtypes/IN/WKS.py", line 25, in <module>
_proto_tcp = socket.getprotobyname('tcp')
OSError: protocol not found
This is how I start my app (start.sh):
gunicorn -w 1 --worker-class eventlet --certfile=myapp/myapp.crt --keyfile=myapp/myapp.key --bind 0.0.0.0:5000 --log-config myapp/gunicorn.conf myapp.run:app
It's been working fine until I moved to an ubuntu based container.
What am I missing here?
I appreciate any help.
Thanks!
I had the same issue, and I found that it is caused by some updates made by eventlet they removed eventlet.wsgi.ALREADY_HANDLED but gunicorn is still using it. So , you better downgrade the eventlet version.
pip install gunicorn==20.1.0 eventlet==0.30.2
Here is a reference https://github.com/eventlet/eventlet/issues/702
I'll post the solution I've found in case it's useful to anyone.
Just I was missing the /etc/protocols, and this is how I've fixed it:
I had to add this step in my Dockerfile:
RUN apt-get -o Dpkg::Options::='--force-confmiss' install --reinstall -y netbase
I tried for a lot and finally working, please check these:
gunicorn==20.1.0 eventlet==0.30.2 (error source: https://github.com/benoitc/gunicorn/pull/2581)
python version under 3.10 (error source: https://github.com/eventlet/eventlet/issues/687)
If u are using heroku, edit the requeriments.txt adding gunicorn and eventlet version and also edit runtime.txt with "python-3.9.13" (In case you are using heroku stack 22). If you are running another heroku stack, please check it's python compatible versions. (source = https://devcenter.heroku.com/articles/python-support#supported-runtimes)
I'm trying to compile a python project including numpy and pandas with nuitka but no matter how I compile it, I have an error when launching the final executable.
For compilation I use the next command :
python -m nuitka --standalone --follow-imports --plugin-enable=numpy --include-plugin-directory=.venv/lib/python3.6/site-packages/pandas --include-plugin-directory=.venv/lib/python3.6/site-packages/numpy --show-progress --show-scons newick2phylipmatrix.py
But when I try to launch the executable generated by the compilation I get the following message :
Traceback (most recent call last):
File "/home/mindsound/sandbox/distMatrix/newick2phylipmatrix.dist/newick2phylipmatrix.py", line 6, in <module>
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
File "/home/mindsound/sandbox/distMatrix/newick2phylipmatrix.dist/pandas/__init__.py", line 22, in <module pandas>
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
File "/home/mindsound/sandbox/distMatrix/newick2phylipmatrix.dist/pandas/compat/numpy/__init__.py", line 3, in <module pandas.compat.numpy>
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
File "/home/mindsound/sandbox/distMatrix/newick2phylipmatrix.dist/distutils/__init__.py", line 11, in <module>
AttributeError: module 'opcode' has no attribute '__file__'
Currently without changing anything to the code, I manage to create a standalone with pyinstaller but I can't do it with nuitka but if it's possible I would like to use it to benefit from compilation optimization.
Do you have any idea where this problem comes from and whether it can be solved ?
Well i finally got the solution for the same. The distutils created / copied by the virtualenv while creation is different from the one in the pythons original distribution. One of the teammember at Nuitka helped me discover the same and helped me reach the solution. Well the option i discovered was to pick up the distutils from the Pythons lib folder and replace the existing distutils folder in the virtualenv folder created and the go ahead - compile the same. I believe strongly that the issue will go away for you too as it have gone away for mine. Now will be looking to get in the proper solution for my complete project work.
Happy Compilation :)
We are deploying python based Azure Function using Azure CLI but getting this below error while importing cv2 library.
ImportError: libgthread-2.0.so.0: cannot open shared object file: No such file or directory
We are not using any docker image. Would prefer a non-docker based solution.
Exception: ImportError: libgthread-2.0.so.0: cannot open shared object file: No such file or directory
Stack: File "/usr/local/lib/python3.6/site-packages/azure_functions_worker/dispatcher.py", line 239, in _handle__function_load_request
func_request.metadata.entry_point)
File "/usr/local/lib/python3.6/site-packages/azure_functions_worker/loader.py", line 66, in load_function
mod = importlib.import_module(fullmodname)
File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/site/wwwroot/FaceFinderHttpTrigger/__init__.py", line 11, in <module>
import cv2
File "/home/site/wwwroot/.python_packages/lib/python3.6/site-packages/cv2/__init__.py", line 3, in <module>
from .cv2 import *
There is a similar SO thread Azure Functions: In a Python function under Linux, how do I import a non-standard module such as opencv/imutils? with the same issue as yours.
Due to the libgthread shared library is a Linux base library, per my experience, it seems to be caused by Azure CLI could not package these Python packages with native dependencies as Azure Function. So the only solution I think is to build your Azure Function as a Docker container by yourself as the offical document Create a function on Linux using a custom image said, or try to use Azure CLI command func azure functionapp publish with --build-native-deps option to automatically build and configure the dependencies using a Docker container.
Also as I known, there is a blog OPENCV WITH AZURE FUNCTIONS which seems to work fine with opencv for Azure Functions, but it just run on local development environment, not on cloud, as my view. So I still recommend you to deploy your app as a Docker contaier for the app using native dependencies.
I have a code for predicting some value that uses xgboost package in the code. When I run it in PyCharm, it runs as expected.
The problem is when I make an executable file using pyinstaller. It will make the exe without any error, but when I run it the following error is raised:
Traceback (most recent call last):
File "test_fraud.py", line 3, in <module>
import xgboost
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in
_find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File
"C:\Users\ShubhamSingh\PycharmProjects\cfna_scoring\venv\lib\site-
packages\PyInstaller\loader\pyimod03_importers.py", line 627, in
exec_module
exec(bytecode, module.__dict__)
File "lib\site-packages\xgboost\__init__.py", line 11, in <module>
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in
_find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "C:\Users\ShubhamSingh\PycharmProjects\cfna_scoring\venv\lib\site-
packages\PyInstaller\loader\pyimod03_importers.py", line 627, in
exec_module
exec(bytecode, module.__dict__)
File "lib\site-packages\xgboost\core.py", line 161, in <module>
File "lib\site-packages\xgboost\core.py", line 123, in _load_lib
File "lib\site-packages\xgboost\libpath.py", line 48, in find_lib_path
xgboost.libpath.XGBoostLibraryNotFound: Cannot find XGBoost Library in
the candidate path, did you install compilers and run build.sh in root
path?
List of candidates:
C:\Users\SHUBHA~1\AppData\Local\Temp\_MEI11402\xgboost\xgboost.dll
C:\Users\SHUBHA~1\AppData\Local\Temp\_MEI11402\xgboost
\../../lib/xgboost.dll
C:\Users\SHUBHA~1\AppData\Local\Temp\_MEI11402\xgboost\./lib/xgboost.dll
C:\Users\SHUBHA~1\AppData\Local\Temp\_MEI11402\xgboost\xgboost.dll
C:\Users\SHUBHA~1\AppData\Local\Temp\_MEI11402\xgboost
\../../windows/x64/Release/xgboost.dll
C:\Users\SHUBHA~1\AppData\Local\Temp\_MEI11402\
xgboost\./windows/x64/Release/xgboost.dll
[6564] Failed to execute script test_fraud
What's wrong here?
It seems that Pyinstaller can't find the xgboost.dll, VERSION files. So you need to add them manually to your output package. I also suggest you use a try/except block to see what is going on.
Suppose this simple example:
import traceback
try:
import xgboost
input("xgboost imported successfully!")
except Exception:
traceback.print_exc()
input("Import Error!")
I suggest you use an env to build your script so, you need to add the xgboost directory located at <path_to_venv>/xgboost and VERSION file located at <path_to_venv>/Lib/site-packages/xboost. Next, add them as a data-file with pyinstaller. Launch your env and execute the below command (My virtualenv named as env):
├───myscript.py
├───env
Command:
(env) > pyinstaller myscript.py -F --add-data "./env/xgboost/*;xgboost/" --add-data "./env/Lib/site-packages/xgboost/VERSION;xgboost/"
I have a python function that I've created as an Azure App Function. It works on my local machine fine, however fails with no descriptive error when executed after a successful publishing.
Originally, it failed with 500 internal server error due to:
Result: Failure
Exception: ModuleNotFoundError: No module named 'pandas'
Stack: File "/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/azure/functions_worker/dispatcher.py", line 218, in _handle__function_load_request
func_request.metadata.entry_point)
File "/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/azure/functions_worker/loader.py", line 66, in load_function
mod = importlib.import_module(fullmodname)
File "/root/.pyenv/versions/3.6.8/lib/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/site/wwwroot/generateGraph/__init__.py", line 2, in <module>
import pandas as pd
I thought this was me not publishing with the --build-native-deps switch. So I republished with this switch: (.env) C:\Temp\python_function>func azure functionapp publish httpGenGraph --build-native-deps
And publishing was successful. Now, when I execute the Http Trigger, I get the same error again. Is there something I'm missing? I'm new to Python so it could be not understanding the packaging requirements or something. The function does work within my local virtual python environment.
Figured it out. I needed to pipe the output of pip freeze to the requirements.txt file. This allowed the packages to be properly referenced when publishing.
For reference:
(.env) C:\Temp\python_function>pip freeze > requirements.txt
Then:
(.env) C:\Temp\python_function>func azure functionapp publish httpGenGraph --build-native-deps