We are deploying python based Azure Function using Azure CLI but getting this below error while importing cv2 library.
ImportError: libgthread-2.0.so.0: cannot open shared object file: No such file or directory
We are not using any docker image. Would prefer a non-docker based solution.
Exception: ImportError: libgthread-2.0.so.0: cannot open shared object file: No such file or directory
Stack: File "/usr/local/lib/python3.6/site-packages/azure_functions_worker/dispatcher.py", line 239, in _handle__function_load_request
func_request.metadata.entry_point)
File "/usr/local/lib/python3.6/site-packages/azure_functions_worker/loader.py", line 66, in load_function
mod = importlib.import_module(fullmodname)
File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/site/wwwroot/FaceFinderHttpTrigger/__init__.py", line 11, in <module>
import cv2
File "/home/site/wwwroot/.python_packages/lib/python3.6/site-packages/cv2/__init__.py", line 3, in <module>
from .cv2 import *
There is a similar SO thread Azure Functions: In a Python function under Linux, how do I import a non-standard module such as opencv/imutils? with the same issue as yours.
Due to the libgthread shared library is a Linux base library, per my experience, it seems to be caused by Azure CLI could not package these Python packages with native dependencies as Azure Function. So the only solution I think is to build your Azure Function as a Docker container by yourself as the offical document Create a function on Linux using a custom image said, or try to use Azure CLI command func azure functionapp publish with --build-native-deps option to automatically build and configure the dependencies using a Docker container.
Also as I known, there is a blog OPENCV WITH AZURE FUNCTIONS which seems to work fine with opencv for Azure Functions, but it just run on local development environment, not on cloud, as my view. So I still recommend you to deploy your app as a Docker contaier for the app using native dependencies.
Related
I want to have a Celery Worker (with Redis as the broker and backend) which will recieve tasks from several remote Celery producers (through the use ".send_task").
I'm trying to package the producer side into a single *.exe with pyinstaller. The idea is to run the *.exe which will send a task to the remote Celery worker.
Here's the "producer" code:
from celery import Celery
def main():
print('Sending Task')
#celery_app.send_task('add',args=(2,2))
celery=Celery('redis://:password#redis_server:6379')
celery.send_task('tasks.add',(2,2))
if __name__=="__main__":
main()
I then compile the code into a single file with pyinstaller. Unfortunately I'm running into an exception when trying to run the executable. I'm getting the following error:
Sending Task
Traceback (most recent call last):
File "sendtask.py", line 11, in <module>
File "sendtask.py", line 6, in main
File "celery\app\base.py", line 291, in __init__
File "celery\app\base.py", line 291, in <listcomp>
File "kombu\utils\imports.py", line 56, in symbol_by_name
File "importlib\__init__.py", line 127, in import_module
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'celery.fixups'
[5588] Failed to execute script 'sendtask' due to unhandled exception!
Python v.3.8.10-64
Pyinstaller v. 4.5.1
Anyone know if it's possible to have a python script compiled into an executable send tasks to a remote Celery worker?
Any help would be appreciated
Thanks
José
I had the same issue with pyinstaller==4.8, celery==5.2.3 and python 3.9.7.
On my project i solved by adding explicity hidden-import requirements on the pyinstaller command line.
pyinstaller.exe --hidden-import celery.fixups --hidden-import celery.fixups.django myprogram.py
I guess it's due to an improper dependecy analysis from Pyinstaller, it seems it's common to manually explicit some deps when deploying project with such kind of tools.
The function was working before the last deployment. It is used for NLC and we are running two functions for two different languages. For the new deployment we removed some code which was only used for one language and mirrored the code to be equal to the other language. Azure CLI tells us:
Upload completed successfully.
Deployment completed successfully.
But when trying to invoke the function afterwards it either returns 500 without a body or it returns 404 with:
<html>
<head>
<title>404 Not Found</title>
</head>
<body bgcolor="white">
<center>
<h1>404 Not Found</h1>
</center>
<hr>
<center>nginx</center>
</body>
</html>
The stacktrace we can see in the monitoring is the following:
Exception while executing function: Functions.classify Result: Failure
Exception: OSError: [Errno 107] Transport endpoint is not connected: '/home/site/wwwroot/.python_packages/lib/site-packages'
Stack: File "/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/dispatcher.py", line 271, in _handle__function_load_request
func = loader.load_function(
File "/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/utils/wrappers.py", line 32, in call
return func(*args, **kwargs)
File "/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/loader.py", line 76, in load_function
mod = importlib.import_module(fullmodname)
File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 783, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/site/wwwroot/classify/__init__.py", line 4, in <module>
File "/home/site/wwwroot/classify/predict.py", line 1, in <module>
File "/home/site/wwwroot/.python_packages/lib/site-packages/fastai/text/all.py", line 1, in <module>
File "/home/site/wwwroot/.python_packages/lib/site-packages/fastai/basics.py", line 1, in <module>
File "/home/site/wwwroot/.python_packages/lib/site-packages/fastai/data/all.py", line 1, in <module>
File "/home/site/wwwroot/.python_packages/lib/site-packages/fastai/torch_basics.py", line 1, in <module>
File "/home/site/wwwroot/.python_packages/lib/site-packages/torch/__init__.py", line 482, in <module>
File "/home/site/wwwroot/.python_packages/lib/site-packages/torch/serialization.py", line 8, in <module>
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 971, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 914, in _find_spec
File "<frozen importlib._bootstrap_external>", line 1342, in find_spec
File "<frozen importlib._bootstrap_external>", line 1314, in _get_spec
File "<frozen importlib._bootstrap_external>", line 1443, in find_spec
File "<frozen importlib._bootstrap_external>", line 1483, in _fill_cache
Commands used for deployment were the following, which worked before:
pip3 install --target="./.python_packages/lib/site-packages" -r requirements.txt
func azure functionapp publish [functionapp name] --no-build
Deleting the python packages and rebuilding etc. did not change anything.
I tried creating a new function app from scratch but it shows the same errors.
I tried to deploy from the folder of the other language with one file swapped (classifier language model) but it
showed the same error. I fear trying to redeploy the working language
for testing as it is working and I don't want to kill it.
It seems to be happening somewhere during module import. Though I am not sure how it was working before (as you mentioned), but I suggest you not to do pip3 install manually and remove --no-build flag to let the func azure functionapp publish handle that. Just make sure you have the requirements.txt in the root.
# no pip install manually
# without --no-build flag
func azure functionapp publish [functionapp name]
I'm trying to compile a python project including numpy and pandas with nuitka but no matter how I compile it, I have an error when launching the final executable.
For compilation I use the next command :
python -m nuitka --standalone --follow-imports --plugin-enable=numpy --include-plugin-directory=.venv/lib/python3.6/site-packages/pandas --include-plugin-directory=.venv/lib/python3.6/site-packages/numpy --show-progress --show-scons newick2phylipmatrix.py
But when I try to launch the executable generated by the compilation I get the following message :
Traceback (most recent call last):
File "/home/mindsound/sandbox/distMatrix/newick2phylipmatrix.dist/newick2phylipmatrix.py", line 6, in <module>
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
File "/home/mindsound/sandbox/distMatrix/newick2phylipmatrix.dist/pandas/__init__.py", line 22, in <module pandas>
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
File "/home/mindsound/sandbox/distMatrix/newick2phylipmatrix.dist/pandas/compat/numpy/__init__.py", line 3, in <module pandas.compat.numpy>
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
File "/home/mindsound/sandbox/distMatrix/newick2phylipmatrix.dist/distutils/__init__.py", line 11, in <module>
AttributeError: module 'opcode' has no attribute '__file__'
Currently without changing anything to the code, I manage to create a standalone with pyinstaller but I can't do it with nuitka but if it's possible I would like to use it to benefit from compilation optimization.
Do you have any idea where this problem comes from and whether it can be solved ?
Well i finally got the solution for the same. The distutils created / copied by the virtualenv while creation is different from the one in the pythons original distribution. One of the teammember at Nuitka helped me discover the same and helped me reach the solution. Well the option i discovered was to pick up the distutils from the Pythons lib folder and replace the existing distutils folder in the virtualenv folder created and the go ahead - compile the same. I believe strongly that the issue will go away for you too as it have gone away for mine. Now will be looking to get in the proper solution for my complete project work.
Happy Compilation :)
Using a Python 3.6 enviroment in Anaconda, I'm able to run an azure function that accesses blob storage. Locally this function works fine, but when I post it on the cloud using VS Code and attempt a HTTP request, I get this error:
Result: Failure
Exception: ModuleNotFoundError: No module named 'azure.storage'
Stack: File "/usr/local/lib/python3.6/site-packages/azure/functions_worker/dispatcher.py", line 230, in _handle__function_load_request
func_request.metadata.entry_point)
File "/usr/local/lib/python3.6/site-packages/azure/functions_worker/loader.py", line 66, in load_function
mod = importlib.import_module(fullmodname)
File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/site/wwwroot/Convert/__init__.py", line 6, in <module>
from azure.storage.blob import BlockBlobService
Why do I only get this error in the cloud enviroment and how do I fix it.
Actually, I recommanded you to follow the offical Quick start tutorial Create an HTTP triggered function in Azure for Python to create a virtual environment for your function app and publish it within the virtual environment via command line, because all installed Python packages will be re-packaged and built to a publish file and deploy to your Azure Function App via func azure functionapp publish <APP_NAME> command. If not publish within the virtual environment, the deployment will be missing the required packages for your app.
And as #JohnGordon said in comment, please make sure your Azure Function App have been set up the correct environment for Python, not NodeJS or other language runtimes. If not, you can use the command func azure functionapp publish <APP_NAME> --force to force change its environment for runtime based on your virtual environment on local.
Also as #JohnGordon said in comment, if the two possible cases above are not real reason of your issue, you need to check your code files whether there is a file named azure.py, because the same file name azure will conflict with the package name of Azure SDK for Python.
Hope it helps.
Ok so the solution I found was using the virtual environment and command line stuff like #Peter Pan was suggesting, but added azure.storage to my requirements.txt file. Once I did that it uploaded and worked fine.
I fixed the azure deployment issue by using below pip command:
pip install --target="$(System.DefaultWorkingDirectory)/.python_packages/lib/site-packages" -r requirements.txt
I have a python function that I've created as an Azure App Function. It works on my local machine fine, however fails with no descriptive error when executed after a successful publishing.
Originally, it failed with 500 internal server error due to:
Result: Failure
Exception: ModuleNotFoundError: No module named 'pandas'
Stack: File "/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/azure/functions_worker/dispatcher.py", line 218, in _handle__function_load_request
func_request.metadata.entry_point)
File "/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/azure/functions_worker/loader.py", line 66, in load_function
mod = importlib.import_module(fullmodname)
File "/root/.pyenv/versions/3.6.8/lib/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/site/wwwroot/generateGraph/__init__.py", line 2, in <module>
import pandas as pd
I thought this was me not publishing with the --build-native-deps switch. So I republished with this switch: (.env) C:\Temp\python_function>func azure functionapp publish httpGenGraph --build-native-deps
And publishing was successful. Now, when I execute the Http Trigger, I get the same error again. Is there something I'm missing? I'm new to Python so it could be not understanding the packaging requirements or something. The function does work within my local virtual python environment.
Figured it out. I needed to pipe the output of pip freeze to the requirements.txt file. This allowed the packages to be properly referenced when publishing.
For reference:
(.env) C:\Temp\python_function>pip freeze > requirements.txt
Then:
(.env) C:\Temp\python_function>func azure functionapp publish httpGenGraph --build-native-deps