We Keep Coding

Some python packages are not installed with docker but others are

TL/DR
I had a horrible puzzle to solve that didn't seem to be tackled anywhere in the web, so now I'm posting it here in case anybody else has the same trouble. Is is already solved, see my answer below.
It was related to pip install, not installing those packages that missed a wheel file.
Long explanation.
When running docker-compose up everything seemed to work like a charm in the building images stage, no errors reported... until the containers where started... then I got some weird errors like this one:
##Traceback (most recent call last):
File "/usr/local/bin/flask", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.7/dist-packages/flask/cli.py", line 894, in main
cli.main(args=args, prog_name=name)
File "/usr/local/lib/python3.7/dist-packages/flask/cli.py", line 557, in main
return super(FlaskGroup, self).main(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1132, in invoke
cmd_name, cmd, args = self.resolve_command(ctx, args)
File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1171, in resolve_command
cmd = self.get_command(ctx, cmd_name)
File "/usr/local/lib/python3.7/dist-packages/flask/cli.py", line 500, in get_command
self._load_plugin_commands()
File "/usr/local/lib/python3.7/dist-packages/flask/cli.py", line 496, in _load_plugin_commands
self.add_command(ep.load(), ep.name)
File "/usr/local/lib/python3.7/dist-packages/pkg_resources/_init_.py", line 2464, in load
self.require(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/pkg_resources/_init_.py", line 2487, in require
items = working_set.resolve(reqs, env, installer, extras=self.extras)
File "/usr/local/lib/python3.7/dist-packages/pkg_resources/_init_.py", line 777, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'alembic>=0.6' distribution was not found and is required by the application
It seemed that many packages were missing even though I didn't run into any errors when docker was installing them.
And more weird: when entering the container with docker run --rm -it --entrypoint bash <service-name> I could manually install all missing packages using pip3 install -r /var/www/webapp/requirements.txt.lock with no problem. And all errors would be gone. This is the exact same command that my Dockerfile uses in order to install python packages when building it's image.
We had no clue on:
Why this was happening now, since it worked until at least a couple
of weeks ago for the last 2 years!
Why did dockerfile fail but manually running exactly the same command worked.
Why only certain packages were missing, and no errors were raised for them.
How to fix this.

I started my research, took many hours but to summarize:
It seemed that the missing packages didn't have a wheel file in their pypi repository. So pip was falling into "legacy" direct installation but only for those packages, that's why the others were installed properly.
I found that many of these missing packages, like alembic, do have a .whl file but in later versions, not the one we were using: 0.9.9.
It seems that the package Wheel is needed to build a .whl file for those packages that do not include them in their pypi repository. But doing a pip list inside the container showed that wheel was indeed already installed...
I tried manually deleting the wheel package, and running pip3 install -r /var/www/webapp/requirements.txt.lock again... voilá, the same errors that docker reported. the packages without a wheel file were not installed. So I found out that wheel package:
has two different roles:
1- A setuptools extension for building wheels that provides the
bdist_wheel setuptools command
2- A command line tool for working with wheel files
From those 2, the first point seemed to be what was missing from happening.
5. Long story short, in my dockerfile, before installing the requirements, I installed wheel, problem solved.
But why did this happen now and not before??
I found that even if docker didn’t install wheel before running the pip install, it did have setuptools. I don’t exactly know how those two packages work, and less I understand how they cowork to get things done. But searching on setuptools changelog I found that it is updated quite frequently, almost daily.
More interesting, version v60.4.0 of 08 Jan 2022 seemed to be pretty big for the standard of the others, and it includes this as one of the changes:
#2968: Removed tmp_src test fixture. Previously this fixture was copying all the files and folders under the project root, including
the .git directory, which is error prone and increases testing time.
Since tmp_src was used to populate virtual environments (installing
the version of setuptools under test via the source tree), it was
replaced by the new setuptools_sdist and setuptools_wheel fixtures
(that are build only once per session testing and can be shared
between all the workers for read-only usage).
Not sure, but maybe it did change sth related to how packages without a wheel file were installed and now it wouldn't work without package Wheel doing it.

How to install SciPy on an Azure C# webapp?

I have a website built with .Net Core 1.0 (C#) and deployed it to Azure WebApp (32 bit mode).
The app uses some python scripts and I was able to create a virtual env (3.4.1) and successfully installed numpy (1.11.0) with pip install numpy.
The problem I'm facing is that I can not install SciPy. Trying pip install scipy fails because of compiler issues which I understand.
Next try was to download Christoph Gohlke's Python Extension Packages for Windows (from here), copied it to my web app and tried to run 'pip install scipy-0.18.0-cp34-cp34m-win32.whl' without success. The error I get is:
scipy-0.18.0-cp34-cp34m-win32.whl is not a supported wheel on this platform.
Storing debug log for failure in D:\home\pip\pip.log
pip.log contains the following:
scipy-0.18.0-cp34-cp34m-win32.whl is not a supported wheel on this platform.
Exception information:
Traceback (most recent call last):
File "D:\home\site\wwwroot\env\lib\site-packages\pip\basecommand.py", line 122, in main
status = self.run(options, args)
File "D:\home\site\wwwroot\env\lib\site-packages\pip\commands\install.py", line 257, in run
InstallRequirement.from_line(name, None))
File "D:\home\site\wwwroot\env\lib\site-packages\pip\req.py", line 167, in from_line
raise UnsupportedWheel("%s is not a supported wheel on this platform." % wheel.filename)
pip.exceptions.UnsupportedWheel: scipy-0.18.0-cp34-cp34m-win32.whl is not a supported wheel on this platform.
I have tried to create a requirement.txt file as stated in Troubleshooting - Package Installation. However since it's not a python app, but instead a dotNet Core C#, it doesn't seem to care about the requirement.txt file and don't see anything about it in the deploy.cmd file.

#mdeblois, your understanding is correct, please see the offical explaination below.
Some packages may not install using pip when run on Azure. It may simply be that the package is not available on the Python Package Index. It could be that a compiler is required (a compiler is not available on the machine running the web app in Azure App Service).
For this case, the solution is that you can refer to the section Troubleshooting - Package Installation of the offical tutorial to know how to deal with.

PyCluster unable to install package

This is the error I am getting when trying to install PyCluster. I am using python 2.7 with anaconda in spyder IDE and on windows.
Downloading/unpacking PyCluster
Getting page http://pypi.python.org/simple/PyCluster
URLs to search for versions for PyCluster:
* httpss://pypi.python.org/simple/PyCluster/
Getting page httpss://pypi.python.org/simple/PyCluster/
Analyzing links from page httpss://pypi.python.org/simple/pycluster/
Could not find any downloads that satisfy the requirement PyCluster
No distributions at all found for PyCluster
Exception information:
Traceback (most recent call last):
File "C:\Users\anankuma\AppData\Local\Continuum\Anaconda\lib\site-packages\pip-1.2.1-py2.7.egg\pip\basecommand.py", line 107, in main
status = self.run(options, args)
File "C:\Users\anankuma\AppData\Local\Continuum\Anaconda\lib\site-packages\pip-1.2.1-py2.7.egg\pip\commands\install.py", line 256, in run
requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
File "C:\Users\anankuma\AppData\Local\Continuum\Anaconda\lib\site-packages\pip-1.2.1-py2.7.egg\pip\req.py", line 1011, in prepare_files
url = finder.find_requirement(req_to_install, upgrade=self.upgrade)
File "C:\Users\anankuma\AppData\Local\Continuum\Anaconda\lib\site-packages\pip-1.2.1-py2.7.egg\pip\index.py", line 157, in find_requirement
raise DistributionNotFound('No distributions at all found for %s' % req)
DistributionNotFound: No distributions at all found for PyCluster
Please suggest a workaround. Thanks

here is the package there is no source or binary available to download.
a module (pyclustering) with similar name.
The Python Package Index (PYPI) is a repository of software for the Python programming language.
One can download packages manually or using pip install
One can upload binary and source packages as well.
For example this package has a source and some binaries as well
Python Wheels and Python Eggs are ease to install, mostly used on windows.
However manual install can be done using MS Windows installer files downloaded directly form PYPI

Had a similar situation installing this Pycluster package
pip was trying to build the package because there were no official wheels
for Python 3.5 & windows.
Solved it by downloading an unofficial wheel from here:
https://www.lfd.uci.edu/~gohlke/pythonlibs/#pycluster
There are many wheel files on this website.
Try different versions to see whichever works the best for you.
You can install the .whl file this way using Anaconda Prompt with the following code
pip install Pycluster-1.56-cp36-cp36m-win_amd64.whl

install with pip:
pip3 install pyclustering
it even works with Anaconda prompt

Windows: Python + OpenCV + Qt: "import cv2" gives "dll not found" error

I have a problem when I'm trying to use OpenCV (v3.1.0) in Python (v3.4). To make things more complicated, OpenCV is built with the "contrib" package and Qt support (v5.5).
This is what I did (this has some pitfalls on it's own, but those are out of the scope of this question):
Install required software
I installed WinPython (v3.4.4, 64 Bit version)
I installed cmake (v3.6.0)
Download sources
I downloaded the Qt sources from the GIT repo
I downloaded the OpenCV sources from the GIT repo
I downloaded the OpenCV "contrib" sources from the GIT repo
build sources -> RELEASE, 64 Bit (!)
build Qt from sources
build OpenCV from sources (with "contrib" modules)
cmake ( OpenCV 3.1.0 for python 3 ) already took care of copying the "cv2.pyd" to the "your/python/folder/Lib/site-modules" directory.
Now I have the problem that calling "import cv2" from python gives the error "
>>> import cv2
Traceback (most recent call last):
File "<pyshell#0>", line 1, in <module>
import cv2
ImportError: DLL load failed: Module not found
How can I find out which module is missing and how to solve it?

What you may not want to do is just adding paths that are likely needed to your environment variables.
There is a way to find out what is missing:
Download DependencyWalker
open your "cv2.pyd" with the dependency walker
Analyze your file (starts automatically when you select your file)
Hit "F9" such that it shows the full paths to the required DLLs
Check, which DLL files are missing
Copy them to those folders
For me, it was the case that python/cv2.pyd searched to all the Qt DLLs in the folder where cv2.pyd is in.
I just copied them there and that was it.

For Windows:
Step 1. Download python installer from https://www.python.org/downloads/
,install python by double clicking this file and check the option for adding python to PATH as in given image.
Step 2. Download file "numpy‑1.15.4+mkl‑cp27‑cp27m‑win32.whl" from unofficial site.
Go to this file's location open command prompt and type-
pip install numpy‑1.15.4+mkl‑cp27‑cp27m‑win32.whl
Now go to python IDLE and type:
import numpy
It should give no error.
Step 3. Download file "scipy‑1.2.1‑cp27‑cp27m‑win32.whl" from unofficial site.
Go to this file's location open command prompt and type-
pip install scipy‑1.2.1‑cp27‑cp27m‑win32.whl
Now go to python IDLE and type:
import scipy
It should give no error.
step 4 : Download file "opencv_python‑2.4.13.7‑cp27‑cp27m‑win32.whl" from unofficial site.
Go to this file's location open command prompt and type-
pip install opencv_python‑2.4.13.7‑cp27‑cp27m‑win32.whl
Now go to python IDLE and type:
import cv2
It should give no error, means opencv installed successfully in windows.

Pycharm Virtulenv "non-zero exit code" after install wheel or module from pip

After installing kivy via a wheel i came upon an error in Pycharm when i refreshed my virtual env for python 3.4. The error is in the section
Pycharm: Projects ->
Project Interpreter -> [Virtualenv packages]
Error/warning along the bottom shows "non-zero exit code"
My environment:
Windows 8.1 64-bit
Python 3.4.3 32-bit
Pycharm 4.5.4
Virtual env - 3.4.3
This happened after i installed a wheel from "http://www.lfd.uci.edu/~gohlke/pythonlibs/" (which is an awesome site... check it out) from the command line.
first i installed pygame from the downloaded wheel file located at"http://www.lfd.uci.edu/~gohlke/pythonlibs/:
C:\Users\<user_name>\3.4Python_32Bit\Scripts> pip3.4.exe install C:\Users\<user_name>\Downloads\pygame-1.9.2a0-cp34-none-win32.whl
next i adjust the cython version:
C:\Users\<user_name>\3.4Python_32Bit\Scripts> pip3.4.exe install -I Cython==0.21.2
then i installed kivymd & kivy garden via the pycharm virtualenv interface within pycharm - both installed successfully.
Finally i tried to install "kivy"
The results were initially good, it showed installed, the package loaded in the virtualenv within pycharm, but when i reloaded the virtualenv in pycharm to associate it to a package the pycharm interface showed "non-zero exit code." How can this be fixed, there is no other errors visible?

The way this can be solved is by running the python from said environment
i.e.
C:\Users\<user_name>\3.4Python_32Bit\Scripts>python.exe C:\Users\<user_name>\3.4Python
_32Bit\Lib\site.py
The output will tell you what package you have that has failed when it gathers the packages from the "C:\Users\\3.4Python_32Bit\Lib\site-packages":
Failed to import the site module
Traceback (most recent call last):
File "C:\Users\<user_name>\3.4Python_32Bit\Lib\site.py", line 703, in <module>
main()
File "C:\Users\<user_name>\3.4Python_32Bit\Lib\site.py", line 683, in main
paths_in_sys = addsitepackages(paths_in_sys)
File "C:\Users\<user_name>\3.4Python_32Bit\Lib\site.py", line 282, in addsitepack
ages
addsitedir(sitedir, known_paths)
File "C:\Users\<user_name>\3.4Python_32Bit\Lib\site.py", line 204, in addsitedir
addpackage(sitedir, name, known_paths)
File "C:\Users\<user_name>\3.4Python_32Bit\Lib\site.py", line 173, in addpackage
exec(line)
File "<string>", line 1, in <module>
KeyError: 'kivy'
In order to use your environment minus the package causing the issue is the remove the folder directly related to this error, in this case kivy:
remove C:\Users\<user_name>\3.4Python_32Bit\Lib\site-packages
If you want to reinstall the package confirm that you have the right dependencies, wheel(if this is what you are using), or you have the correct versions of installed packages. Then reinstall as needed.This can be applied to other packages with the same issues.