We Keep Coding

Some python packages are not installed with docker but others are

TL/DR
I had a horrible puzzle to solve that didn't seem to be tackled anywhere in the web, so now I'm posting it here in case anybody else has the same trouble. Is is already solved, see my answer below.
It was related to pip install, not installing those packages that missed a wheel file.
Long explanation.
When running docker-compose up everything seemed to work like a charm in the building images stage, no errors reported... until the containers where started... then I got some weird errors like this one:
##Traceback (most recent call last):
File "/usr/local/bin/flask", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.7/dist-packages/flask/cli.py", line 894, in main
cli.main(args=args, prog_name=name)
File "/usr/local/lib/python3.7/dist-packages/flask/cli.py", line 557, in main
return super(FlaskGroup, self).main(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1132, in invoke
cmd_name, cmd, args = self.resolve_command(ctx, args)
File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1171, in resolve_command
cmd = self.get_command(ctx, cmd_name)
File "/usr/local/lib/python3.7/dist-packages/flask/cli.py", line 500, in get_command
self._load_plugin_commands()
File "/usr/local/lib/python3.7/dist-packages/flask/cli.py", line 496, in _load_plugin_commands
self.add_command(ep.load(), ep.name)
File "/usr/local/lib/python3.7/dist-packages/pkg_resources/_init_.py", line 2464, in load
self.require(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/pkg_resources/_init_.py", line 2487, in require
items = working_set.resolve(reqs, env, installer, extras=self.extras)
File "/usr/local/lib/python3.7/dist-packages/pkg_resources/_init_.py", line 777, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'alembic>=0.6' distribution was not found and is required by the application
It seemed that many packages were missing even though I didn't run into any errors when docker was installing them.
And more weird: when entering the container with docker run --rm -it --entrypoint bash <service-name> I could manually install all missing packages using pip3 install -r /var/www/webapp/requirements.txt.lock with no problem. And all errors would be gone. This is the exact same command that my Dockerfile uses in order to install python packages when building it's image.
We had no clue on:
Why this was happening now, since it worked until at least a couple
of weeks ago for the last 2 years!
Why did dockerfile fail but manually running exactly the same command worked.
Why only certain packages were missing, and no errors were raised for them.
How to fix this.

I started my research, took many hours but to summarize:
It seemed that the missing packages didn't have a wheel file in their pypi repository. So pip was falling into "legacy" direct installation but only for those packages, that's why the others were installed properly.
I found that many of these missing packages, like alembic, do have a .whl file but in later versions, not the one we were using: 0.9.9.
It seems that the package Wheel is needed to build a .whl file for those packages that do not include them in their pypi repository. But doing a pip list inside the container showed that wheel was indeed already installed...
I tried manually deleting the wheel package, and running pip3 install -r /var/www/webapp/requirements.txt.lock again... voilá, the same errors that docker reported. the packages without a wheel file were not installed. So I found out that wheel package:
has two different roles:
1- A setuptools extension for building wheels that provides the
bdist_wheel setuptools command
2- A command line tool for working with wheel files
From those 2, the first point seemed to be what was missing from happening.
5. Long story short, in my dockerfile, before installing the requirements, I installed wheel, problem solved.
But why did this happen now and not before??
I found that even if docker didn’t install wheel before running the pip install, it did have setuptools. I don’t exactly know how those two packages work, and less I understand how they cowork to get things done. But searching on setuptools changelog I found that it is updated quite frequently, almost daily.
More interesting, version v60.4.0 of 08 Jan 2022 seemed to be pretty big for the standard of the others, and it includes this as one of the changes:
#2968: Removed tmp_src test fixture. Previously this fixture was copying all the files and folders under the project root, including
the .git directory, which is error prone and increases testing time.
Since tmp_src was used to populate virtual environments (installing
the version of setuptools under test via the source tree), it was
replaced by the new setuptools_sdist and setuptools_wheel fixtures
(that are build only once per session testing and can be shared
between all the workers for read-only usage).
Not sure, but maybe it did change sth related to how packages without a wheel file were installed and now it wouldn't work without package Wheel doing it.

Permission error with pip Python 3

I am using Python 3 on Mac OS X. Everything was perfect, I used to install packages like this:
pip install somePackage
But IDLE stopped working so I had to re-install it. After that, when I ran pip, I got pip is not defined so I had to install pip. Now, every time I try to install a package using pip, I get this error:
Exception:
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/commands/install.py", line 342, in run
prefix=options.prefix_path,
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_set.py", line 784, in install
**kwargs
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_install.py", line 851, in install
self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_install.py", line 1064, in move_wheel_files
isolated=self.isolated,
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/wheel.py", line 377, in move_wheel_files
clobber(source, dest, False, fixer=fixer, filter=filter)
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/wheel.py", line 287, in clobber
ensure_dir(dest) # common for the 'include' path
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/utils/__init__.py", line 83, in ensure_dir
os.makedirs(path)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 1] Operation not permitted: '/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7/pygame'
I thought it may be from the version so I updated it. Now, whenever I run pip install something I get this error:
Command "/usr/bin/python -u -c "import setuptools, tokenize;file='/private/var/folders/bc/hwpn9sqn5352xcbst0p89snm0000gn/T/pip-build-hSNyxK/termcolor/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /var/folders/bc/hwpn9sqn5352xcbst0p89snm0000gn/T/pip-U3ZtNq-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/bc/hwpn9sqn5352xcbst0p89snm0000gn/T/pip-build-hSNyxK/termcolor/
I can't even delete a package - when I run pip uninstall something I get this error:
Exception:
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/commands/uninstall.py", line 76, in run
requirement_set.uninstall(auto_confirm=options.yes)
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_set.py", line 346, in uninstall
req.uninstall(auto_confirm=auto_confirm)
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_install.py", line 754, in uninstall
paths_to_remove.remove(auto_confirm)
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_uninstall.py", line 115, in remove
renames(path, new_path)
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/utils/__init__.py", line 267, in renames
shutil.move(old, new)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 302, in move
copy2(src, real_dst)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 131, in copy2
copystat(src, dst)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 103, in copystat
os.chflags(dst, st.st_flags)
OSError: [Errno 1] Operation not permitted: '/var/folders/bc/hwpn9sqn5352xcbst0p89snm0000gn/T/pip-OQKY_a-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy-1.8.0rc1-py2.7.egg-info'
My pip version is :
pip 9.0.1 from /Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg (python 2.7)
I am using Mac 10.12.6
How can I fix this?

The permission error you're seeing is due to the fact that your site-packages directory (where pip is trying to install to) is protected and owned by an administrator.
Where is the site packages directory
It depends on your Python distribution. That's why I asked for you to include the output of the $ which python command (from the command line), and the >>> import site; site.getsitepackages() command (from Python). These will help you figure out where your site packages directory is. Also, running pip -v will print the directory to which pip is trying to install.
It is important to note that Macs come with a system version of Python. You should not install any packages accessible to this system version of Python, because it is used by the system and you can potentially interfere with its normal workings and/or install malicious versions of packages that will do bad things when the system uses that Python (e.g., if someone made a nasty egg and called it "os" and put it in the site-packages directory for your system Python, then the next time the system python version ran "import os" it would import the nasty egg.)
Why overriding with sudo pip install is not a good solution
While running your pip install command as sudo would probably solve the problem, this is a short-term fix that is very bad for the reasons mentioned in my comment in #Nenand's answer - running pip install X downloads the egg for X from the internet, and runs python setup.py build && python setup.py install on whatever happens to be in the setup.py. While it's easy to be lulled into complacency because, hey, it works!, it's also naive to trust strangers on the internet (and let's face it, the people who make pip packages are strangers to you).
EDIT: I didn't even think of this, but there is also the potential that you mis-type a library, or guess the wrong name, and end up with one you REALLY didn't want (typosquatting). See this BleepingComputer article on typosquatting pip packages.
The proper solution
The proper solution is to figure out why your site-packages directory is protected, and either fix its permissions, or change the site-packages directory pip is using.
I suspect the reason you're seeing this permissions error is that pip is trying to install to the system Python's site-packages directory, which is owned by root for a good reason. The system Python is located at /usr/bin/ and its site-packages directory is located at the following directories:
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages'
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/site-python'
'/Library/Python/2.7/site-packages'
My advice, in this case, is to install Python using Homebrew, Python.org, or Canopy, which will keep your snakes in separate cages and will leave the system Python alone. On my system, python is the system python, python2 is Python version 2, and python3 is Python version 3. This also makes it more clear which version I'm using. I also do not have a pip command - not on my path - so I can't accidentally install something to my system Python. I have to explicitly call pip2 or pip3. Using Homebrew Python will do all of this for you, so that's my most-favored recommendation of the many Python distributions available.

Python pip broken - installing Pelican

I'm trying to install Pelican for python using pip install pelican, but I get these errors. I use Python 2.7.9 bundled with Mac OSX
Exception:
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/pip-7.1.2-py2.7.egg >/pip/basecommand.py", line 211, in main
status = self.run(options, args)
File "/Library/Python/2.7/site-packages/pip-7.1.2-py2.7.egg
/pip/commands /install.py", line 311, in run
root=options.root_path,
File "/Library/Python/2.7/site-packages/pip-7.1.2-py2.7.egg/pip/req
/req_set.py", line 646, in install
**kwargs
File "/Library/Python/2.7/site-packages/pip-7.1.2-py2.7.egg/pip/req
/req_install.py", line 803, in install
self.move_wheel_files(self.source_dir, root=root)
File "/Library/Python/2.7/site-packages/pip-7.1.2-py2.7.egg/pip/req
/req_install.py", line 998, in move_wheel_files
isolated=self.isolated,
File "/Library/Python/2.7/site-packages/pip-7.1.2-py2.7.egg/pip/wheel.py",
line 339, in move_wheel_files
clobber(source, lib_dir, True)
File "/Library/Python/2.7/site-packages/pip-7.1.2-py2.7.egg/pip/wheel.py", >line 310, in clobber
ensure_dir(destdir)
File "/Library/Python/2.7/site-packages/pip-7.1.2-py2.7.egg/pip/utils
/__init__.py", line 71, in ensure_dir
os.makedirs(path)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/Library/Python/2.7/site-packages/pelican'
Evens-MacBook-Pro:~ even$

Your normal user cannot install to the system-wide Python package directory. Instead of using sudo, install the package locally for your user account:
pip install --user pelican

That error occured because you don't have permission to write on that system directory. Hence the OSError: [Errno 13] Permission denied.
If you want to install it system wide, you have to run pip with superuser rights in order to gain permission:
sudo pip install pelican
Although, that wouldn't be necessary -- nor a good practice --, for instance, if you are installing inside a virtualenv.
Another possibility, as referred in this answer pointed out by #pherris, is to append the --user flag to the pip command. This would install the package only to the current user:
pip install pelican --user

Modifying the system-level installation of Python (or Ruby, or Perl...) can turn into a mess, and I have had my work there trashed in the past by system updates. Apple doesn't really guarantee your local changes will be preserved after an OS update.
You are likely better off installing your own versions using something like Homebrew, which installs into /usr/local.
Homebrew has Python 2.x and 3.x available (and they can co-exist very well). It keeps up with new releases better than Apple does. Homebrew also has many other Unix things that you may or may not find useful.
After installing Homebrew, installing Python is simple:
brew install python
brew install python3
Make sure /usr/local/bin is early in your $PATH. It also provides you with pip and pip3 to install Python package for both 2.x and 3.x.

Exception Error when Installing NumPy with pip

I have been trying to install NumPy and have been have a brutal time with it. I keep getting an exception error no matter what I try. I used the command
$pip install numpy
but it threw this error
Exception:
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/pip-6.1.1-py2.7.egg/pip/basecommand.py", line 246, in main
status = self.run(options, args)
File "/Library/Python/2.7/site-packages/pip-6.1.1-py2.7.egg/pip/commands/install.py", line 352, in run
root=options.root_path,
File "/Library/Python/2.7/site-packages/pip-6.1.1-py2.7.egg/pip/req/req_set.py", line 693, in install
**kwargs
File "/Library/Python/2.7/site-packages/pip-6.1.1-py2.7.egg/pip/req/req_install.py", line 817, in install
self.move_wheel_files(self.source_dir, root=root)
File "/Library/Python/2.7/site-packages/pip-6.1.1-py2.7.egg/pip/req/req_install.py", line 1018, in move_wheel_files
isolated=self.isolated,
File "/Library/Python/2.7/site-packages/pip-6.1.1-py2.7.egg/pip/wheel.py", line 269, in move_wheel_files
clobber(source, dest, False, fixer=fixer, filter=filter)
File "/Library/Python/2.7/site-packages/pip-6.1.1-py2.7.egg/pip/wheel.py", line 215, in clobber
shutil.copyfile(srcfile, destfile)
File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 83, in copyfile
with open(dst, 'wb') as fdst:
IOError: [Errno 13] Permission denied: '/usr/local/man/man1/nosetests.1'
Just to check, I used import in Python to see if it got anything, it also threw an error though. I have no idea what is going on.
$pip install scipy
gave me no problems. Any help is appreciated! I can't seem to find anything on how to fix this.
Thanks!

Homebrew tries to leave /usr/local writable, so you don't need sudo. See the FAQ for details.
However, if you accidentally use sudo once—or if some other installer you run installs stuff into /usr/local that doesn't realize you wanted it Homebrew-style—then you'll start getting errors, when your Homebrew stuff attempt to modify files that were saved with sudo.
There's a particular problem if you try to use both Apple's pre-installed Python 2.7 and a Homebrew Python 2.7: they both want to install scripts to /usr/local/bin, man pages to /usr/local/man, etc. And Apple's wants to use sudo.
So, most likely, you did a sudo pip install nose for Apple's Python in the past, and now pip install nose for Homebrew's Python is trying to overwrite its files and doesn't have permissions to do so. (Or maybe not nose itself, but something else that requires nose without you realizing it.)
Using sudo with the Homebrew Python will just make the problem worse; don't do that.
The right solution is to either:
Not use a third-party Python 2.7, Homebrew or otherwise, and just stick with Apple's (or upgrade to Python 3; then there's usually no conflict with Apple's 2.7…), or
Never touch Apple's Python 2.7, and only use the other one.
But at this point, you've already screwed things up, and I doubt you want to reinstall your OS from scratch, right?
You can fix things by brew uninstall python for the former, or by uninstalling everything you installed with Apple's Python for the latter. (You can't uninstall Apple's Python; that would break the OS, and the next OS update would just undo it anyway…) And then, either way, you'll probably want to reinstall every package you need for whichever Python you chose to go with, to be safe.
Or, for a quick&dirty solution, every time you get an error like this, you can either delete the conflicting file (sudo rm /usr/local/man/man1/nosetests.1) or make it overwritable (sudo chmod a+w /usr/local/man/man1/nosetests.1); then, your pip will work. Until the next error, which you can fix the same way.

Just run cmd as Admin it worked for me.

How to make source-controlled pip packages fulfill other packages' requirements

When a package is installed from a repository, pip freeze yields a repository path for that package rather than a package name/version. Example:
-e git+https://github.com/ryneeverett/Python-Markdown.git#11f0b010395a86eac93db0816bcf984639b839e9#egg=Markdown-master
When such a package is required by another application, it seems to be unrecognized. Example:
$ hyde gen
Traceback (most recent call last):
File "/home/ryne/.virtualenvs/DEV/bin/hyde", line 5, in <module>
from pkg_resources import load_entry_point
File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 2675, in <module>
def extras(self):
File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 552, in resolve
if item not in self.entry_keys:
pkg_resources.DistributionNotFound: Markdown==2.3.1
How should such issues be avoided?

I can't confirm if this would have solved my ancient issue, but (having looked through pip's source code) I'm pretty certain of what the problem was:
I was installing from git because I wanted the latest development version plus my modifications. Hyde (a rarely maintained third party package) pinned the markdown version, which was almost certainly several releases behind. So the problem was that the markdown version specified in the setup.py of my fork was not in fact 2.3.1.
To quote my own answer to another question:
Pip decides whether a requirement is met solely based on the version number (in setup.py).