TL/DR
I had a horrible puzzle to solve that didn't seem to be tackled anywhere in the web, so now I'm posting it here in case anybody else has the same trouble. Is is already solved, see my answer below.
It was related to pip install, not installing those packages that missed a wheel file.
Long explanation.
When running docker-compose up everything seemed to work like a charm in the building images stage, no errors reported... until the containers where started... then I got some weird errors like this one:
##Traceback (most recent call last):
File "/usr/local/bin/flask", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.7/dist-packages/flask/cli.py", line 894, in main
cli.main(args=args, prog_name=name)
File "/usr/local/lib/python3.7/dist-packages/flask/cli.py", line 557, in main
return super(FlaskGroup, self).main(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1132, in invoke
cmd_name, cmd, args = self.resolve_command(ctx, args)
File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1171, in resolve_command
cmd = self.get_command(ctx, cmd_name)
File "/usr/local/lib/python3.7/dist-packages/flask/cli.py", line 500, in get_command
self._load_plugin_commands()
File "/usr/local/lib/python3.7/dist-packages/flask/cli.py", line 496, in _load_plugin_commands
self.add_command(ep.load(), ep.name)
File "/usr/local/lib/python3.7/dist-packages/pkg_resources/_init_.py", line 2464, in load
self.require(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/pkg_resources/_init_.py", line 2487, in require
items = working_set.resolve(reqs, env, installer, extras=self.extras)
File "/usr/local/lib/python3.7/dist-packages/pkg_resources/_init_.py", line 777, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'alembic>=0.6' distribution was not found and is required by the application
It seemed that many packages were missing even though I didn't run into any errors when docker was installing them.
And more weird: when entering the container with docker run --rm -it --entrypoint bash <service-name> I could manually install all missing packages using pip3 install -r /var/www/webapp/requirements.txt.lock with no problem. And all errors would be gone. This is the exact same command that my Dockerfile uses in order to install python packages when building it's image.
We had no clue on:
Why this was happening now, since it worked until at least a couple
of weeks ago for the last 2 years!
Why did dockerfile fail but manually running exactly the same command worked.
Why only certain packages were missing, and no errors were raised for them.
How to fix this.
I started my research, took many hours but to summarize:
It seemed that the missing packages didn't have a wheel file in their pypi repository. So pip was falling into "legacy" direct installation but only for those packages, that's why the others were installed properly.
I found that many of these missing packages, like alembic, do have a .whl file but in later versions, not the one we were using: 0.9.9.
It seems that the package Wheel is needed to build a .whl file for those packages that do not include them in their pypi repository. But doing a pip list inside the container showed that wheel was indeed already installed...
I tried manually deleting the wheel package, and running pip3 install -r /var/www/webapp/requirements.txt.lock again... voilá, the same errors that docker reported. the packages without a wheel file were not installed. So I found out that wheel package:
has two different roles:
1- A setuptools extension for building wheels that provides the
bdist_wheel setuptools command
2- A command line tool for working with wheel files
From those 2, the first point seemed to be what was missing from happening.
5. Long story short, in my dockerfile, before installing the requirements, I installed wheel, problem solved.
But why did this happen now and not before??
I found that even if docker didn’t install wheel before running the pip install, it did have setuptools. I don’t exactly know how those two packages work, and less I understand how they cowork to get things done. But searching on setuptools changelog I found that it is updated quite frequently, almost daily.
More interesting, version v60.4.0 of 08 Jan 2022 seemed to be pretty big for the standard of the others, and it includes this as one of the changes:
#2968: Removed tmp_src test fixture. Previously this fixture was copying all the files and folders under the project root, including
the .git directory, which is error prone and increases testing time.
Since tmp_src was used to populate virtual environments (installing
the version of setuptools under test via the source tree), it was
replaced by the new setuptools_sdist and setuptools_wheel fixtures
(that are build only once per session testing and can be shared
between all the workers for read-only usage).
Not sure, but maybe it did change sth related to how packages without a wheel file were installed and now it wouldn't work without package Wheel doing it.
Related
Running setup.py install for anyjson ... done
Found existing installation: six 1.4.1
DEPRECATION: Uninstalling a distutils installed project (six) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
Uninstalling six-1.4.1:
Exception:
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/Library/Python/2.7/site-packages/pip/commands/install.py", line 342, in run
prefix=options.prefix_path,
File "/Library/Python/2.7/site-packages/pip/req/req_set.py", line 778, in install
requirement.uninstall(auto_confirm=True)
File "/Library/Python/2.7/site-packages/pip/req/req_install.py", line 754, in uninstall
paths_to_remove.remove(auto_confirm)
File "/Library/Python/2.7/site-packages/pip/req/req_uninstall.py", line 115, in remove
renames(path, new_path)
File "/Library/Python/2.7/site-packages/pip/utils/__init__.py", line 267, in renames
shutil.move(old, new)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 302, in move
copy2(src, real_dst)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 131, in copy2
copystat(src, dst)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 103, in copystat
os.chflags(dst, st.st_flags)
OSError: [Errno 1] Operation not permitted: '/tmp/pip-29Cml5-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six-1.4.1-py2.7.egg-info'
Snippet of error stacktrace.
While running requirements.txt on python.
Is it due to six.
Ignoring it does NOT help.
sudo -H pip install -r requirements.txt --ignore-installed six
Double requirement given: six==1.10.0 (from -r requirements.txt (line 107)) (already in six, name='six')
This post does not help - https://github.com/pypa/pip/issues/3165
Sometimes, the Extras problem I'm going to describe does not affect venvs. Sometimes it does, but if using virtualenv is at all an option, you should try that before doing anything else.
So, let's assume you either can't do that, or you tried it and this is one of the cases where it doesn't actually help.
First, let me say that all of the following is terrible, terrible advice for any purpose except installing new versions of packages that Apple pre-installed in Extras with Apple's Python 2.7 in OS X 10.7-10.13. Anyone reading this who does not have exactly that problem, stop reading now.
Apple's system version of Python 2.7 comes with a nifty set of third-party packages preinstalled in:
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/
Rather than try to keep these packages up to date, they designed things so that you can easily shadow them with a newer version in your site-packages directory. Which wasn't a terrible idea.
Unfortunately, they stopped doing any serious maintenance on this a few versions of OS X back, but didn't stop shipping it. In particular, they never updated it to work with setuptools and pip, and if you install pip properly (using the get-pip.py script instead of easy_install), shadowing installs will basically never work.
If your pre-installed easy_install is still working, it should still work to install shadowing packages. Do not use it for anything else, but for this specific purpose, it's the right tool. But it doesn't work for every package on every OS X version, so it's a bit of trial and error. You may want to back up Extras and site-packages first just in case.
The way to test it (assuming easy_install succeeds, of course—if it fails with a slew of errors, it obviously didn't work… and it's time to restore your backups) is to start Python, import six, and look at six.__version__ or six.__file__. If it's the new version in site-packages, you win.
If that doesn't work, there's a hacky workaround that may solve your problem: Temporarily move the file Extras/lib/python/six-1.4.1-py2.7.egg-info somewhere else, then see if pip install six successfully installs into your normal site-packages. If so, restore the egg-info file, and test the shadowing again.
If that still didn't work… well, you can install manually, but I think at this point, the pain of having two Python 2.7 installations in parallel is less than the pain of managing the one you have, so I'd consider installing another one (python.org, Anaconda, or Homebrew) and being careful to never touch the Apple one again (virtualenv can help with this).
This has been bugging me for a little while. I recently upgraded to x64 Python, and I started getting this error (example pip install).
C:\Users\<uname>\distribute-0.6.35>pip install python-qt
Collecting python-qt
Downloading python-qt-0.50.tar.gz
Building wheels for collected packages: python-qt
Running setup.py bdist_wheel for python-qt
Complete output from command C:\Python27\python.exe -c "import setuptools;__file__='c:\\users\\<uname>\\appdata\\local\\t
emp\\pip-build-vonat7\\python-qt\\setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bd
ist_wheel -d c:\users\<uname>\appdata\local\temp\tmpghy5gtpip-wheel-:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "c:\users\<uname>\appdata\local\temp\pip-build-vonat7\python-qt\setup.py", line 11, in <module>
packages=['Qt'],
File "C:\Python27\lib\distutils\core.py", line 137, in setup
ok = dist.parse_command_line()
File "C:\Python27\lib\site-packages\distribute-0.6.35-py2.7.egg\setuptools\dist.py", line 232, in parse_command_line
result = _Distribution.parse_command_line(self)
File "C:\Python27\lib\distutils\dist.py", line 467, in parse_command_line
args = self._parse_command_opts(parser, args)
File "C:\Python27\lib\site-packages\distribute-0.6.35-py2.7.egg\setuptools\dist.py", line 558, in _parse_command_opts
nargs = _Distribution._parse_command_opts(self, parser, args)
File "C:\Python27\lib\distutils\dist.py", line 523, in _parse_command_opts
cmd_class = self.get_command_class(command)
File "C:\Python27\lib\site-packages\distribute-0.6.35-py2.7.egg\setuptools\dist.py", line 362, in get_command_class
ep.require(installer=self.fetch_build_egg)
File "C:\Python27\lib\site-packages\distribute-0.6.35-py2.7.egg\pkg_resources.py", line 2027, in require
working_set.resolve(self.dist.requires(self.extras),env,installer))
File "C:\Python27\lib\site-packages\distribute-0.6.35-py2.7.egg\pkg_resources.py", line 2237, in requires
dm = self._dep_map
File "C:\Python27\lib\site-packages\distribute-0.6.35-py2.7.egg\pkg_resources.py", line 2466, in _dep_map
self.__dep_map = self._compute_dependencies()
File "C:\Python27\lib\site-packages\distribute-0.6.35-py2.7.egg\pkg_resources.py", line 2499, in _compute_dependencies
common = frozenset(reqs_for_extra(None))
File "C:\Python27\lib\site-packages\distribute-0.6.35-py2.7.egg\pkg_resources.py", line 2496, in reqs_for_extra
if req.marker_fn(override={'extra':extra}):
File "C:\Python27\lib\site-packages\distribute-0.6.35-py2.7.egg\_markerlib\markers.py", line 109, in marker_fn
return eval(compiled_marker, environment)
File "<environment marker>", line 1, in <module>
NameError: name 'sys_platform' is not defined
----------------------------------------
Failed building wheel for python-qt
Failed to build python-qt
Installing collected packages: python-qt
Running setup.py install for python-qt
Successfully installed python-qt-0.50
The package was installed fine, but I cannot build wheels. I tried re-installing distribute manually by downloading a zip and running python setup.py install. That installed wonderfuly, without a hitch. But I still have the above problem.
How can I re-define sys_platform?
Alright, I rolled back to x86 good ole 32 bit Python, and I still have the problem. This is really concerning, because I cannot reset this after re-installing. I looked at markerlib, which looks promising, but I don't know how to use it safely. Currently I am unable to install pretty much anything from PyPI, so I am giving points to increase interest.
Any help? I really want to be able to use PyPI again.
I chose the selected answer as it is the most likely to solve the problem. I myself have moved back to x86 Python, so I cannot test this myself. Therefore, I encourage future visitors to try this answer, but I have not myself been able to test it.
Might be a bug. Check out: https://bugs.python.org/
You can manually check the markers.py file and try to fix it. I think there would a reference to sys_platform that has to be changed to sys.platform
Regarding markerlib, you can try this out-
import markerlib
marker = markerlib.compile("sys.platform == 'win32'")
marker(environment=markerlib.default_environment(), override={'sys.platform':'win32'})
To fix this error, I found that installing the 0.7.3 version of distribute fixed this. I had also upgraded setuptools and pip along the way (so they may be needed as well), but after upgrading distribute this error finally went away.
Try removing pip and distribute and setuptools, and then manually install pip using get-pip.py .
Then , download setuptools from here , extract -> go inside the extracted folder in command prompt and do python setup.py install
Then , download distribute from here , extract -> go inside the extracted folder in command prompt and do python setup.py install
I ran into this issue today myself, though on OSX. I had run --upgrade as well as trying to uninstall and reinstall it completely.
Eventually, though I went into my site packages, and saw a "correct" version of setuptools (18.1) AS WELL as an older lingering version of it (completely separate version 15.1 of it). Removing it all and reinstalling setuptools fresh fixed it for me.
Hopefully this helps someone else!
I had an old verison of distribute, which didn't correctly resolve dependencies. It was fixed with
C:\Users\cshucks>pip install --upgrade distribute
Collecting distribute
Downloading distribute-0.7.3.zip (145kB)
100% |################################| 147kB 375kB/s
Collecting setuptools>=0.7 (from distribute)
Downloading setuptools-19.2-py2.py3-none-any.whl (463kB)
100% |################################| 466kB 440kB/s
Installing collected packages: setuptools, distribute
Found existing installation: setuptools 16.0
Uninstalling setuptools-16.0:
Successfully uninstalled setuptools-16.0
Found existing installation: distribute 0.6.49
Uninstalling distribute-0.6.49:
Successfully uninstalled distribute-0.6.49
Running setup.py install for distribute
Successfully installed distribute-0.7.3 setuptools-19.2
I have been trying to install NumPy and have been have a brutal time with it. I keep getting an exception error no matter what I try. I used the command
$pip install numpy
but it threw this error
Exception:
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/pip-6.1.1-py2.7.egg/pip/basecommand.py", line 246, in main
status = self.run(options, args)
File "/Library/Python/2.7/site-packages/pip-6.1.1-py2.7.egg/pip/commands/install.py", line 352, in run
root=options.root_path,
File "/Library/Python/2.7/site-packages/pip-6.1.1-py2.7.egg/pip/req/req_set.py", line 693, in install
**kwargs
File "/Library/Python/2.7/site-packages/pip-6.1.1-py2.7.egg/pip/req/req_install.py", line 817, in install
self.move_wheel_files(self.source_dir, root=root)
File "/Library/Python/2.7/site-packages/pip-6.1.1-py2.7.egg/pip/req/req_install.py", line 1018, in move_wheel_files
isolated=self.isolated,
File "/Library/Python/2.7/site-packages/pip-6.1.1-py2.7.egg/pip/wheel.py", line 269, in move_wheel_files
clobber(source, dest, False, fixer=fixer, filter=filter)
File "/Library/Python/2.7/site-packages/pip-6.1.1-py2.7.egg/pip/wheel.py", line 215, in clobber
shutil.copyfile(srcfile, destfile)
File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 83, in copyfile
with open(dst, 'wb') as fdst:
IOError: [Errno 13] Permission denied: '/usr/local/man/man1/nosetests.1'
Just to check, I used import in Python to see if it got anything, it also threw an error though. I have no idea what is going on.
$pip install scipy
gave me no problems. Any help is appreciated! I can't seem to find anything on how to fix this.
Thanks!
Homebrew tries to leave /usr/local writable, so you don't need sudo. See the FAQ for details.
However, if you accidentally use sudo once—or if some other installer you run installs stuff into /usr/local that doesn't realize you wanted it Homebrew-style—then you'll start getting errors, when your Homebrew stuff attempt to modify files that were saved with sudo.
There's a particular problem if you try to use both Apple's pre-installed Python 2.7 and a Homebrew Python 2.7: they both want to install scripts to /usr/local/bin, man pages to /usr/local/man, etc. And Apple's wants to use sudo.
So, most likely, you did a sudo pip install nose for Apple's Python in the past, and now pip install nose for Homebrew's Python is trying to overwrite its files and doesn't have permissions to do so. (Or maybe not nose itself, but something else that requires nose without you realizing it.)
Using sudo with the Homebrew Python will just make the problem worse; don't do that.
The right solution is to either:
Not use a third-party Python 2.7, Homebrew or otherwise, and just stick with Apple's (or upgrade to Python 3; then there's usually no conflict with Apple's 2.7…), or
Never touch Apple's Python 2.7, and only use the other one.
But at this point, you've already screwed things up, and I doubt you want to reinstall your OS from scratch, right?
You can fix things by brew uninstall python for the former, or by uninstalling everything you installed with Apple's Python for the latter. (You can't uninstall Apple's Python; that would break the OS, and the next OS update would just undo it anyway…) And then, either way, you'll probably want to reinstall every package you need for whichever Python you chose to go with, to be safe.
Or, for a quick&dirty solution, every time you get an error like this, you can either delete the conflicting file (sudo rm /usr/local/man/man1/nosetests.1) or make it overwritable (sudo chmod a+w /usr/local/man/man1/nosetests.1); then, your pip will work. Until the next error, which you can fix the same way.
Just run cmd as Admin it worked for me.
When a package is installed from a repository, pip freeze yields a repository path for that package rather than a package name/version. Example:
-e git+https://github.com/ryneeverett/Python-Markdown.git#11f0b010395a86eac93db0816bcf984639b839e9#egg=Markdown-master
When such a package is required by another application, it seems to be unrecognized. Example:
$ hyde gen
Traceback (most recent call last):
File "/home/ryne/.virtualenvs/DEV/bin/hyde", line 5, in <module>
from pkg_resources import load_entry_point
File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 2675, in <module>
def extras(self):
File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 552, in resolve
if item not in self.entry_keys:
pkg_resources.DistributionNotFound: Markdown==2.3.1
How should such issues be avoided?
I can't confirm if this would have solved my ancient issue, but (having looked through pip's source code) I'm pretty certain of what the problem was:
I was installing from git because I wanted the latest development version plus my modifications. Hyde (a rarely maintained third party package) pinned the markdown version, which was almost certainly several releases behind. So the problem was that the markdown version specified in the setup.py of my fork was not in fact 2.3.1.
To quote my own answer to another question:
Pip decides whether a requirement is met solely based on the version number (in setup.py).
I'm trying to install Python Fabric on Windows 7 using the guide from Getting Python and Fabric Installed on Windows.
To install PyCrypto and Fabric, i used easy_install, as recommended in the guide, but both failed, returning an chmod-error:
Using c:\python27\lib\site-packages\fabric-1.3.4-py2.7.egg
Processing dependencies for fabric
Searching for pycrypto>=2.1,!=2.4
Reading http://pypi.python.org/simple/pycrypto/
Reading http://pycrypto.sourceforge.net
Reading http://www.amk.ca/python/code/crypto
Reading http://www.pycrypto.org/
Best match: pycrypto 2.5
Downloading http://ftp.dlitz.net/pub/dlitz/crypto/pycrypto/pycrypto-2.5.tar.gz
Processing pycrypto-2.5.tar.gz
Running pycrypto-2.5\setup.py -q bdist_egg --dist-dir c:\users\birgit\appdata\local\temp\easy_install-nzrlow\pycrypto-2.5\egg-dist-tmp-_pwkm4
The command "chmod" is spelled wrong or could not be found.
Traceback (most recent call last):
File "C:\Python27\Scripts\easy_install-script.py", line 8, in <module> load_entry_point('setuptools==0.6c12dev-r88846', 'console_scripts', 'easy_install')()
File "C:\Python27\lib\site-packages\setuptools-0.6c12dev_r88846-py2.7.egg\setuptools\command\easy_install.py", line 1712, in main
[... lots and lots of lines... (if they are relevant, I'll post them)]
File "C:\Python27\lib\distutils\dist.py", line 972, in run_command cmd_obj.run()
File "setup.py", line 269, in run
RuntimeError: chmod error
I don't know much about this chmod-thing, but I thought there is no chmod in Windows?
How can i get easy_install to actually work?
I postet a similar question here, where (thanks to #J.F. Sebastian) I found a workaround to install those packages without fabric. But now I do want to know, how to actually solve the problem I'm having with easy_install.
Download and install MinGW - Minimalist GNU for Windows.
To making some Unix commands accessible from the windows console, set in your env variables:
C:\MinGW\bin;C:\MinGW\mingw32\bin;C:\MinGW\msys\1.0\bin;C:\MinGW\msys\1.0\sbin
.
Alternatively, from the console:
PATH=%PATH%;C:\MinGW\bin;C:\MinGW\mingw32\bin;C:\MinGW\msys\1.0\bin;C:\MinGW\msys\1.0\sbin
Log in as the administrator of your machine. chmod refers to permissions for accessing directories, and in this case, I have a feeling python is complaining about Windows 7's UAC (user account control). Creating directories in C:\ requires elevated permissions in Windows.
If there's something obvious happening at line 269, you could just edit the script to take out the offending line.
If not, you could install all of the dependencies, and manually install Fabric.
Also, consider using virtualenv and pip.
I can see that you are on a Python 2.x. Thus, I will suggest the method that worked for me.
Download the Pycrypto installer from : Here.
Then do the usual steps. Select the Lib/Site-packages in which you want to install it, I had two Python installations (Python 2 and 3, thus I selected Python 2/Lib/Site-packages).
Go till the end.
After successful installation, open the IDLE and type:
from Crypto.Hash import SHA256
If it works without any error, you are good to go.
Cheers.
Note : I am on a windows 8 machine.