I've done many zope/plone related projects and many bootstraps and buildouts. Since yesterday I get an error on every bootstrap I try to run. Nothing has changed on my system. I don't have any clue what could be wrong. Here's the error code:
Traceback (most recent call last):
File "bootstrap.py", line 258, in <module>
ws.require(requirement)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/../../Extras/lib/python/pkg_resources.py", line 666, in require
needed = self.resolve(parse_requirements(requirements))
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/../../Extras/lib/python/pkg_resources.py", line 569, in resolve
raise VersionConflict(dist,req) # XXX put more info here
pkg_resources.VersionConflict: (setuptools 0.6c12dev-r88846 (/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python), Requirement.parse('setuptools>=0.7'))
I started with python coding some days ago since I know several other languages but not python. Seems like this has something to do with the version of setuptools. Sadly I don't know how to update it or pin it to a specific version.
Or is it something else?
The version of setuptools you are using is 0.6
try upgrading it and everything will be fine.
For package you are installing the version of setuptools should be >=0.7
pip install setuptools --upgrade
Related
TL/DR
I had a horrible puzzle to solve that didn't seem to be tackled anywhere in the web, so now I'm posting it here in case anybody else has the same trouble. Is is already solved, see my answer below.
It was related to pip install, not installing those packages that missed a wheel file.
Long explanation.
When running docker-compose up everything seemed to work like a charm in the building images stage, no errors reported... until the containers where started... then I got some weird errors like this one:
##Traceback (most recent call last):
File "/usr/local/bin/flask", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.7/dist-packages/flask/cli.py", line 894, in main
cli.main(args=args, prog_name=name)
File "/usr/local/lib/python3.7/dist-packages/flask/cli.py", line 557, in main
return super(FlaskGroup, self).main(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1132, in invoke
cmd_name, cmd, args = self.resolve_command(ctx, args)
File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1171, in resolve_command
cmd = self.get_command(ctx, cmd_name)
File "/usr/local/lib/python3.7/dist-packages/flask/cli.py", line 500, in get_command
self._load_plugin_commands()
File "/usr/local/lib/python3.7/dist-packages/flask/cli.py", line 496, in _load_plugin_commands
self.add_command(ep.load(), ep.name)
File "/usr/local/lib/python3.7/dist-packages/pkg_resources/_init_.py", line 2464, in load
self.require(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/pkg_resources/_init_.py", line 2487, in require
items = working_set.resolve(reqs, env, installer, extras=self.extras)
File "/usr/local/lib/python3.7/dist-packages/pkg_resources/_init_.py", line 777, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'alembic>=0.6' distribution was not found and is required by the application
It seemed that many packages were missing even though I didn't run into any errors when docker was installing them.
And more weird: when entering the container with docker run --rm -it --entrypoint bash <service-name> I could manually install all missing packages using pip3 install -r /var/www/webapp/requirements.txt.lock with no problem. And all errors would be gone. This is the exact same command that my Dockerfile uses in order to install python packages when building it's image.
We had no clue on:
Why this was happening now, since it worked until at least a couple
of weeks ago for the last 2 years!
Why did dockerfile fail but manually running exactly the same command worked.
Why only certain packages were missing, and no errors were raised for them.
How to fix this.
I started my research, took many hours but to summarize:
It seemed that the missing packages didn't have a wheel file in their pypi repository. So pip was falling into "legacy" direct installation but only for those packages, that's why the others were installed properly.
I found that many of these missing packages, like alembic, do have a .whl file but in later versions, not the one we were using: 0.9.9.
It seems that the package Wheel is needed to build a .whl file for those packages that do not include them in their pypi repository. But doing a pip list inside the container showed that wheel was indeed already installed...
I tried manually deleting the wheel package, and running pip3 install -r /var/www/webapp/requirements.txt.lock again... voilá, the same errors that docker reported. the packages without a wheel file were not installed. So I found out that wheel package:
has two different roles:
1- A setuptools extension for building wheels that provides the
bdist_wheel setuptools command
2- A command line tool for working with wheel files
From those 2, the first point seemed to be what was missing from happening.
5. Long story short, in my dockerfile, before installing the requirements, I installed wheel, problem solved.
But why did this happen now and not before??
I found that even if docker didn’t install wheel before running the pip install, it did have setuptools. I don’t exactly know how those two packages work, and less I understand how they cowork to get things done. But searching on setuptools changelog I found that it is updated quite frequently, almost daily.
More interesting, version v60.4.0 of 08 Jan 2022 seemed to be pretty big for the standard of the others, and it includes this as one of the changes:
#2968: Removed tmp_src test fixture. Previously this fixture was copying all the files and folders under the project root, including
the .git directory, which is error prone and increases testing time.
Since tmp_src was used to populate virtual environments (installing
the version of setuptools under test via the source tree), it was
replaced by the new setuptools_sdist and setuptools_wheel fixtures
(that are build only once per session testing and can be shared
between all the workers for read-only usage).
Not sure, but maybe it did change sth related to how packages without a wheel file were installed and now it wouldn't work without package Wheel doing it.
I have been trying to install rasa on my Window 10 machine. But no success so far. Environment wise I have python 3.7, pip 20.3. When I give pip install rasa==1.7.0 on anaconda powershell it takes ages to download the dependencies and finally errors out as:
ERROR: Exception:
Traceback (most recent call last):
File "C:\Users\user\Anaconda3\envs\py37\lib\site-packages\pip\_internal\cli\base_command.py", line 224, in _main
status = self.run(options, args)
File "C:\Users\user\Anaconda3\envs\py37\lib\site-packages\pip\_internal\cli\req_command.py", line 180, in wrapper
return func(self, options, args)
File "C:\Users\user\Anaconda3\envs\py37\lib\site-packages\pip\_internal\commands\install.py", line 321, in run
reqs, check_supported_wheels=not options.target_dir
File "C:\Users\user\Anaconda3\envs\py37\lib\site-packages\pip\_internal\resolution\resolvelib\resolver.py", line 122, in resolve
requirements, max_rounds=try_to_avoid_resolution_too_deep,
File "C:\Users\user\Anaconda3\envs\py37\lib\site-packages\pip\_vendor\resolvelib\resolvers.py", line 445, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
File "C:\Users\user\Anaconda3\envs\py37\lib\site-packages\pip\_vendor\resolvelib\resolvers.py", line 325, in resolve
for item in self.state.criteria.items()
File "C:\Users\user\Anaconda3\envs\py37\lib\site-packages\pip\_vendor\resolvelib\resolvers.py", line 326, in <listcomp>
if not self._is_current_pin_satisfying(*item)
File "C:\Users\user\Anaconda3\envs\py37\lib\site-packages\pip\_vendor\resolvelib\resolvers.py", line 193, in _is_current_pin_satisfying
for r in criterion.iter_requirement()
AttributeError: 'NoneType' object has no attribute 'iter_requirement'
I could not find much details on this. As already dependencies were taking lot of time, I came across New resolver takes time article which states pip 20.3 takes more time. One way to solve it is to give exact versions of dependencies which I was not sure of as am quite new to rasa. I tried using older resolver as pip install rasa==1.7.0 --use-deprecated=legacy-resolver. After sometime it errored out as below:
ERROR: Could not find a version that satisfies the requirement jaxlib>=0.1.51 (from dopamine-rl->tensor2tensor~=1.14.0->rasa==1.7.0) (from versions: none)
ERROR: No matching distribution found for jaxlib>=0.1.51 (from dopamine-rl->tensor2tensor~=1.14.0->rasa==1.7.0)
To resolve this: I did dopamine install of different version as suggested here as I was not sure what version to go with. The packages started downloading and finally gave the dependency conflict error, occurred due to usage of legacy resolver.
ERROR: pip's legacy dependency resolver does not consider dependency conflicts when selecting packages. This behaviour is the source of the following dependency conflicts.
tensorflow-metadata 0.26.0 requires absl-py<0.11,>=0.9, but you'll have absl-py 0.11.0 which is incompatible.
kfac 0.2.3 requires tensorflow-probability==0.8, but you'll have tensorflow-probability 0.7.0 which is incompatible.
If we go with new resolver then it takes lot of time in resolving dependencies. Also we are not sure what all dependency versions needs to be downloaded to fine tune it.
Any help pointers here would be much appreciated.
UPDATE
I tied the following command sequence:
pip install dopamine-rl==3.0.1
pip install absl-py>=0.9
pip install rasa==1.7.0 --use-deprecated=legacy-resolver
Rasa got installed. Installed tensor-flow after this as conda install tensorflow==1.15.0. Now when tried to check tensorflow version, got the below warning and error.
The TensorFlow Distributions library has moved to TensorFlow Probability (https://github.com/tensorflow/probability). You should update all references to use `tfp.distributions` instead of `tf.distributions`.
ImportError: cannot import name 'Layer' from 'tensorflow.python.keras.engine.base_layer' (C:\Users\user\Anaconda3\envs\py37\lib\site-packages\tensorflow_core\python\keras\engine\base_layer.py)
I see you already started a discussion on Rasa Forum, let's move the conversation there! :) https://forum.rasa.com/t/unable-to-install-rasa-1-7-0/38313
Running setup.py install for anyjson ... done
Found existing installation: six 1.4.1
DEPRECATION: Uninstalling a distutils installed project (six) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
Uninstalling six-1.4.1:
Exception:
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/Library/Python/2.7/site-packages/pip/commands/install.py", line 342, in run
prefix=options.prefix_path,
File "/Library/Python/2.7/site-packages/pip/req/req_set.py", line 778, in install
requirement.uninstall(auto_confirm=True)
File "/Library/Python/2.7/site-packages/pip/req/req_install.py", line 754, in uninstall
paths_to_remove.remove(auto_confirm)
File "/Library/Python/2.7/site-packages/pip/req/req_uninstall.py", line 115, in remove
renames(path, new_path)
File "/Library/Python/2.7/site-packages/pip/utils/__init__.py", line 267, in renames
shutil.move(old, new)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 302, in move
copy2(src, real_dst)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 131, in copy2
copystat(src, dst)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 103, in copystat
os.chflags(dst, st.st_flags)
OSError: [Errno 1] Operation not permitted: '/tmp/pip-29Cml5-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six-1.4.1-py2.7.egg-info'
Snippet of error stacktrace.
While running requirements.txt on python.
Is it due to six.
Ignoring it does NOT help.
sudo -H pip install -r requirements.txt --ignore-installed six
Double requirement given: six==1.10.0 (from -r requirements.txt (line 107)) (already in six, name='six')
This post does not help - https://github.com/pypa/pip/issues/3165
Sometimes, the Extras problem I'm going to describe does not affect venvs. Sometimes it does, but if using virtualenv is at all an option, you should try that before doing anything else.
So, let's assume you either can't do that, or you tried it and this is one of the cases where it doesn't actually help.
First, let me say that all of the following is terrible, terrible advice for any purpose except installing new versions of packages that Apple pre-installed in Extras with Apple's Python 2.7 in OS X 10.7-10.13. Anyone reading this who does not have exactly that problem, stop reading now.
Apple's system version of Python 2.7 comes with a nifty set of third-party packages preinstalled in:
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/
Rather than try to keep these packages up to date, they designed things so that you can easily shadow them with a newer version in your site-packages directory. Which wasn't a terrible idea.
Unfortunately, they stopped doing any serious maintenance on this a few versions of OS X back, but didn't stop shipping it. In particular, they never updated it to work with setuptools and pip, and if you install pip properly (using the get-pip.py script instead of easy_install), shadowing installs will basically never work.
If your pre-installed easy_install is still working, it should still work to install shadowing packages. Do not use it for anything else, but for this specific purpose, it's the right tool. But it doesn't work for every package on every OS X version, so it's a bit of trial and error. You may want to back up Extras and site-packages first just in case.
The way to test it (assuming easy_install succeeds, of course—if it fails with a slew of errors, it obviously didn't work… and it's time to restore your backups) is to start Python, import six, and look at six.__version__ or six.__file__. If it's the new version in site-packages, you win.
If that doesn't work, there's a hacky workaround that may solve your problem: Temporarily move the file Extras/lib/python/six-1.4.1-py2.7.egg-info somewhere else, then see if pip install six successfully installs into your normal site-packages. If so, restore the egg-info file, and test the shadowing again.
If that still didn't work… well, you can install manually, but I think at this point, the pain of having two Python 2.7 installations in parallel is less than the pain of managing the one you have, so I'd consider installing another one (python.org, Anaconda, or Homebrew) and being careful to never touch the Apple one again (virtualenv can help with this).
I've looked at quite a lot of related questions here - no help for me. I need to repair easy_install, so that I can update distribute to a version >=0.6.28, so that I can update python matplotlib using pip and not yum.
In the process of trying to update python matplotlib from 0.99 (the default with Centos64) to 1.4.x, I got into this trouble:
pip install --upgrade matplotlib tells me this:
Complete output from command python setup.py egg_info:
The required version of distribute (>=0.6.28) is not available,
and can't be installed while this script is running. Please
install a more recent version first, using
'easy_install -U distribute'.
(Currently using distribute 0.6.10 (/usr/lib/python2.6/site-packages))
So I followed the suggestion and saw that some 0.7.x version of distribute was installed (so it seemed). Re-ran the pip command and got the exact same complaint.
Trouble with easy_install?
It was either now, or maybe after performing yum remove, then install python-setuptools that I discovered easy_install is thoroughly broken. Now when I try to run easy_install, I get this stacktrace:
Traceback (most recent call last):
File "/usr/local/bin/easy_install", line 9, in <module>
load_entry_point('distribute', 'console_scripts', 'easy_install')()
File "/usr/local/lib/python3.3/site-packages/setuptools-18.2-py3.3.egg/pkg_resources/__init__.py", line 558, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/local/lib/python3.3/site-packages/setuptools-18.2-py3.3.egg/pkg_resources/__init__.py", line 2681, in load_entry_point
raise ImportError("Entry point %r not found" % ((group, name),))
ImportError: Entry point ('console_scripts', 'easy_install') not found
Look at that - why is it referring to python3.3? I don't think I've got that installed. My default python is 2.6.6.
Any ideas? TIA.
Fixed it. The python3.3 bit was my clue. I found the python script for easy_install, in /usr/local/bin. The very first line designated /usr/bin/python3.3 to execute the script. Changed to 2.6 and then was good to go.
In the end I was not able to upgrade matplotlib from 0.99 to 1.4.3, because I could not upgrade basemap from 0.99 and some conflicts arose. Rather than punish myself with trying to build from source, I think I should simply upgrade my entire VM to Centos7 and I will then have much newer matplotlib.
When a package is installed from a repository, pip freeze yields a repository path for that package rather than a package name/version. Example:
-e git+https://github.com/ryneeverett/Python-Markdown.git#11f0b010395a86eac93db0816bcf984639b839e9#egg=Markdown-master
When such a package is required by another application, it seems to be unrecognized. Example:
$ hyde gen
Traceback (most recent call last):
File "/home/ryne/.virtualenvs/DEV/bin/hyde", line 5, in <module>
from pkg_resources import load_entry_point
File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 2675, in <module>
def extras(self):
File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 552, in resolve
if item not in self.entry_keys:
pkg_resources.DistributionNotFound: Markdown==2.3.1
How should such issues be avoided?
I can't confirm if this would have solved my ancient issue, but (having looked through pip's source code) I'm pretty certain of what the problem was:
I was installing from git because I wanted the latest development version plus my modifications. Hyde (a rarely maintained third party package) pinned the markdown version, which was almost certainly several releases behind. So the problem was that the markdown version specified in the setup.py of my fork was not in fact 2.3.1.
To quote my own answer to another question:
Pip decides whether a requirement is met solely based on the version number (in setup.py).