We Keep Coding

Why does Pip disregard configured repository with nested dependencies?

Problem
Let us say I have a completely empty Python+Pip+R (pip 19.3.1) environment on a Linux machine and I want to install the package rpy2 with pip. Since I am behind a corporate firewall I configure pip to use a private repository.
[global]
index-url = http://private.com/artifactory/api/pypi/PyPI/simple
trusted-host = private.com
Now I execute pip install rpy2 and I will get back the following error:
Couldn't find index page for 'cffi'
Download error on https://pypi.python.org/simple/
So pip tries to resolve the nested dependency by looking and installing cffi from the official PyPi repository. It completely ignores the repo I have configured.
When I run pip install cffi && pip install rpy2 one after another everything works as expected.
Here is the full error output:
ERROR: Command errored out with exit status 1:
command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-8vuadu93/rpy2/setup.py'"'"'; __file__='"'"'/tmp/pip-install-8vuadu93/rpy2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-8vuadu93/rpy2/pip-egg-info
cwd: /tmp/pip-install-8vuadu93/rpy2/
Complete output (25 lines):
Download error on https://pypi.python.org/simple/cffi/: [Errno -2] Name or service not known -- Some packages may not be found!
Couldn't find index page for 'cffi' (maybe misspelled?)
Download error on https://pypi.python.org/simple/: [Errno -2] Name or service not known -- Some packages may not be found!
No local packages or working download links found for cffi>=1.13.1
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-8vuadu93/rpy2/setup.py", line 183, in <module>
'rinterface_lib/R_API_eventloop.h']}
File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 128, in setup
_install_setup_requires(attrs)
File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 123, in _install_setup_requires
dist.fetch_build_eggs(dist.setup_requires)
File "/usr/lib/python3/dist-packages/setuptools/dist.py", line 513, in fetch_build_eggs
replace_conflicting=True,
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 774, in resolve
replace_conflicting=replace_conflicting
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 1057, in best_match
return self.obtain(req, installer)
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 1069, in obtain
return installer(requirement)
File "/usr/lib/python3/dist-packages/setuptools/dist.py", line 580, in fetch_build_egg
return cmd.easy_install(req)
File "/usr/lib/python3/dist-packages/setuptools/command/easy_install.py", line 692, in easy_install
raise DistutilsError(msg)
distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('cffi>=1.13.1')
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
Question
Is this a bug in Pip, a specific problem with rpy2 or am I missing something?
Updates
Running the following commands gives me the same error:
pip install rpy2 --no-index --find-links http://private.com/artifactory/api/pypi/PyPI/simple
pip install rpy2 --index-url http://private.com/artifactory/api/pypi/PyPI/simple
I used -vvv and it seems like the problem occurs somewhere inside of setuptools
Solution
The package rpy2 uses setuptools which again uses easy_install.py. It also works with an index_url variable. But it gets the value not from pip.config but distutils.cfg.
I identified all Python versions I have installed with find / -name "distutils". Then I added the a distutils.cfg with the following content to each of these directories:
[easy_install]
index_url = blablabla
And now it works, I execute pip install rpy2 and all missing requirements are installed in one go

I believe it could be caused by the fact that cffi is listed as setup_requires in rpy2's setup.py. Most likely because cffi is required to build the project itself before it can be installed. This kind of build dependencies are not handled by pip directly, so its index-url option has no effect.
The solution is to tell setuptools about the alternative index in a distutils configuration file
[easy_install]
index_url = https://my.index-mirror.com
References:
"Controlling setup_requires" in pip's documentation
setuptools, easy_install, and a custom pypi server
distutils configuration file

pip install produces the following error on mac: error: command 'gcc' failed with exit status 1

So I tried to install a package from PyPI with the following command:
sudo pip3 install switcheo
But it fails to install the package in the end.
Error checking for conflicts.
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2584, in version
return self._version
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2691, in __getattr__
raise AttributeError(attr)
AttributeError: _version
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 503, in _warn_about_conflicts
package_set, _dep_info = check_install_conflicts(to_install)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pip/_internal/operations/check.py", line 108, in check_install_conflicts
package_set, _ = create_package_set_from_installed()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pip/_internal/operations/check.py", line 47, in create_package_set_from_installed
package_set[name] = PackageDetails(dist.version, dist.requires())
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2589, in version
raise ValueError(tmpl % self.PKG_INFO, self)
ValueError: ("Missing 'Version:' header and/or METADATA file", Unknown [unknown version] (/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages))
Installing collected packages: scrypt, neocore, switcheo
scrypt-1.2.1/libcperciva/crypto/crypto_aes.c:6:10: fatal error: 'openssl/aes.h' file not found
#include <openssl/aes.h>
^~~~~~~~~~~~~~~
1 error generated.
error: command 'gcc' failed with exit status 1
----------------------------------------
Command "/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7 -u -c "import setuptools, tokenize;__file__='/private/tmp/pip-install-b6w97vc0/scrypt/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/tmp/pip-record-qvammwag/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/tmp/pip-install-b6w97vc0/scrypt/
This is the full output in the terminal after running the pip install line.
I found some solutions for linux, but there are not a lot of posts here for mac users.

I dealt with this problem for a while while trying to install another library that relies on openssl in a MacOS virtual env.
Mac has deprecated openssl in favor of their own SSL. So what you need to do is:
brew install openssl
or if it is already installed,
brew reinstall openssl
Now you need to to change the LD and CPP Flags. Refer to the openssl reinstall dialog. Should tell you where the commands!
openssl is keg-only, which means it was not symlinked into /usr/local,
because Apple has deprecated use of OpenSSL in favor of its own TLS and crypto libraries.
If you need to have openssl first in your PATH run:
echo 'export PATH="/usr/local/opt/openssl/bin:$PATH"' >> ~/.bash_profile
For compilers to find openssl you may need to set:
export LDFLAGS="-L/usr/local/opt/openssl/lib"
export CPPFLAGS="-I/usr/local/opt/openssl/include"
For pkg-config to find openssl you may need to set:
export PKG_CONFIG_PATH="/usr/local/opt/openssl/lib/pkgconfig"

You're either missing the OpenSSL package, or your system can't find where it is on your system. Try brew install openssl.
A more detailed answer openssl/aes.h' file not found on Mac

after spending hours, i successfullly installed.
If you are using mac and if you are using anaconda
please open python terminal from anaconda
then paste the following link and click enter.
conda install -c conda-forge/label/gcc7 wordcloud
if you need more clarity please visit following link
https://anaconda.org/conda-forge/wordcloud

I am unable to connect to MySQL on Pycharm even after installing pip and mysql connector properly

I have installed mysql connector properly and the command prompt even shows successful installation but for some reason Pycharm is showing error in installing the connector.
Coder Starts here
Collecting mysqlconnector
Using cached https://files.pythonhosted.org/packages/e4/7c/957e0c387d4c65df42ba56d5160456b0ff6f83f3cda122e18287605e3711/mysqlconnector-0.1.1.tar.gz
Installing collected packages: mysqlconnector
Running setup.py install for mysqlconnector: started
Running setup.py install for mysqlconnector: finished with status 'error'
Complete output from command "C:\Users\Farhan Hasnat\PycharmProjects\untitled1\venv\Scripts\python.exe" -u -c "import setuptools, tokenize;__file__='C:\\Users\\FARHAN~1\\AppData\\Local\\Temp\\pycharm-packaging\\mysqlconnector\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record C:\Users\FARHAN~1\AppData\Local\Temp\pip-record-hnanue0h\install-record.txt --single-version-externally-managed --compile --install-headers "C:\Users\Farhan Hasnat\PycharmProjects\untitled1\venv\include\site\python3.7\mysqlconnector":
running install
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\FARHAN~1\AppData\Local\Temp\pycharm-packaging\mysqlconnector\setup.py", line 47, in <module>
'mysqlconnector = mysqlconnector.cli:cli',
File "C:\Users\Farhan Hasnat\AppData\Local\Programs\Python\Python37-32\lib\distutils\core.py", line 148, in setup
dist.run_commands()
File "C:\Users\Farhan Hasnat\AppData\Local\Programs\Python\Python37-32\lib\distutils\dist.py", line 966, in run_commands
self.run_command(cmd)
File "C:\Users\Farhan Hasnat\AppData\Local\Programs\Python\Python37-32\lib\distutils\dist.py", line 985, in run_command
cmd_obj.run()
File "C:\Users\FARHAN~1\AppData\Local\Temp\pycharm-packaging\mysqlconnector\setup.py", line 20, in run
raise Exception("You probably meant to install and run mysql-connector")
Exception: You probably meant to install and run mysql-connector
----------------------------------------
Command ""C:\Users\Farhan Hasnat\PycharmProjects\untitled1\venv\Scripts\python.exe" -u -c "import setuptools, tokenize;__file__='C:\\Users\\FARHAN~1\\AppData\\Local\\Temp\\pycharm-packaging\\mysqlconnector\\setup.py';f=getattr**strong text**(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record C:\Users\FARHAN~1\AppData\Local\Temp\pip-record-hnanue0h\install-record.txt --single-version-externally-managed --compile --install-headers "C:\Users\Farhan Hasnat\PycharmProjects\untitled1\venv\include\site\python3.7\mysqlconnector"" failed with error code 1 in C:\Users\FARHAN~1\AppData\Local\Temp\pycharm-packaging\mysqlconnector\
You are using pip version 10.0.1, however version 18.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

"msql-connector-repackaged" should be "mysql-connector-repackaged", also try to pass in a database in your "myDB" variable (database must have already been created):
mydb = mysql.connector.connect(
host="localhost",
user="root",
password="password",
database="name of your database"
)
Other than that, what edition of pycharm are you using? From my understanding pycharm ce edition doesn't support the use of database management systems such as Oracle, MySQL, SQL etc..., you need the professional version to do so.
I've been able to build databases with SQLite in pycharm, but that's built into python. I don't know how large the data your working with is, but if it's small you can resort to SQLite instead.
https://www.jetbrains.com/help/pycharm/relational-databases.html
Also, you can try other connection packages (CAUTION: IntelliJ, creators of PyCharm, may eventually remove them in order to preserve the mysql compatibility of non-free PyCharm editions) such as: mysql-connector, mysql-connector-python, mysql-connector-dd and mysql-connector-rf.

You may need to install the Python 3 and MySQL development headers and libraries like so:
$sudo apt-get install python3-dev default-libmysqlclient-dev build-essential
Then run the pip install:
$pip install mysqlclient
Now you can add the package mysqlclient on Pycharm
Documentation: https://github.com/PyMySQL/mysqlclient-python
:)

Pip install --target=. Alfred-Workflow gives an error

I am trying to install a python library on macOS following through these instructions.
However I get an error every time I run this command : pip install --target=. Alfred-Workflow
And I always get this error for running it :
pip install --target=. Alfred-Workflow
Collecting Alfred-Workflow
Installing collected packages: Alfred-Workflow
Exception:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/usr/local/lib/python2.7/site-packages/pip/commands/install.py", line 342, in run
prefix=options.prefix_path,
File "/usr/local/lib/python2.7/site-packages/pip/req/req_set.py", line 784, in install
**kwargs
File "/usr/local/lib/python2.7/site-packages/pip/req/req_install.py", line 851, in install
self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
File "/usr/local/lib/python2.7/site-packages/pip/req/req_install.py", line 1064, in move_wheel_files
isolated=self.isolated,
File "/usr/local/lib/python2.7/site-packages/pip/wheel.py", line 247, in move_wheel_files
prefix=prefix,
File "/usr/local/lib/python2.7/site-packages/pip/locations.py", line 153, in distutils_scheme
i.finalize_options()
File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/install.py", line 264, in finalize_options
"must supply either home or prefix/exec-prefix -- not both"
DistutilsOptionError: must supply either home or prefix/exec-prefix -- not both
I tried googling and searching for this but still can't figure it out. Thank you for any help.

This question answers that (I found it when googling for the last line of your error message).
First I also got the same error message as you did, but after doing this:
$ echo "[install]
prefix=" > ~/.pydistutils.cfg
It works:
$ pip install --target=. Alfred-Workflow
Collecting Alfred-Workflow
Installing collected packages: Alfred-Workflow
Successfully installed Alfred-Workflow-1.24
Important note: it breaks normal pip install commands, so you need to rm ~/.pydistutils.cfg afterward.

I have a similar error installing python modules using pip with -t(--target) option.
The pip log show the next message:
Complete output from command /usr/bin/python -c "import setuptools, tokenize;file='/tmp/pip-build-LvB_CW/xlrd/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /tmp/pip-UNJizV-record/install-record.txt --single-version-externally-managed --compile --user --home=/tmp/tmphjBN23
and the next error:
can't combine user with prefix, exec_prefix/home, or install_(plat)base
Reading about alternate installation on python docs I see the next info
Note that the various alternate installation schemes are mutually exclusive: you can pass --user, or --home, or --prefix and --exec-prefix, or --install-base and --install-platbase, but you can’t mix from these groups.
So the command executed by pip has two mutually exclusive schemes --user and --home (i think it could be a bug on pip).
I use the --system option to avoid the error, eliminating the --user flag on the setup command.
pip install -t path_to_dir module_name --system
I don't known the aditional implications of this usage, but i think it's better than modifing the config file that messed up with normal installations.
PD: I use ubuntu 15.10 with pip 1.5.6

Virtualenv shell errors

I've just installed virtualenv (with Python 2.7.2) on my Mac, and I followed the guide here: http://virtualenvwrapper.readthedocs.org/en/latest/install.html
But I now get the following errors when I start up my shell every time:
stevedore.extension Could not load 'user_scripts': distribute
stevedore.extension distribute
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/stevedore/extension.py", line 62, in __init__
invoke_kwds,
File "/Library/Python/2.7/site-packages/stevedore/extension.py", line 74, in _load_one_plugin
plugin = ep.load()
File "/Library/Python/2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 1953, in load
if require: self.require(env, installer)
File "/Library/Python/2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 1966, in require
working_set.resolve(self.dist.requires(self.extras),env,installer))
File "/Library/Python/2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 565, in resolve
raise DistributionNotFound(req) # XXX put more info here
DistributionNotFound: distribute
stevedore.extension Could not load 'project': distribute
stevedore.extension distribute
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/stevedore/extension.py", line 62, in __init__
invoke_kwds,
File "/Library/Python/2.7/site-packages/stevedore/extension.py", line 74, in _load_one_plugin
plugin = ep.load()
File "/Library/Python/2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 1953, in load
if require: self.require(env, installer)
File "/Library/Python/2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 1966, in require
working_set.resolve(self.dist.requires(self.extras),env,installer))
File "/Library/Python/2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 565, in resolve
raise DistributionNotFound(req) # XXX put more info here
DistributionNotFound: distribute
stevedore.extension Could not load 'user_scripts': distribute
stevedore.extension distribute
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/stevedore/extension.py", line 62, in __init__
invoke_kwds,
File "/Library/Python/2.7/site-packages/stevedore/extension.py", line 74, in _load_one_plugin
plugin = ep.load()
File "/Library/Python/2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 1953, in load
if require: self.require(env, installer)
File "/Library/Python/2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 1966, in require
working_set.resolve(self.dist.requires(self.extras),env,installer))
File "/Library/Python/2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 565, in resolve
raise DistributionNotFound(req) # XXX put more info here
DistributionNotFound: distribute
I don't know if it is affecting this problem, but I am using ZSH.
I tried to install stevedore through pip (sudo pip install stevedore), but I get the following error:
sudo sh setuptools-0.6c11-py2.7.egg
Processing setuptools-0.6c11-py2.7.egg
removing '/Library/Python/2.7/site-packages/setuptools-0.6c11-py2.7.egg' (and everything under it)
Copying setuptools-0.6c11-py2.7.egg to /Library/Python/2.7/site-packages
setuptools 0.6c11 is already the active version in easy-install.pth
Installing easy_install script to /usr/local/bin
Installing easy_install-2.7 script to /usr/local/bin
Installed /Library/Python/2.7/site-packages/setuptools-0.6c11-py2.7.egg
Processing dependencies for setuptools==0.6c11
Finished processing dependencies for setuptools==0.6c11
TXSLs-MacBook-Pro% sudo pip install stevedore --upgrade
Requirement already up-to-date: stevedore in /Library/Python/2.7/site-packages
Downloading/unpacking distribute (from stevedore)
Running setup.py egg_info for package distribute
Installing collected packages: distribute
Running setup.py install for distribute
Before install bootstrap.
Scanning installed packages
Setuptools installation detected at /Library/Python/2.7/site-packages/setuptools-0.6c11-py2.7.egg
Egg installation
Patching...
Renaming /Library/Python/2.7/site-packages/setuptools-0.6c11-py2.7.egg into /Library/Python/2.7/site-packages/setuptools-0.6c11-py2.7.egg.OLD.1348764450.4
Patched done.
Relaunching...
Traceback (most recent call last):
File "<string>", line 1, in <module>
NameError: name 'install' is not defined
Complete output from command /usr/bin/python -c "import setuptools;__file__='/tmp/pip-build/distribute/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-FAPgYH-record/install-record.txt --single-version-externally-managed:
Before install bootstrap.
Scanning installed packages
Setuptools installation detected at /Library/Python/2.7/site-packages/setuptools-0.6c11-py2.7.egg
Egg installation
Patching...
Renaming /Library/Python/2.7/site-packages/setuptools-0.6c11-py2.7.egg into /Library/Python/2.7/site-packages/setuptools-0.6c11-py2.7.egg.OLD.1348764450.4
Patched done.
Relaunching...
Traceback (most recent call last):
File "<string>", line 1, in <module>
NameError: name 'install' is not defined
----------------------------------------
Command /usr/bin/python -c "import setuptools;__file__='/tmp/pip-build/distribute/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-FAPgYH-record/install-record.txt --single-version-externally-managed failed with error code 1 in /tmp/pip-build/distribute
Storing complete log in /Users/txsl/Library/Logs/pip.log
I manually installed setuptools as I couldn't install anything through pip without it.
What has gone wrong here and how can i fix it? The internet doesn't seem to have many cases of the error with stevedore. I feel rather stuck at the moment!
Many thanks!

I also use zsh and had a similar problem. I solved with this:
sudo pip install virtualenv virtualenvwrapper
I have the package python-pip installed in my Ubuntu 12.04.

Based on the error you are getting, it looks like you are having the following error: install glitch when using pip + virtualenv.
The issue is created when using the -distribute switch. The fix is (without re-running virtualenv with --distribute):
source bin/activate
wget http://python-distribute.org/distribute_setup.py
python distribute_setup.py

Had the same error message, upgrading the setuptools resolved the issue for me.
pip install --upgrade setuptools
I've found the information in this thread: http://blog.gmane.org/gmane.comp.python.virtualenv/month=20131001

I had the same problem after upgrading to OSX Mavericks. I fixed it by installing distribute globally.
sudo pip install distribute
I deactivated my current virtualenv first.