I am new to setuptools on python.
I added a package 'numpy' and 'tensorflow' to install_requires list and running python setup.py install. It does not install due to SSL issue. We use self signed SSL for https based urls.
In case of pip for individual packages I can use --cert option. As I know setuptools use pip indirectly. If so, is there anyway to force it to use --cert option when setup.py is ran?
It turns out that we have actually two problems here.
The problem about tensorflow is related some difference between egg and wheel packages.
To solve the problem about numpy and other packages it is enough to add the following line into ~/.pip/pip.conf file:
[global]
cert=/path/to/cert.crt
If necessarry you may add any other option values (including proxy) with same format.
Related
I would like to use the Python library pyhash in my project. The dependencies are managed by Poetry. If I add pyhash as a dependency, I get a build error: error in pyhash setup command: use_2to3 is invalid.
This is a well-known bug due to setuptools > 58.0.0 not supporting use_2to3 anymore. In a non-Poetry setup, the fix is easy. Just downgrade setuptools to <= 58.0.0: pip3 install setuptools==58.0.0.
However, in a Poetry project, I could not make this work. I added setuptools=58.0.0 as a dependency, but when I install my project I still get the use_2to3 error. I assume that poetry still uses a setuptools>58.0.0.
How can I fix this?
I found a workaround for my problem. In the case of pyhash, the dependency on use_2to3 has already been removed in the master branch. This fix has unfortunately not been released yet. However, it is possible for pip and also for poetry to install directly from a github repository. Any ref can be specified, so branches, tags and also individual commits.
The workaround with poetry means you have to add the pyhash dependency with the git repository as source:
poetry add git+https://github.com/flier/pyfasthash.git#20a53f9bb7bf15f98e3e549f523b49e1e0f62e15
One can also specify master, but this is not advisable, as any branch is a moving target and will lead to non-reproducable releases.
I created a private package for my employer. Since I’m forbidden to upload it to PyPI (it’s proprietary), I uploaded it to the packages index for my project on our private Gitlab hub. I can install it manually with:
$ pip install my-package --extra-index-url https://__token__:my-token-xxx#gitlab.company-domain.com/api/v4/projects/123456/packages/pypi/simple
Now I also want setuptools to be able to find it when listed in the install_requires argument to setup(). I tried:
setup(
install_requires=[
f"my-package # https://__token__:{API_TOKEN}#gitlab.company-domain.com/api/v4/projects/123456/packages/pypi/simple",
...
],
...
pip install -e . results in
ERROR: HTTP error 404 while getting https://__token__:****#gitlab.company-domain.com/api/v4/projects/123456/packages/pypi/simple
This is different than
my-package # git+https://user:password#gitlab.company-domain.com/..../my-package.git
That works, but I want to be able to download it as a pre-built wheel.
I’m not sure whether this is a setuptools issue or a Gitlab issue. The 404 response tells me that it might be a gitlab issue, yet the same URI works perfectly when used with the pip install CLI command.
This question is similar to Include python packages from gitlab's package registry and other external indexes directly into setup.py dependencies, but I don't think that one got sufficient response. I posted the same question to discuss.python.org, but that discussion is old and I think I might get a quicker response here.
I also found this response to a similar question, which wasn't encouraging. It recommends Poetry or Pipenv. I've tried both, and found each to be excruciatingly slow when resolving dependencies, so I fell back on setuptools.
Only include the package name in install_requires. Then, configure your (extra) index URL in your pip configuration (either environment variables or pip.conf/.pypirc or CLI argument). Then using pip install as normal will work.
For example:
In setup.py:
# ...
install_requires=[
'my-package-name',
# ...
],
# ...
Then the install command (assuming the environment variable API_TOKEN exists):
GITLAB_INDEX="https://__token__:${API_TOKEN}#gitlab.company-domain.com/api/v4/projects/123456/packages/pypi/simple"
pip install --extra-index-url "${GITLAB_INDEX}" -e .
I've installed python 2.7.14 on my 1and1 shared hosting server using the following tutorial: http://geeksta.net/geeklog/python-shared-hosting/
it works and has a bunch of the modules that I use. However, i'd like to have numpy and pandas on this installation as well.
I'm having all sorts of issues, but the most basic one is when typing:
pip install numpy
which leads to the following error:
Could not fetch URL https://pypi.python.org/simple/numpy/: There was a problem confirming the ssl certificate: Can't connect to HTTPS URL because the SSL module is not available. - skipping
Could not find a version that satisfies the requirement numpy (from versions: )
No matching distribution found for numpy
So it looks like maybe it's failing because I don't have SSL installed, but all the suggestions for installing SSL seems to require sudo (which I can't access root on a shared host).
A second question is that I'm wondering about is if I need to use a virtualenv or not. I'm new to this but it seems like it will allow me to install things, but again I tried to install virtualenv (as stated in the tutorial):
pip install virtualenvwrapper
but I get the same sort of SSL error:
Could not fetch URL https://pypi.python.org/simple/virtualenvwrapper/: There was a problem confirming the ssl certificate: Can't connect to HTTPS URL because the SSL module is not available. - skipping
Could not find a version that satisfies the requirement virtualenvwrapper (from versions: )
No matching distribution found for virtualenvwrapper
any help that can direct me in the right direction to get these modules installed would be greatly appreciated. Thanks.
Updated to ask additional approach Given the issue seems to be not being able to download a python package on a https site (b/c SSL isn't available), is it possible for me to download the package manually using my webbrowser then FTPing the file into my server filesystem and then using SSH commands to install the modules? If so, any tips would be much appreciated.
Okay, I finally figured it out. Basically, downloading each package manually from their various websites and ftp-ing them to the shared host and then unzipping and installing them manually is what worked for me. This avoids having to connect to an SSL site to download the file on the shared host.
for numpy:
I downloaded the source file (numpy-1.14.2.zip) for numpy from here: https://pypi.org/project/numpy/#files
FTP that to the python folder in my shared host.
unzipped it
unzip numpy-1.14.2.zip
then ran setup.py from inside the numpy-1.14.2 folder
python setup.py install
Then I repeated this for pandas and it's dependencies:
setuptools
NumPy: 1.9.0 or higher
python-dateutil: 1.5 or higher
pytz: Needed for time zone support
Edit: Yes I know this question already exists, except my question is a bit different and none of the solutions fixed it.
I do most of my Python stuff when I'm at work and not on my personal machine, but I decided to install it on my personal computer as well. I fresh installed python 3.6.1, and created a virtual environment with virtualenv. Then within the virtualenv I tried to pip install urllib (or any module) and I received the error:
(pdbot) C:\Users\user\Documents\pdbot>pip install urllib
Collecting urllib
Using cached urllib-1.21.1.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\user\AppData\Local\Temp\pip-build-50tn0wlb\urllib\setup.py", line 191
s.connect((base64.b64decode(rip), 017620))
^
SyntaxError: invalid token
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in C:\Users\user\AppData\Local\Temp\pip-build-50tn0wlb\urllib\
I read elsewhere that this error had something to do with setuptools not being properly installed. So I ran this to attempt to fix the issue:
easy_install -U setuptools
I ended up receiving an even weirder error next:
(pdbot) C:\Users\zeke\Documents\pdbot>easy_install -U setuptools
Searching for setuptools
Reading https://pypi.python.org/simple/setuptools/
Downloading https://pypi.python.org/packages/a9/23/720c7558ba6ad3e0f5ad01e0d6ea2288b486da32f053c73e259f7c392042/setuptools-36.0.1.zip#md5=430eb106788183eefe9f444a300007f0
Best match: setuptools 36.0.1
Processing setuptools-36.0.1.zip
Writing C:\Users\zeke\AppData\Local\Temp\easy_install-jhg1val_\setuptools-36.0.1\setup.cfg
Running setuptools-36.0.1\setup.py -q bdist_egg --dist-dir C:\Users\zeke\AppData\Local\Temp\easy_install-jhg1val_\setuptools-36.0.1\egg-dist-tmp-8apak7kn
warning: no files found matching '*' under directory 'setuptools\_vendor'
Copying setuptools-36.0.1-py3.6.egg to c:\users\zeke\documents\pdbot\lib\site-packages
Adding setuptools 36.0.1 to easy-install.pth file
Installing easy_install-script.py script to c:\users\zeke\documents\pdbot\Scripts
Installing easy_install.exe script to c:\users\zeke\documents\pdbot\Scripts
error: [WinError 5] Access is denied: 'c:\\users\\zeke\\documents\\pdbot\\Scripts\\easy_install.exe'
This looks like a permissions error, but I ran these both in an administrator command prompt (Windows 10) and got the same result. I am the only user on this computer and I have all admin permissions. Is this virtualenv causing an issue? How do I remedy it?
EDIT: I was able to fix the permissions issue by leveraging the python executable like so:
python -m easy_install -U setuptools
But it didn't fix the python setup.py egg_info issue. I still get this error message when trying to pip install anything:
Command "python setup.py egg_info" failed with error code 1 in C:\Users\user\AppData\Local\Temp\pip-build-50tn0wlb\urllib\
I have tried both python -m pip install urllib and pip install urllib and neither work.
I had the same problem when trying to install urllib, but after doing a pip search urllib, I discovered that the problem was due to the version of urllib. From the search:
$ pip search urllib
...
> urllib5 (5.0.0) - Just increment the number and create a new lib. Never fix the original one.
At the end, a simple
pip install urllib5
within an elevated shell solved it.
Your problem has to do with permissions. The related/similar tools setup_tools, easy_install, and pip all tend to set a default set of permissions on files and folders they try to create in the package installation folder(s), rather than trying to match access permissions of the location they're installing in.
On Linux systems, where files and folders individually have permissions, this is frequently bypassed with the sudo command. On Windows, the equivalent is to run the installer as an Administrator. Since you're in the console, you have to open a console with Administrator privileges to run the pip command in.
Notable under Windows, the modules installed with pip from an Administrator console are still accessible to all users of the system that have the proper path in the PYTHONPATH system environment variable. Under Linux however, the problem is exacerbated by the fact that the files themselves may not be created with read and execute access for other users and may need to have their permissions manually modified after installation.
WARNING: urllib vs urllib2 vs urllibx
Both other answers claim that the problem is you're not specifying the correct "version" of the module in the call to pip. Neither is correct, as the error clearly indicates an installation folder access permissions violation causing the failure, but they also incorrectly recommended VERY unsafe behavior.
pip install urllib != pip install urllib5 these are two completely different packages.
The documentation for pip (https://packaging.python.org/tutorials/installing-packages/#id17) clearly says the way to specify a module version explicitly is pip install 'urllib==5'.
As part of how the package management engine implemented by pip works, running the command pip install urllib will always try to use the latest version of the urllib package, so you shouldn't need to specify the version unless you have some reason that you need a very specific version of the module.
There are two points to make in order to answer your question:
1. You are lucky you did not install that package!
The package you were trying to install was a maliciously created python package that was designed to look like a real package (in this case urllib3). If you had installed it, the package would have operated as normal except it would have sent some basic information about the system on which you installed the package to a URL (you can see more details on this here). You can read more about this fake package at either of the following links:
https://app.threatconnect.com/auth/incident/incident.xhtml?incident=5256822&owner=Common%20Community (you can sign up for a free account to view this one)
http://www.nbu.gov.sk/skcsirt-sa-20170909-pypi/index.html
Sending basic information about your systems to an unknown source isn't the worst thing you could do, but is certainly something you want to avoid when possible.
2. To properly install a package...
Specifically urllib:
To install urllib, you need to specify the version of the package you would like to install. For example, pip install urllib3.
Any package in general:
As #Elisabete Coelho suggested, you can use the pip search <package-name> feature to view the available packages. This is not perfect, however, as it may list malicious libraries like the one you were trying to install. A good guideline is that you should follow the installation instructions in a package's documentation closely to avoid any unforeseen issues. This is just an unfortunate necessity of living in a world where people make pretend python packages.
My python package footools needs html5lib via install_requires in setup.py.
setup.py develop fails
Installing via setup.py develop fails:
cd src/footools/
python setup.py develop
Processing dependencies for footools==2016.205
Searching for html5lib==0.9999999
Reading https://source.example.com/pypi/simple/html5lib/
Download error on https://source.example.com/pypi/simple/html5lib/:
[Errno 185090050] _ssl.c:354: error:0B084002:x509
certificate routines:X509_load_cert_crl_file:system lib --
Some packages may not be found!
Couldn't find index page for 'html5lib' (maybe misspelled?)
pip works
But direct download works:
bar#workdevel123:~/src/footools> pip install html5lib==0.9999999
/home/bar/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79:
InsecurePlatformWarning: A true SSLContext object is not available.
This prevents urllib3 from configuring SSL appropriately
and may cause certain SSL connections to fail.
For more information, see
https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
Collecting html5lib==0.9999999
/home/bar/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79:
InsecurePlatformWarning: A true SSLContext object is not available.
This prevents urllib3 from configuring SSL appropriately and
may cause certain SSL connections to fail.
For more information,
see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
Downloading https://source.example.com/pypi/packages/html5lib-0.9999999.tar.gz
Requirement already satisfied (use --upgrade to upgrade):
six in /usr/lib/python2.7/site-packages (from html5lib==0.9999999)
Installing collected packages: html5lib
Running setup.py install for html5lib
Successfully installed html5lib-0.9999999
Questions
What is the difference between these two methods?
Why are they different?
What is the correct way to install a dependency in python?
setup.py
The setup.py is not special:
import setuptools
setuptools.setup(
name='foo',
version='2016.210',
long_description=open('README.txt').read(),
packages=setuptools.find_packages(),
install_requires=[
# about twenty packages before this line
'html5lib==0.9999999'
],
include_package_data=True,
entry_points={
'console_scripts': [
'foo=foo.utils.bar:main',
],
},
)
python setup.py develop, or setuptools in general uses easy_install to satisfy dependencies which in turn uses urllib2 whereas pip uses requests. See here for easy_install vs pip.
pip is more modern and among other things has capability to uninstall packages and complies with PEP 438 -- Transitioning to release-file hosting on PyPI. You can achieve the same thing as python setup.py develop with pip install -e src/footools/, note if the project path is in the current directory use, ./footools.
The requests package bundles CA certs in the package itself, python -c 'import pip;print(pip.download.requests.certs.where())'.
setuptools uses system installed CA certs python -c 'from setuptools import ssl_support;print(ssl_support.cert_paths)'.
You have to update system installed CA certs using tools like update-ca-certificates for Ubuntu to either update CA certs automatically or download from https://curl.haxx.se/docs/caextract.html and install into one of the paths shown by setuptools or set setuptools.ssl_support.cert_paths to an empty sequence like [] and do pip install certifi.
Calling setuptools.ssl_support.find_ca_bundle() will reveal the location of CA certs.
setuptools is a collection of enhancements to the Python distutils (for Python 2.6 and up) that allow developers to more easily build and distribute Python packages, especially ones that have dependencies on other packages.
So, among other stuff, you can create packages that can be upload to Pypi, and later installed using pip (therefore distibuting your module).
That said, they actually should not be that different in the installation part. You are running the develop mode, so maybe you have to fiddle a bit with the directories or fix the authorization error.
In the Development Mode the project is deployed into a staging area (in some way similar to the process of a virtual environment)
deployment is done in such a way that changes to the project source are immediately available in the staging area(s), without needing to run a build or install step after each change.
Meaning also everything will be available for that python interpreter. It can be later unstaged.
I noticed that html5lib is being fetched from different places: /pypi/simple/in one case and /pypi/packages/ in the other.
dependency_links
A list of strings naming URLs to be searched when satisfying dependencies. These links will be used if needed to install packages specified by setup_requires or tests_require.
Back to the problem I think it is most probably the ssl issue, since in pip it handled graciously (i.e., nice warning and there is some kind of workaround), but the same does not happen with setuptools. If there is an error in the request that is not handled then Couldn't find index page for 'html5lib' could be because of that.
What is the difference between these two methods?
Nothing terribly important to you as a user, aside from their different user interfaces. They are two stops on the scenic historical train ride of Python package management. There were others along the way.
Why are they different?
Back in the day, Python didn't ship with a package management system. Third party solutions filled the void. They were designed by different people at different moments in time. If you look at other programming languages, you sometimes see similar stories; sometimes you see happier stories; sometimes more tragic.
What is the correct way to install a dependency in python?
Both of these methods are technically correct. Pip is the more modern method, and in my experience it is both more popular and handier to work with. As of Python 3.4 and up, Pip has been included in the CPython distribution and is officially 'preferred'. So you can see which way the wind is blowing.