We installed the our root cert on the client, and the https connection works for curl.
But if we try to use pip, it fails:
Could not fetch URL https://installserver:40443/pypi/simple/pep8/:
There was a problem confirming the ssl certificate:
<urlopen error [Errno 1] _ssl.c:499: error:14090086:SSL
routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed>
The cert is on the client. See:
(foo_fm_qti)foo_fm_qti#vis-work:~$ curl -v https://installserver:40443/pypi/simple/pep8/
* About to connect() to installserver port 40443 (#0)
* Trying 127.0.0.1... connected
* Connected to installserver (127.0.0.1) port 40443 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs/
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS alert, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using DHE-RSA-AES256-SHA
* Server certificate:
* subject: C=DE; ST=Sachsen; L=Chemnitz; O=FOO-COM GmbH; OU=DV; CN=gray.foo-com.lan; emailAddress=info#foo-com.de
* start date: 2013-09-09 10:47:50 GMT
* expire date: 2019-05-24 10:47:50 GMT
* subjectAltName: installserver matched
* issuer: C=DE; ST=Sachsen; L=Chemnitz; O=FOO-COM GmbH; CN=FOO-COM Root CA; emailAddress=info#foo-com.de
* SSL certificate verify ok.
> GET /pypi/simple/pep8/ HTTP/1.1
Version: pip 1.4.1
Unfortunately pip does not use the system certs, but curl does.
I found a solution:
pip --cert /etc/ssl/certs/FOO_Root_CA.pem install pep8
This is not nice (curl and other libraries find the cert without adding a parameter) but works.
If you don't want to use the command line argument, you can set the cert in ~/.pip/pip.conf:
[global]
cert = /etc/ssl/certs/Foo_Root_CA.pem
My solution is downloading cacert.pem from http://curl.haxx.se/ca/cacert.pem
and add the path for cacert.pem to ~/.pip/pip.conf as guettli suggested
[global]
cert = /path/to/cacert.pem
For me, none of the config-file workarounds worked. I'm using pip 1.5.4 on Ubuntu 14.04
The command posted by #arjenve didn't work on my system either. I get: /usr/bin/python: No module named _vendor.requests
UPDATE
An even better solution than my first workaround is installing the certificate on the system first (for me on ubuntu this would be)
sudo cp ~/my_cert.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates
The previous automatically updates the bundle file (checking at the bottom of /etc/ssl/certs/ca-certificates.crt you should now see the same certificate as in my_cert.crt)
Now export that path into PIP_CERT and add it to your .bashrc:
echo export PIP_CERT=/etc/ssl/certs/ca-certificates.crt >> ~/.bashrc
OLDER WORKAROUND
My workaround was to create a bundle file from /etc/ssl/certs/ca-certificates.crt and my corporate's crt (just concatenated both files). And then export a variable (put that on my .bashrc) like this:
export PIP_CERT=/my/path/to/the/bundle.crt
I use:
export PIP_CERT=`python -m pip._vendor.requests.certs`
pip install pep8
PIP always validates the certificate of HTTPS connections (and all pypi packages redirect to HTTPS).
The algorithm for determining the CA file is based on 3 steps:
Look in a list of default locations for different linux distributions
(in my case this file turned out to be out of date, as I am building on a very old linux distribution)
If available, override the value found in (1) from a value in the pip.conf file, the environment or the command-line (in that order),
If both (1) and (2) did not result in a value, use a bundled file
Note that pip does not use the default SSL directories and files (from ssl.get_default_verify_paths()). But only supports a bundled CA file.
PIP does support a command-line action to list the bundled file from step 3 and that is what I use for this answer.
Related
Facing below error while communicating with https pact broker url, using pact-python lib for contract verification test against broker.
Conn close because of connect error SSL_connect returned=1 errno=0 state=error: certificate verify failed.
I have already installed required certificates on container (Docker environment) where the code is running.
Below is the sample code:
broker_opts():
return {
"broker_url": "https://ebpact01.devlab2k.testnet.rim.net:443",
"publish_verification_results": True
}
def test_user_service_provider_against_broker(server, broker_opts):
verifier = Verifier(provider="user#provider", provider_base_url=PROVIDER_URL)
success, logs = verifier.verify_with_broker(
**broker_opts,
verbose=True,
provider_states_setup_url=f"{PROVIDER_URL}/_pact/provider_states",
enable_pending=False,
)
assert success == 0
Sample Docker file :
FROM python:3.7-slim-buster
COPY ./ .
RUN pwd
RUN pip install --upgrade pip
RUN pip install -r requirements.txt
RUN apt-get update \
&& apt-get install openssl \
&& apt-get install ca-certificates
#Add certificate required to access broker
ADD 'http://certificate-url/Root-CA-1.crt' $capath/broker-root-ca1.crt
RUN update-ca-certificates
RUN export SSL_CERT_DIR=/etc/ssl/certs
RUN export SSL_CERT_FILE=/etc/ssl/certs/broker-root-ca1.crt
#invokes pytest.main method
CMD ["python3", "./test/invoke_pact_test.py"]
Sample Docker compose file :
pactverify:
build:
context: ../../
dockerfile: deploy/docker/dockerfile-pact
image: app-pact:$CI_COMMIT_SHORT_SHA
environment:
- PACT_BROKER=$PACT_BROKER
Stack trace :
Error making request - OpenSSL::SSL::SSLError SSL_connect returned=1 errno=0 state=error: certificate verify failed , attempt 1 of 3
Error making request - OpenSSL::SSL::SSLError SSL_connect returned=1 errno=0 state=error: certificate verify failed , attempt 2 of 3
Error making request - OpenSSL::SSL::SSLError SSL_connect returned=1 errno=0 state=error: certificate verify failed , attempt 3 of 3
/usr/local/lib/python3.7/site-packages/pact/bin/pact/lib/ruby/lib/ruby/2.2.0/net/http.rb:923:in connect': SSL_connect returned=1 errno=0 state=error: certificate verify failed (OpenSSL::SSL::SSLError) from /usr/local/lib/python3.7/site-packages/pact/bin/pact/lib/ruby/lib/ruby/2.2.0/net/http.rb:923:in block in connect'
from /usr/local/lib/python3.7/site-packages/pact/bin/pact/lib/ruby/lib/ruby/2.2.0/timeout.rb:74:in timeout' from /usr/local/lib/python3.7/site-packages/pact/bin/pact/lib/ruby/lib/ruby/2.2.0/net/http.rb:923:in connect'
from /usr/local/lib/python3.7/site-packages/pact/bin/pact/lib/ruby/lib/ruby/2.2.0/net/http.rb:863:in do_start' from /usr/local/lib/python3.7/site-packages/pact/bin/pact/lib/ruby/lib/ruby/2.2.0/net/http.rb:852:in start'
from /usr/local/lib/python3.7/site-packages/pact/bin/pact/lib/vendor/ruby/2.2.0/gems/pact-1.62.0/lib/pact/hal/http_client.rb:62:in block in perform_request' from /usr/local/lib/python3.7/site-packages/pact/bin/pact/lib/vendor/ruby/2.2.0/gems/pact-1.62.0/lib/pact/retry.rb:23:in until_true'
from /usr/local/lib/python3.7/site-packages/pact/bin/pact/lib/vendor/ruby/2.2.0/gems/pact-1.62.0/lib/pact/hal/http_client.rb:50:in perform_request' from /usr/local/lib/python3.7/site-packages/pact/bin/pact/lib/vendor/ruby/2.2.0/gems/pact-1.62.0/lib/pact/hal/http_client.rb:25:in get'
from /usr/local/lib/python3.7/site-packages/pact/bin/pact/lib/vendor/ruby/2.2.0/gems/pact-1.62.0/lib/pact/hal/link.rb:49:in get' from /usr/local/lib/python3.7/site-packages/pact/bin/pact/lib/vendor/ruby/2.2.0/gems/pact-1.62.0/lib/pact/pact_broker/fetch_pact_uris_for_verification.rb:57:in index'
from /usr/local/lib/python3.7/site-packages/pact/bin/pact/lib/vendor/ruby/2.2.0/gems/pact-1.62.0/lib/pact/pact_broker/fetch_pact_uris_for_verification.rb:42:in call' from /usr/local/lib/python3.7/site-packages/pact/bin/pact/lib/vendor/ruby/2.2.0/gems/pact-1.62.0/lib/pact/pact_broker/fetch_pact_uris_for_verification.rb:38:in call'
from /usr/local/lib/python3.7/site-packages/pact/bin/pact/lib/vendor/ruby/2.2.0/gems/pact-1.62.0/lib/pact/pact_broker.rb:18:in fetch_pact_uris_for_verification' from /usr/local/lib/python3.7/site-packages/pact/bin/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.1/lib/pact/provider_verifier/aggregate_pact_configs.rb:46:in pacts_for_verification'
from /usr/local/lib/python3.7/site-packages/pact/bin/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.1/lib/pact/provider_verifier/aggregate_pact_configs.rb:39:in pacts_urls_from_broker' from /usr/local/lib/python3.7/site-packages/pact/bin/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.1/lib/pact/provider_verifier/aggregate_pact_configs.rb:26:in call'
from /usr/local/lib/python3.7/site-packages/pact/bin/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.1/lib/pact/provider_verifier/aggregate_pact_configs.rb:10:in call' from /usr/local/lib/python3.7/site-packages/pact/bin/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.1/lib/pact/provider_verifier/app.rb:211:in all_pact_urls'
from /usr/local/lib/python3.7/site-packages/pact/bin/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.1/lib/pact/provider_verifier/app.rb:225:in warn_empty_pact_set' from /usr/local/lib/python3.7/site-packages/pact/bin/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.1/lib/pact/provider_verifier/app.rb:40:in call'
from /usr/local/lib/python3.7/site-packages/pact/bin/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.1/lib/pact/provider_verifier/app.rb:35:in call' from /usr/local/lib/python3.7/site-packages/pact/bin/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.1/lib/pact/provider_verifier/cli/verify.rb:49:in verify'
from /usr/local/lib/python3.7/site-packages/pact/bin/pact/lib/vendor/ruby/2.2.0/gems/thor-1.2.1/lib/thor/command.rb:27:in run' from /usr/local/lib/python3.7/site-packages/pact/bin/pact/lib/vendor/ruby/2.2.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in invoke_command'
from /usr/local/lib/python3.7/site-packages/pact/bin/pact/lib/vendor/ruby/2.2.0/gems/thor-1.2.1/lib/thor.rb:392:in dispatch' from /usr/local/lib/python3.7/site-packages/pact/bin/pact/lib/vendor/ruby/2.2.0/gems/thor-1.2.1/lib/thor/base.rb:485:in start'
from /usr/local/lib/python3.7/site-packages/pact/bin/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.1/lib/pact/provider_verifier/cli/custom_thor.rb:17:in start' from /usr/local/lib/python3.7/site-packages/pact/bin/pact/lib/app/pact-provider-verifier.rb:33:in '
opening connection to broker.url:443 ...
opened
starting SSL for broker.url:443 ...
SSL established
Conn close because of connect error SSL_connect returned=1 errno=0 state=error: certificate verify failed
opening connection to broker.url:443 ...
opened
starting SSL for broker.url:443 ...
SSL established
Conn close because of connect error SSL_connect returned=1 errno=0 state=error: certificate verify failed
opening connection to broker.url:443 ...
opened
starting SSL for broker.url:443 ...
SSL established
Conn close because of connect error SSL_connect returned=1 errno=0 state=error: certificate verify failed
Is your self signed certificate?
I would advise reading the below
https://docs.pact.io/pact_broker/advanced_topics/using-tls#connecting-to-a-pact-broker-running-over-tls
and raising an issue on the repo, as well as a repo as a sample.
We hang out over at https://slack.pact.io
Does it work if you disable ssl on the verifier?
Hey I got a couple of errors and I need help to fix them because I can't find a solution for that.
Why am I using an old af version of ubuntu?
Because I need to build android 4 and the easiest way to get the right versions of the needed packages is to use an older version of ubuntu which is confirmed to work
So please don't tell me to just upgrade to the latest ubuntu
Some commands that don't work because of a failure with ssl
when trying to access any website with firefox it returns ssl_error_no_cypher_overlap
BUT google searches are working fine
$ repo init --depth=1 -u http://github.com/CyanogenMod/android.git -b ics
Downloading Repo source from http://gerrit.googlesource.com/git-repo
fatal: Cannot get http://gerrit.googlesource.com/git-repo/clone.bundle
fatal: error unknown url type: https
fatal: cloning the git-repo repository failed, will remove '.repo/repo'
a small information how I got repo to work but didn't break apt (apt uses python2 and repo needs python3)
I aliased python as python3 so when I run python as user it refers to python3 but when I run python as root it refers to the python symlink (/usr/bin/python) which is python2
$ sudo apt-add-repository ppa:relan/exfat
Traceback (most recent call last):
File "/usr/bin/apt-add-repository", line 88, in <module>
ppa_info = get_ppa_info_from_lp(user, ppa_name)
File "/usr/lib/python2.7/dist-packages/softwareproperties/ppa.py", line 83, in get_ppa_info_from_lp
curl.perform()
pycurl.error: (60, 'server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none')
$ wget https://www.python.org/ftp/python/3.10.1/Python-3.10.1.tgz
--2021-12-25 21:43:11-- https://www.python.org/ftp/python/3.10.1/Python-3.10.1.tgz
Resolving www.python.org... 2a04:4e42:3::223, 151.101.12.223
Connecting to www.python.org|2a04:4e42:3::223|:443... connected.
OpenSSL: error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version
Unable to establish SSL connection.
$ curl -v https://www.python.org/ftp/python/3.10.1/Python-3.10.1.tgz
* About to connect() to www.python.org port 443 (#0)
* Trying 2a04:4e42:3::223... connected
* Connected to www.python.org (2a04:4e42:3::223) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS alert, Server hello (2):
* error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version
* Closing connection #0
curl: (35) error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version
Solutions I found online and tried but didn't work
checked if time is correct
update-ca-certificates -f
Tell me if you need further information/logs or anything else
Every help is appreciated!
The problem is not (yet) the certificates, it fails before validating these. Instead the versions of the SSL libraries used a simply too old. This means your software stack is way to old for today's requirements. There is no easy way to fix this.
In detail:
The openssl version in 11.10 is 0.9.8 which has no support for modern protocols like TLS 1.2 or even TLS 1.3. Similar the version of Firefox at the time of Ubuntu 11.10 did not support TLS 1.2 either (even though NSS and not openssl was used as SSL library).
I'm setting up a new server using Pytoh, tornado.
I'm going to set HTTPS.
But when server loads server certificate files, load_cert_chain function generates an error.
I purchased those certificate files from GoDaddy and it was okay on Apache Server before.
So certificate file and key file match and ca file is also okay.
python version: 3.7.4
tornado version : 4.5.2
openssl version: OpenSSL 1.1.1c 28 May 2019
ssl_ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
ssl_ctx.load_cert_chain("../server.crt", "../server.key")
ssl_ctx.load_verify_locations("../ca.crt")
ssl_ctx.verify_mode = ssl.CERT_REQUIRED
https_server = tornado.httpserver.HTTPServer(app, ssl_options=ssl_ctx)
This is error message.
ssl_ctx.load_cert_chain("../server.crt", "../server.key")
ssl.SSLError: [SSL] PEM lib (_ssl.c:3854)
I solved this problem. I use NginX server to host python server with digital signed certificates.
Introduction
I want to create a dependency from one private Python project (myproject) on another private Python project (example). Currently I'm trying to run python setup.py install and have setuptools find the example library and install it too.
Testing
I've published the dependency to a local instance of Artifactory as an egg.
I'm using pip version 19.1.1.
I'm running macOS 10.13.6.
Constructing a URL
According to the pip release notes, the following feature was added:
Allow PEP 508 URL requirements to be used as dependencies.
As a security measure, pip will raise an exception when installing packages from PyPI if those packages depend on packages not also hosted on PyPI. In the future, PyPI will block uploading packages with such external URL dependencies directly. (#4187)
I've tried adding the following to the parent project's setup.py:
install_requires=['example'],
dependency_links=['https://artifactory.company.com/api/pypi/pypi-local/simple#egg=example-0.1.0.dev27'],
Running
When running python setup.py install, I get the following output:
...
Processing dependencies for myproject==0.0.0
Searching for example
Downloading https://artifactory.company.com/api/pypi/pypi-local/simple#egg=example-0.1.0.dev27
Authenticating as me for https://artifactory.company.com/api/pypi/pypi-local/simple#egg=example-0.1.0.dev27 (from .pypirc)
error: Download error for https://artifactory.company.com/api/pypi/pypi-local/simple#egg=example-0.1.0.dev27: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:726)
So I am confused by [SSL: CERTIFICATE_VERIFY_FAILED]. If I run:
curl -v https://me:<token>#artifactory.company.com/api/pypi/pypi-local/simple#egg=example-0.1.0.dev27
I get the following:
...
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
...
GET /api/pypi/pypi-local/simple HTTP/1.1
> Host: artifactory.company.com
> Authorization: Basic abcxyz0d
> User-Agent: curl/7.54.0
> Accept: */*
>
>
< HTTP/1.1 302 Found
...
So it seems like SSL certification verification is working outside of setuptools.
If I go to the site specified in this output (https://artifactory.company.com/api/pypi/pypi-local/simple/#egg=example-0.1.0.dev27), I can see a hyperlink for example. If I click the hyperlink, it takes me to a list of hyperlinks of all the wheels and eggs I've published, looks like this:
...
example-0.1.0.dev26-py2-none-any.whl
example-0.1.0.dev27-py2-none-any.whl
example-0.1.0.dev27-py2.7.egg
...
And I can download these without any issue.
Question
How can I create a dependency from one private Python project (myproject) on another private Python project (example) using setuptools?
curl is probably using a different set of trusted certificates than Python does. Maybe your company pre-installs its own certificate(s) into the system trust store? If you don't find it there, you can fetch the certificate from the artifactory server itself.
Here's an answer about adding certificates for use by pip:
https://stackoverflow.com/a/52961564/11451509
I am newbie to vault and currently trying to enable client tls authentication (all on the only CentOS 7 node) via self-signed certificates.
Of course, first of all i ran: vault auth-enable cert
Then, I created a policy.hcl file with such contents:
path "secret/policy/test/foo" {policy="read"}
Then I did the following:
$ vault policy-write test policy.hcl
$ vault write "secret/policy/test/foo" vaule=s3cr3t policy=test
Generate client certificate:
$ openssl genrsa -out client.key 2048
$ openssl req -new -key client.key -out client.csr
$ openssl x509 -req -in client.csr -signkey client.key -out client.crt
And the final action:
$ vault write auth/cert/certs/test display_name=test policies=test certificate=#/home/vagrant/ssl/client.crt
And it all execute without any errors or warnings. Then, i wrote a simple script which must read just a one value after authentication with TLS:
import os
import hvac
client = hvac.Client(url='https://127.0.0.1:8200', cert=('client.crt', 'client.key'))
print(client.read('secret/policy/test/foo'))
But when I run it I get the following error:
requests.exceptions.SSLError: HTTPSConnectionPool(host='127.0.0.1', port=8200): Max retries exceeded with url: /v1/secret/policy/test/foo (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)'),))
Server log says:
http: TLS handshake error from 127.0.0.1:33744: remote error: tls: unknown certificate authority
Any help is highly appreciated
EDIT:
to clarify what the actual problem is, I'd like to mention that when i authenticate to the Vault using CLI:
$ vault auth -method=cert -client-cert=client.crt -client-key=client.key
# returns:
The token below is already saved in the session. You do not
need to "vault auth" again with the token.
token: 8afdb5bd-b6f1-e33e-a391-ced99fa18b5f
token_duration: 2764800
token_policies: [default test]
... it seems to be working just fine cuz the policy I've created is being attached and I can read the data from secret/policy/test/foo, but Python+HVAC approach does not work.