We Keep Coding

Python Requests and SSLContext

I'm trying to figure out how to specify an SSLContext with Request.
I have two functions which in theory should do the same, however the one with Requests doesn't work.
def func_OK(token):
ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH,cafile='myCA.crt.pem')
ctx.load_cert_chain(certfile='myprivate.pem')
url = 'https://my_url.com'
hdr = {"Content-Type": "application/json","Authorization":"Bearer "+token}
data = '{"filterList":[{}]}'
bdata = data.encode('utf-8')
req = urllib.request.Request(url, headers=hdr)
resp = urllib.request.urlopen(req, data=bdata, context=ctx)
content = resp.read()
data = json.loads(content.decode('utf-8'))
def func_NOK(token):
import requests
url = 'https://my_url.com'
hdr = {"Content-Type": "application/json","Authorization":"Bearer "+token}
data = '{"filterList":[{}]}'
bdata = data.encode('utf-8')
resp = requests.post(url,headers=hdr, data={"filterList":[{}]})
The only the difference between the two functions are the sslContext.
In the func_NOK, I try :
resp = requests.post(url,headers=hdr, data={"filterList":[{}]}, verify=False) - it doesn't work
resp = requests.post(url,headers=hdr, data={"filterList":[{}]}, cert=('myCA.crt.pem','myprivate.pem')) - it doesn't work
resp = requests.post(url,headers=hdr, data={"filterList":[{}]}, verify="concat_file.crt") with "concat_file.crt" file a concatenation of 'myCA.crt.pem' and 'myprivate.pem'
In any cases I have an SSL error.
For example, on my last example the error msg is :
requests.exceptions.ConnectionError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1131)
I'm just trying to use an SSLContext with Requests.

ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH,cafile='myCA.crt.pem')
ctx.load_cert_chain(certfile='myprivate.pem')
load_cert_chain loads the cert and private key for use as client certificate - which would be the cert argument with requests. cafile describes the CA it should use to verify the server certificate - which would be the verify argument for requests. Combined this would result in:
requests.post(..., cert='myprivate.pem', verify='myCA.crt.pem')

I find where my cacert.pem : /home/<soome_path>/pyEnv/myEnv/lib/python3.8/site-packages/certifi/cacert.pem
I concatenated the files :
myCA.crt.pem >> cacert.pem
myprivate.pem>> cacert.pem
then I specified the path using verify :
requests.post(...,verify='/home/<soome_path>/pyEnv/myEnv/lib/python3.8/site-packages/certifi/cacert.pem')
And I don't have the ssl error anymore.
However I retrieve an html msg instead of a json.
Maybe an issue on the parameters that I send to the endpoint.

I solved it using :
requests.post(url,headers=hdr,json={"filterList":[{}]}, cert='myprivate.pem')

SSL: CERTIFICATE_VERIFY_FAILED trying to validate a reCAPTCHA with django

I'm getting a
<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)>
When I try to validate captchas in my django project.
This is how i do it:
recaptcha_response = request.POST.get('g-recaptcha-response')
print(recaptcha_response)
url = 'https://www.google.com/recaptcha/api/siteverify'
values = {
'secret': settings.CAPTCHA_SECRET_KEY,
'response': recaptcha_response
}
data = urllib.parse.urlencode(values).encode()
req = urllib.request.Request(url, data=data)
response = urllib.request.urlopen(req) # It fails here
result = json.loads(response.read().decode())
print(result)
The site has a valid certificate, and it works on local. In the log i'm getting this:
Request Method: POST
Request URL: http://prod.xxxx.com/evalua
Which is weird because the site works in https. Its on kubernetes, could that be the problem? I really have no idea what the problem IS? I have the captcha keys correctly set up in de recaptcha admin console. And the certificate are not autosign. I use lets encrypt

Check how you build the container image for your app and if it has very old CA certificates in it. You can use something like ADD https://curl.haxx.se/ca/cacert.pem /etc/ssl/certs/cacert.pem to ensure you have the latest standard bundle. You can also switch to Requests and Certifi instead of urllib, as that embeds a copy of the current cert bundle and ensures it is used.

python requests verify SSL certificate

I'm trying to pull data from an API which is secured by SSL. I wrote a python script to pull the data. Beforehand I have to convert a .p12 file to an openSSL certificate. When I use the following code it works just fine:
# ----- SCRIPT 1 -----
def pfx_to_pem(pfx_path, pfx_password):
''' Decrypts the .pfx file to be used with requests. '''
with tempfile.NamedTemporaryFile(suffix='.pem') as t_pem:
f_pem = open(t_pem.name, 'wb')
pfx = open(pfx_path, 'rb').read()
p12 = OpenSSL.crypto.load_pkcs12(pfx, pfx_password)
f_pem.write(OpenSSL.crypto.dump_privatekey(OpenSSL.crypto.FILETYPE_PEM, p12.get_privatekey()))
f_pem.write(OpenSSL.crypto.dump_certificate(OpenSSL.crypto.FILETYPE_PEM, p12.get_certificate()))
ca = p12.get_ca_certificates()
if ca is not None:
for cert in ca:
f_pem.write(OpenSSL.crypto.dump_certificate(OpenSSL.crypto.FILETYPE_PEM, cert))
f_pem.close()
yield t_pem.name
# read some config
with open('config.json') as config_json:
config = json.load(config_json)
api_url = config['api_url']
cert = config['cert']
cert_pem_path = cert['file']
cert_key_file = cert['pass']
# make the request
with pfx_to_pem(cert_pem_path, cert_key_file) as cert:
r = requests.get(api_url, cert = cert)
Because I'm also using the same functionality to authenticate my Flask web service towards the server I split up the cert file into three files:
# ----- SCRIPT 1 -----
# get certificate
f_pem.write(OpenSSL.crypto.dump_certificate(
OpenSSL.crypto.FILETYPE_PEM, p12.get_certificate())
)
# get keyfile
f_key.write(OpenSSL.crypto.dump_privatekey(
OpenSSL.crypto.FILETYPE_PEM, p12.get_privatekey())
)
# get CA_BUNDLE
ca = p12.get_ca_certificates()
if ca is not None:
for cert in ca:
f_ca.write(
OpenSSL.crypto.dump_certificate(
OpenSSL.crypto.FILETYPE_PEM, cert
))
Then I'm running the web service with the following code:
# ----- SCRIPT 2 -----
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
context.verify_mode = ssl.CERT_REQUIRED
context.load_verify_locations(cert_ca)
context.load_cert_chain(cert_pem, cert_key)
app.run(ssl_context = context, host = '0.0.0.0')
and changed the requests call to
# ----- SCRIPT 1 -----
r = requests.get(api_url, cert = (cert_pem, cert_key), verify = cert_ca)
When trying to pull data from the API I get the error
requests.exceptions.SSLError: HTTPSConnectionPool(host='some.host', port=443): Max retries exceeded with url: /some/path/var?ID=xxxxxx (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:847)'),))
Question 1: What am I doing wrong creating the CA_BUNDLE?
Question 2: Am I handling the creation of the web service correctly? My goal is to verify my server against the server holding the data to eventually be able to receive the data by push request.
EDIT: when connecting to my web service (in a browser) I receive the warning that the connection is not secure, because the certificate is not valid, despite the fact that I imported the .p12 certificate into my browser.

So I'm using the request and json library to call API, in my case I can set-up the request to ignore the certificate and this quickly solved my issue
requests.get(url, headers=headers, verify=False)
the argument verify=False ignore the certificate but when you run your code it will show a warning message as output saying that the certificate is wrong, so you can add this other piece of code to don't get request warning showed:
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
I know that doesn't answer your question but maybe you can try to see if without certificate you are able to get information without problem.

Use python to access a site with PKI security

I have a site that has PKI security enabled. Each client used either a card reader to load their certificate, or the certificate is installed in the IE certificate storage on their box.
So my question are:
How can I use either the card reader certificate or the certificate stored on the system to verify the system?
How do I pass the credentials onto the site to say, hey I'm me and I can access the service? They example can be using soft certificates. I can figure out the card reader part later.
I've been searching around, and I haven't come up with anything to help me in this situation. Django has a bunch of modules, but this isn't an option because I'm only concerned of the client side of things. I'm not creating a site to host the service. I need to just access these services.
I have this code working sort of. I just do not know how to handle the redirect I am getting:
import httplib
KEYFILE = r"C:\cert\my.key"
CERTFILE = r"c:\cert\my.pem"
HOSTNAME = 'machine.com'
conn = httplib.HTTPSConnection(
HOSTNAME,
key_file = KEYFILE,
cert_file = CERTFILE
)
conn.putrequest('GET', '/arcgis/sharing/rest?f=json')
conn.endheaders()
response = conn.getresponse()
print response.read()
The result of all of this is:
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved here.</p>
</body></html>
Any help provided would be great!
Software specs: python 2.7.8, Windows 2012 R2

I created a PKI handler to handle the requests so I can use it work urllib2 library.
import httplib, urllib2
class HTTPSClientAuthHandler(urllib2.HTTPSHandler):
def __init__(self, key, cert):
urllib2.HTTPSHandler.__init__(self)
self.key = key
self.cert = cert
def https_open(self, req):
#Rather than pass in a reference to a connection class, we pass in
# a reference to a function which, for all intents and purposes,
# will behave as a constructor
return self.do_open(self.getConnection, req)
def getConnection(self, host, timeout=300):
return httplib.HTTPSConnection(host,
key_file=self.key,
cert_file=self.cert,
timeout=timeout)
To use this, you will need to use a cookiejar with the handler.
from cookielib import CookieJar
cookiejar = CookieJay()
handlers = []
handlers.append(HTTPSClientAuthHandler(somekey, somecert))
handlers.append(urllib2.HTTPCookieProcessor(cookiejar))
opener = urllib2.build_opener(*handlers)
... do other urllib2 calls ....
Hope this helps everyone!

Try this code
#!/usr/bin/env python
import httplib
CERTFILE = '/home/robr/mycert'
HOSTNAME = 'localhost'
conn = httplib.HTTPSConnection(
HOSTNAME,
key_file = CERTFILE,
cert_file = CERTFILE
)
conn.putrequest('GET', '/ssltest/')
conn.endheaders()
response = conn.getresponse()
print response.read()

urllib2/pycurl in Django: Fetch XML, check HTTP status, check HTTPS connection

I need to make an API call (of sorts) in Django as a part of the custom authentication system we require. A username and password is sent to a specific URL over SSL (using GET for those parameters) and the response should be an HTTP 200 "OK" response with the body containing XML with the user's info.
On an unsuccessful auth, it will return an HTTP 401 "Unauthorized" response.
For security reasons, I need to check:
The request was sent over an HTTPS connection
The server certificate's public key matches an expected value (I use 'certificate pinning' to defend against broken CAs)
Is this possible in python/django using pycurl/urllib2 or any other method?

Using M2Crypto:
from M2Crypto import SSL
ctx = SSL.Context('sslv3')
ctx.set_verify(SSL.verify_peer | SSL.verify_fail_if_no_peer_cert, depth=9)
if ctx.load_verify_locations('ca.pem') != 1:
raise Exception('No CA certs')
c = SSL.Connection(ctx)
c.connect(('www.google.com', 443)) # automatically checks cert matches host
c.send('GET / \n')
c.close()
Using urllib2_ssl (it goes without saying but to be explicit: use it at your own risk):
import urllib2, urllib2_ssl
opener = urllib2.build_opener(urllib2_ssl.HTTPSHandler(ca_certs='ca.pem'))
xml = opener.open('https://example.com/').read()
Related: Making HTTPS Requests secure in Python.
Using pycurl:
c = pycurl.Curl()
c.setopt(pycurl.URL, "https://example.com?param1=val1&param2=val2")
c.setopt(pycurl.HTTPGET, 1)
c.setopt(pycurl.CAINFO, 'ca.pem')
c.setopt(pycurl.SSL_VERIFYPEER, 1)
c.setopt(pycurl.SSL_VERIFYHOST, 2)
c.setopt(pycurl.SSLVERSION, 3)
c.setopt(pycurl.NOBODY, 1)
c.setopt(pycurl.NOSIGNAL, 1)
c.perform()
c.close()
To implement 'certificate pinning' provide different 'ca.pem' for different domains.

httplib2 can do https requests with certificate validation:
import httplib2
http = httplib2.Http(ca_certs='/path/to/cert.pem')
try:
http.request('https://...')
except httplib2.SSLHandshakeError, e:
# do something
Just make sure that your httplib2 is up to date. The one which is shipped with my distribution (ubuntu 10.04) does not have ca_certs parameter.
Also in similar question to yours there is an example of certificate validation with pycurl.