We Keep Coding

Python Requests and SSLContext

I'm trying to figure out how to specify an SSLContext with Request.
I have two functions which in theory should do the same, however the one with Requests doesn't work.
def func_OK(token):
ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH,cafile='myCA.crt.pem')
ctx.load_cert_chain(certfile='myprivate.pem')
url = 'https://my_url.com'
hdr = {"Content-Type": "application/json","Authorization":"Bearer "+token}
data = '{"filterList":[{}]}'
bdata = data.encode('utf-8')
req = urllib.request.Request(url, headers=hdr)
resp = urllib.request.urlopen(req, data=bdata, context=ctx)
content = resp.read()
data = json.loads(content.decode('utf-8'))
def func_NOK(token):
import requests
url = 'https://my_url.com'
hdr = {"Content-Type": "application/json","Authorization":"Bearer "+token}
data = '{"filterList":[{}]}'
bdata = data.encode('utf-8')
resp = requests.post(url,headers=hdr, data={"filterList":[{}]})
The only the difference between the two functions are the sslContext.
In the func_NOK, I try :
resp = requests.post(url,headers=hdr, data={"filterList":[{}]}, verify=False) - it doesn't work
resp = requests.post(url,headers=hdr, data={"filterList":[{}]}, cert=('myCA.crt.pem','myprivate.pem')) - it doesn't work
resp = requests.post(url,headers=hdr, data={"filterList":[{}]}, verify="concat_file.crt") with "concat_file.crt" file a concatenation of 'myCA.crt.pem' and 'myprivate.pem'
In any cases I have an SSL error.
For example, on my last example the error msg is :
requests.exceptions.ConnectionError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1131)
I'm just trying to use an SSLContext with Requests.

ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH,cafile='myCA.crt.pem')
ctx.load_cert_chain(certfile='myprivate.pem')
load_cert_chain loads the cert and private key for use as client certificate - which would be the cert argument with requests. cafile describes the CA it should use to verify the server certificate - which would be the verify argument for requests. Combined this would result in:
requests.post(..., cert='myprivate.pem', verify='myCA.crt.pem')

I find where my cacert.pem : /home/<soome_path>/pyEnv/myEnv/lib/python3.8/site-packages/certifi/cacert.pem
I concatenated the files :
myCA.crt.pem >> cacert.pem
myprivate.pem>> cacert.pem
then I specified the path using verify :
requests.post(...,verify='/home/<soome_path>/pyEnv/myEnv/lib/python3.8/site-packages/certifi/cacert.pem')
And I don't have the ssl error anymore.
However I retrieve an html msg instead of a json.
Maybe an issue on the parameters that I send to the endpoint.

I solved it using :
requests.post(url,headers=hdr,json={"filterList":[{}]}, cert='myprivate.pem')

How do I disable verification of the SSL Certificate while using PySocks and Socks5 Proxy?

This seems to get me close:
SOCKS5 proxy using urllib2 and PySocks
But it seems that if I try to add the context to disable the SSL verification it just ignores it.
I am not the greatest at python, but it looks like the inheritance of the class in PySocks takes the same things as HTTPShandler.
https://github.com/Anorov/PySocks/blob/master/sockshandler.py
If so I thought I could just pass the context=context in without and issue.
But it doesn't work.
Here is my method...
def make_http_call(url, socks_hostname=None, socks_port=None, socks_username=None, socks_password=None, params=None):
"""
Make a HTTP GET request to given url.
"""
import ssl
url = add_url_params(url, params)
opener = urllib2.build_opener()
context = ssl._create_unverified_context
if socks_hostname and socks_port and socks_username and socks_password:
# Use proxy instead if params are provided
print "Socks Proxy is being used..."
opener = urllib2.build_opener(
SocksiPyHandler(socks.PROXY_TYPE_SOCKS5, socks_hostname, socks_port, True, socks_username, socks_password, context=context))
else:
print "Socks Proxy not in use..."
request = urllib2.Request(url)
response = opener.open(request).read()
return response
Is this possible?

For the context variable i would do this(untested code):
context = ssl.create_default_context()
context.verify_mode = CERT_NONE
context.check_hostname = False
The last line is not necessary as we are not checking the certificate, but it is just in case. Now i do not recommend this as this will not ensure that you are communicating with who you think you are and i do not recommend using any python version below 3 because they are known to have big security issues, See Here.

Use python to access a site with PKI security

I have a site that has PKI security enabled. Each client used either a card reader to load their certificate, or the certificate is installed in the IE certificate storage on their box.
So my question are:
How can I use either the card reader certificate or the certificate stored on the system to verify the system?
How do I pass the credentials onto the site to say, hey I'm me and I can access the service? They example can be using soft certificates. I can figure out the card reader part later.
I've been searching around, and I haven't come up with anything to help me in this situation. Django has a bunch of modules, but this isn't an option because I'm only concerned of the client side of things. I'm not creating a site to host the service. I need to just access these services.
I have this code working sort of. I just do not know how to handle the redirect I am getting:
import httplib
KEYFILE = r"C:\cert\my.key"
CERTFILE = r"c:\cert\my.pem"
HOSTNAME = 'machine.com'
conn = httplib.HTTPSConnection(
HOSTNAME,
key_file = KEYFILE,
cert_file = CERTFILE
)
conn.putrequest('GET', '/arcgis/sharing/rest?f=json')
conn.endheaders()
response = conn.getresponse()
print response.read()
The result of all of this is:
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved here.</p>
</body></html>
Any help provided would be great!
Software specs: python 2.7.8, Windows 2012 R2

I created a PKI handler to handle the requests so I can use it work urllib2 library.
import httplib, urllib2
class HTTPSClientAuthHandler(urllib2.HTTPSHandler):
def __init__(self, key, cert):
urllib2.HTTPSHandler.__init__(self)
self.key = key
self.cert = cert
def https_open(self, req):
#Rather than pass in a reference to a connection class, we pass in
# a reference to a function which, for all intents and purposes,
# will behave as a constructor
return self.do_open(self.getConnection, req)
def getConnection(self, host, timeout=300):
return httplib.HTTPSConnection(host,
key_file=self.key,
cert_file=self.cert,
timeout=timeout)
To use this, you will need to use a cookiejar with the handler.
from cookielib import CookieJar
cookiejar = CookieJay()
handlers = []
handlers.append(HTTPSClientAuthHandler(somekey, somecert))
handlers.append(urllib2.HTTPCookieProcessor(cookiejar))
opener = urllib2.build_opener(*handlers)
... do other urllib2 calls ....
Hope this helps everyone!

Try this code
#!/usr/bin/env python
import httplib
CERTFILE = '/home/robr/mycert'
HOSTNAME = 'localhost'
conn = httplib.HTTPSConnection(
HOSTNAME,
key_file = CERTFILE,
cert_file = CERTFILE
)
conn.putrequest('GET', '/ssltest/')
conn.endheaders()
response = conn.getresponse()
print response.read()

python ignore certificate validation urllib2

I want to ignore the certification validation during my request to the server with an internal corporate link.
With python requests library I would do this:
r = requests.get(link, allow_redirects=False,verify=False)
How do I do the same with urllib2 library?

In the meantime urllib2 seems to verify server certificates by default.
The warning, that was shown in the past disappeared for 2.7.9 and I currently ran into this problem in a test environment with a self signed certificate (and Python 2.7.9).
My evil workaround (don't do this in production!):
import urllib2
import ssl
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
urllib2.urlopen("https://your-test-server.local", context=ctx)
According to docs calling SSLContext constructor directly should work, too. I haven't tried that.

The easiest way:
python 2
import urllib2, ssl
request = urllib2.Request('https://somedomain.co/')
response = urllib2.urlopen(request, context=ssl._create_unverified_context())
python 3
from urllib.request import urlopen
import ssl
response = urlopen('https://somedomain.co', context=ssl._create_unverified_context())

For those who uses an opener, you can achieve the same thing based on Enno Gröper's great answer:
import urllib2, ssl
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
opener = urllib2.build_opener(urllib2.HTTPSHandler(context=ctx), your_first_handler, your_second_handler[...])
opener.addheaders = [('Referer', 'http://example.org/blah.html')]
content = opener.open("https://localhost/").read()
And then use it as before.
According to build_opener and HTTPSHandler, a HTTPSHandler is added if ssl module exists, here we just specify our own instead of the default one.

According to #Enno Gröper 's post, I've tried the SSLContext constructor and it works well on my machine. code as below:
import ssl
ctx = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
urllib2.urlopen("https://your-test-server.local", context=ctx)
if you need opener, just added this context like:
opener = urllib2.build_opener(urllib2.HTTPSHandler(context=ctx))
NOTE: all above test environment is python 2.7.12. I use PROTOCOL_SSLv23 here since the doc says so, other protocol might also works but depends on your machine and remote server, please check the doc for detail.

A more explicit example, built on Damien's code (calls a test resource at http://httpbin.org/). For python3. Note that if the server redirects to another URL, uri in add_password has to contain the new root URL (it's possible to pass a list of URLs, also).
import ssl
import urllib.parse
import urllib.request
def get_resource(uri, user, passwd=False):
"""
Get the content of the SSL page.
"""
uri = 'https://httpbin.org/basic-auth/user/passwd'
user = 'user'
passwd = 'passwd'
context = ssl.create_default_context()
context.check_hostname = False
context.verify_mode = ssl.CERT_NONE
password_mgr = urllib.request.HTTPPasswordMgrWithDefaultRealm()
password_mgr.add_password(None, uri, user, passwd)
auth_handler = urllib.request.HTTPBasicAuthHandler(password_mgr)
opener = urllib.request.build_opener(auth_handler, urllib.request.HTTPSHandler(context=context))
urllib.request.install_opener(opener)
return urllib.request.urlopen(uri).read()

urllib2 does not verify server certificate by default. Check this documentation.
Edit: As pointed out in below comment, this is not true anymore for newer versions (seems like >= 2.7.9) of Python. Refer the below ANSWER

How to specify an authenticated proxy for a python http connection?

What's the best way to specify a proxy with username and password for an http connection in python?

This works for me:
import urllib2
proxy = urllib2.ProxyHandler({'http': 'http://
username:password#proxyurl:proxyport'})
auth = urllib2.HTTPBasicAuthHandler()
opener = urllib2.build_opener(proxy, auth, urllib2.HTTPHandler)
urllib2.install_opener(opener)
conn = urllib2.urlopen('http://python.org')
return_str = conn.read()

Use this:
import requests
proxies = {"http":"http://username:password#proxy_ip:proxy_port"}
r = requests.get("http://www.example.com/", proxies=proxies)
print(r.content)
I think it's much simpler than using urllib. I don't understand why people love using urllib so much.

Setting an environment var named http_proxy like this: http://username:password#proxy_url:port

The best way of going through a proxy that requires authentication is using urllib2 to build a custom url opener, then using that to make all the requests you want to go through the proxy. Note in particular, you probably don't want to embed the proxy password in the url or the python source code (unless it's just a quick hack).
import urllib2
def get_proxy_opener(proxyurl, proxyuser, proxypass, proxyscheme="http"):
password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_mgr.add_password(None, proxyurl, proxyuser, proxypass)
proxy_handler = urllib2.ProxyHandler({proxyscheme: proxyurl})
proxy_auth_handler = urllib2.ProxyBasicAuthHandler(password_mgr)
return urllib2.build_opener(proxy_handler, proxy_auth_handler)
if __name__ == "__main__":
import sys
if len(sys.argv) > 4:
url_opener = get_proxy_opener(*sys.argv[1:4])
for url in sys.argv[4:]:
print url_opener.open(url).headers
else:
print "Usage:", sys.argv[0], "proxy user pass fetchurls..."
In a more complex program, you can seperate these components out as appropriate (for instance, only using one password manager for the lifetime of the application). The python documentation has more examples on how to do complex things with urllib2 that you might also find useful.

Or if you want to install it, so that it is always used with urllib2.urlopen (so you don't need to keep a reference to the opener around):
import urllib2
url = 'www.proxyurl.com'
username = 'user'
password = 'pass'
password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
# None, with the "WithDefaultRealm" password manager means
# that the user/pass will be used for any realm (where
# there isn't a more specific match).
password_mgr.add_password(None, url, username, password)
auth_handler = urllib2.HTTPBasicAuthHandler(password_mgr)
opener = urllib2.build_opener(auth_handler)
urllib2.install_opener(opener)
print urllib2.urlopen("http://www.example.com/folder/page.html").read()

Here is the method use urllib
import urllib.request
# set up authentication info
authinfo = urllib.request.HTTPBasicAuthHandler()
proxy_support = urllib.request.ProxyHandler({"http" : "http://ahad-haam:3128"})
# build a new opener that adds authentication and caching FTP handlers
opener = urllib.request.build_opener(proxy_support, authinfo,
urllib.request.CacheFTPHandler)
# install it
urllib.request.install_opener(opener)
f = urllib.request.urlopen('http://www.python.org/')
"""