We are relatively new to in Python therefore may be the question is too simple.
We are using Python version 2.7.15.
We are trying to use Python over TLS without success.
This is our code:
import ssl,socket
import urllib2
context = ssl.SSLContext(ssl.PROTOCOL_TLS)
context.verify_mode = ssl.CERT_REQUIRED
context.check_hostname = False
context.load_verify_locations("/py-test/python/bin/certificate.pem")
url = "https://10.0.0.12"
request = urllib2.Request(url)
websocket = urllib2.urlopen(request,None,None,None,None,None,context)
pages=websocket.readlines()
print pages
As you see, we have configured context.check_hostname = False
Unfortunately, it fails with the following exception
Traceback (most recent call last):
File "./test.py", line 11, in <module>
websocket = urllib2.urlopen(request,None,None,None,None,None,context)
File "/py-test/python/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/py-test/python/lib/python2.7/urllib2.py", line 429, in open
response = self._open(req, data)
File "/py-test/python/lib/python2.7/urllib2.py", line 447, in _open
'_open', req)
File "/py-test/python/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/py-test/python/lib/python2.7/urllib2.py", line 1241, in https_open
context=self._context)
File "/py-test/python/lib/python2.7/urllib2.py", line 1198, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:726)>
It is definitely the hostname verification.
If we use the correct certificate and correct hostname the request successful.
If we use the wrong certificate it fails with the following exception.
File "./test.py", line 8, in <module>
context.load_verify_locations("/py-test/python/bin/certificate_bad.pem")
ssl.SSLError: [X509] PEM lib (_ssl.c:3027)
Therefore, we need help to understand how to configure Python to ignore the hostname verification.
One more question (can be asked in the separate thread).
Do we have in Python a trustore file that include all known CA? Like cacerts.jks in Java.
Where can we find the trustore?
Added
We “want to verify that the certificate was signed by a valid CA”, but we “don't care whether it identifies the site you're actually connecting to”.
We need help to configure Python to ignore the hostname verification?
What is mistake in our code?
We have tried to create the code according to the documentation https://docs.python.org/2/library/ssl.html
Added 2
We have invested a lot of time but unfortunately we do not have the progress.
Is anyone has the working example in Python 2.7?
I mean is the code works if you access with other URL then appears in a certificate.
May be Python 2.7 cannot be configured to ignore the hostname verification?
What can be our problem?
We use it on CentOS 6.
May be it is related to OpenSSL? We use the latest version openssl-1.0.1e-57.el6.x86_64.
May be we should upgrade to Python 3.x?
As you discovered, you can accomplish this by customizing the SSLContext object used for verifying the connection.
However, to hook this into urllib2.urlopen you'll need to build a custom opener and install that.
Here's an example:
import httplib
import socket
import ssl
import urllib2
import certifi
class InterceptedHttpsConnection(httplib.HTTPSConnection):
def connect(self):
# Open an unencrypted TCP socket first
sock = socket.create_connection((self.host, self.port), self.timeout)
# Build up a custom SSLContext. (Might be better to do this once rather
# than on every request.)
context = ssl.SSLContext(ssl.PROTOCOL_TLS)
# We require the SSL context to verify the certificate.
context.verify_mode = ssl.CERT_REQUIRED
# But we instruct the SSL context to *not* validate the hostname.
context.check_hostname = False
# Load CA certificates from the certifi bundle.
context.load_verify_locations(cafile=certifi.where())
# Use our SSLContext object to wrap the bare socket into an SSL socket.
self.sock = context.wrap_socket(sock, server_hostname=self.host)
class InterceptedHttpsHandler(urllib2.HTTPSHandler):
def https_open(self, req):
return self.do_open(InterceptedHttpsConnection, req)
def main():
opener = urllib2.build_opener(InterceptedHttpsHandler)
urllib2.install_opener(opener)
contents = urllib2.urlopen('https://example.com/').read()
print contents
import ssl
ssl._create_default_https_context = ssl._create_unverified_context
Forces python to skip the validation of certificates by default. Hope it helps.
Related
Hello I would ask if i could do connection to extract some data from Tool Server.
so i making this code but i need to modify it to open this tool with username and password and extract data from My tool server.
import socket
import ssl
HOST, PORT = '10.74.159.82', 31039
def handle(conn):
conn.write(b'GET / HTTP/1.1\n')
print(conn.recv() . decode())
def main():
sock = socket.socket(socket.AF_INET)
context = ssl.create_default_context(ssl.Purpose.SERVER_AUTH)
conn = context.wrap_socket(sock, server_hostname=HOST)
try:
conn.connect((HOST, PORT))
handle(conn)
finally:
conn.close()
if __name__ == '__main__':
main()
RESULT
!! C:\Users\Admin\.PyCharmCE2018.1\config\venv\Scripts\python.exe!! C:/Users/Admin/.PyCharmCE2018.1/config/codestyles/Under_Building.py
Traceback (most recent call last):
File "C:/Users/Admin/.PyCharmCE2018.1/config/codestyles/Under_Building.py", line 22, in <module>
main()
File "C:/Users/Admin/.PyCharmCE2018.1/config/codestyles/Under_Building.py", line 16, in main
conn.connect((HOST, PORT))
File "C:\Users\Admin\AppData\Local\Programs\Python\Python37-32\lib\ssl.py", line 1141, in connect
self._real_connect(addr, False)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python37-32\lib\ssl.py", line 1132, in _real_connect
self.do_handshake()
File "C:\Users\Admin\AppData\Local\Programs\Python\Python37-32\lib\ssl.py", line 1108, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1045)
Process finished with exit code 1
I'm Beginner so i need to learn and supporting
It's hard to see exactly, but it seems the server you're connecting to (on https://10.74.159.82:31039) is using a self-signed certificate (or its CA is, perhaps).
It's perhaps better to create or buy a proper (non-self-signed) certificate, but assuming this is a test server this is fine.
So with what you have, your best bet is to allow self-signed certs in your SSL Context (and also, not to check the hostname - that will probably fail too as you're using an IP address):
context = ssl.create_default_context()
context.check_hostname=False
context.verify_mode=ssl.CERT_NONE
# etc...
OS: Windows 7; Python 2.7.3 using the Python GUI Shell
I'm trying to read a website through Python, and several authors use the urllib and urllib2 libraries. To store the site in a variable, I've seen a similar approach proposed:
import urllib
import urllib2
g = "http://www.google.com/"
read = urllib2.urlopen(g)
The last line generates an error after a 120+ seconds:
> Traceback (most recent call last): File "<pyshell#27>", line 1, in
> <module>
> r = urllib2.urlopen(o) File "C:\Python27\lib\urllib2.py", line 126, in urlopen
> return _opener.open(url, data, timeout) File "C:\Python27\lib\urllib2.py", line 400, in open
> response = self._open(req, data) File "C:\Python27\lib\urllib2.py", line 418, in _open
> '_open', req) File "C:\Python27\lib\urllib2.py", line 378, in _call_chain
> result = func(*args) File "C:\Python27\lib\urllib2.py", line 1207, in http_open
> return self.do_open(httplib.HTTPConnection, req) File "C:\Python27\lib\urllib2.py", line 1177, in do_open
> raise URLError(err) URLError: <urlopen error [Errno 10060] A connection attempt failed because the connected party did not properly
> respond after a period of time, or established connection failed
> because connected host has failed to respond>
I tried bypassing the g variable and trying to urlopen("http://www.google.com/") with no success either (it generates the same error after the same length of time).
The error code 10060 means it cannot connect to the remote peer. It might be because of the network problem or mostly your setting issues, such as proxy setting.
You could try to connect the same host with other tools(such as ncat) and/or with another PC within your same local network to find out where the problem is occuring.
For proxy issue, there are some material here:
Using an HTTP PROXY - Python
Why can't I get Python's urlopen() method to work on Windows?
Hope it helps!
Answer (Basic is advance!):
Error: 10060
Adding a timeout parameter to request solved the issue for me.
Example 1
import urllib
import urllib2
g = "http://www.google.com/"
read = urllib2.urlopen(g, timeout=20)
Example 2
A similar error also occurred while I was making a GET request. Again, passing a timeout parameter solved the 10060 Error.
response = requests.get(param_url, timeout=20)
This is because of the proxy settings.
I also had the same problem, under which I could not use any of the modules which were fetching data from the internet.
There are simple steps to follow:
1. open the control panel
2. open internet options
3. under connection tab open LAN settings
4. go to advance settings and unmark everything, delete every proxy in there. Or u can just unmark the checkbox in proxy server this will also do the same
5. save all the settings by clicking ok.
you are done.
try to run the programme again, it must work
it worked for me at least
just change your internet connection it is going to work..
So I have an Elasticsearch cluster that I have been accessing from .Net web services via a ACL protected http endpoint. I have spun up a new reverse proxy that uses SSL and requires a client certificate to forward the request on to the (now protected) Elasticsearch cluster.
I have this all working fine in .Net, but I have one service that I inherited that running Python. I have not worked with Python much, and am struggling to figure out how to implement this.
I am using ES 2.3.2 and Python 3.2.
For .Net, I have the certificate installed on the hosting machine where the code pulls it directly from the store. I have used OpenSSL and exported this into a .pem (as well as .crt and .key as well). The problems I am encountering are A) how to get the transport to be https instead of http, and B) how to include my cert. I first tried:
client = Elasticsearch(hosts=[self.host], timeout=80, use_ssl=True, verify_certs=True, client_cert=cert_file_path, client_key=key_file_path)
This does not seem to work. So I figured I needed to build up my own connection/transport, so I tried this:
connection = Urllib3HttpConnection(host=self.host['host'], port=self.host['port'], use_ssl=True,ca_certs=cert_file_path, client_cert=cert_file_path, client_key=key_file_path, verify_certs=True )
transport = Transport(hosts=[self.host], connection_class=connection)
client = Elasticsearch(hosts=[self.host], timeout=80, use_ssl=True, verify_certs=True, Transport=transport, client_cert=cert_file_path, client_key=key_file_path)
I need to manually in the debugger change the scheme to 'https', but when I run now, I get:
TypeError("'Urllib3HttpConnection' object is not callable",)
In .Net what I am doing is overriding the HttpConnection in the ConnectionSetings for the ElasticClient. In it, I override CreateHttpWebRequest and add my certificate to the the outbound request:
// add the cert to the collection
webRequest.ClientCertificates.Add(this.clientCertificate);
It seems this should be a common thing to do, but I can't figure it out (though that could be due to my lack of python experience).
Any help out there?
Thanks!
~john
UPDATE
So here is the definition of my host:
{"use_ssl": True, "host": "[my server].cloudapp.net", "port": "443", "scheme": "https", "transport_schema": "https"}
From the documentation:
kwargs – any additional arguments will be passed on to the Transport class and, subsequently, to the Connection instances.
I would assume that the scheme and/or transport_schema would be forwarded on through.
I create the client as:
client = Elasticsearch(hosts=[self.host], timeout=80, use_ssl=True, scheme="https", verify_certs=True, client_cert=cert_file_path, client_key=key_file_path)
When I create the client, I get:
ERROR Exception encountered. Detailed info: ImproperlyConfigured('Scheme specified in connection (https) is not the same as the connection class (Urllib3HttpConnection) specifies (http).',)
This is happening in transport.py in _create_connection():
if 'scheme' in host and host['scheme'] != self.connection_class.transport_schema:
raise ImproperlyConfigured(
'Scheme specified in connection (%s) is not the same as the connection class (%s) specifies (%s).' % (
host['scheme'], self.connection_class.__name__, self.connection_class.transport_schema
))
When I break here, I can see that:
self.kwargs['transport_schema'] = 'https'
but:
self.connection_class.transport_schema = 'http'
I would have thought that the kwarg would have been forwarded on to the connection_class. I manually set the transport_schema to 'https' and this allows me to get past this error.
So that is the first part. After doing this, I get a certificate validation error. This is probably resolveable, I just haven't been through to this part yet. The problems here are:
A) I am prompted for a pass phrase. How can I supply this? This is a service that will be running unattended.
B) Anyway, when I supply it, I get:
GET https://[my server].cloudapp.net:443/ocv_int/_search?request_cache=true [status:N/A request:6.085s]
Traceback (most recent call last):
File "E:\Source\Repos\OfficeCustomerVoice\UserExperience\OCVClusteringWebAPI\env3\lib\site-packages\urllib3\connectionpool.py", line 578, in urlopen
chunked=chunked)
File "E:\Source\Repos\OfficeCustomerVoice\UserExperience\OCVClusteringWebAPI\env3\lib\site-packages\urllib3\connectionpool.py", line 351, in _make_request
self._validate_conn(conn)
File "E:\Source\Repos\OfficeCustomerVoice\UserExperience\OCVClusteringWebAPI\env3\lib\site-packages\urllib3\connectionpool.py", line 814, in _validate_conn
conn.connect()
File "E:\Source\Repos\OfficeCustomerVoice\UserExperience\OCVClusteringWebAPI\env3\lib\site-packages\urllib3\connection.py", line 289, in connect
ssl_version=resolved_ssl_version)
File "E:\Source\Repos\OfficeCustomerVoice\UserExperience\OCVClusteringWebAPI\env3\lib\site-packages\urllib3\util\ssl_.py", line 308, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "C:\Users\jthoni\AppData\Local\Continuum\Anaconda3-2-3\lib\ssl.py", line 365, in wrap_socket
_context=self)
File "C:\Users\jthoni\AppData\Local\Continuum\Anaconda3-2-3\lib\ssl.py", line 583, in init
self.do_handshake()
File "C:\Users\jthoni\AppData\Local\Continuum\Anaconda3-2-3\lib\ssl.py", line 810, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "E:\Source\Repos\OfficeCustomerVoice\UserExperience\OCVClusteringWebAPI\env3\lib\site-packages\elasticsearch\connection\http_urllib3.py", line 94, in perform_request
response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
File "E:\Source\Repos\OfficeCustomerVoice\UserExperience\OCVClusteringWebAPI\env3\lib\site-packages\urllib3\connectionpool.py", line 604, in urlopen
raise SSLError(e)
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)
P.S. to get my cert files I did the following with my pfx file:
openssl pkcs12 -in E:\[myCert].pfx -out E:\[myCert].pem -nodes
openssl pkcs12 -in E:\[myCert].pfx -nocerts -out E:\[myCert].key
openssl pkcs12 -in E:\[myCert].pfx -clcerts -nokeys -out E:\[myCert].crt
OS: Windows 7; Python 2.7.3 using the Python GUI Shell
I'm trying to read a website through Python, and several authors use the urllib and urllib2 libraries. To store the site in a variable, I've seen a similar approach proposed:
import urllib
import urllib2
g = "http://www.google.com/"
read = urllib2.urlopen(g)
The last line generates an error after a 120+ seconds:
> Traceback (most recent call last): File "<pyshell#27>", line 1, in
> <module>
> r = urllib2.urlopen(o) File "C:\Python27\lib\urllib2.py", line 126, in urlopen
> return _opener.open(url, data, timeout) File "C:\Python27\lib\urllib2.py", line 400, in open
> response = self._open(req, data) File "C:\Python27\lib\urllib2.py", line 418, in _open
> '_open', req) File "C:\Python27\lib\urllib2.py", line 378, in _call_chain
> result = func(*args) File "C:\Python27\lib\urllib2.py", line 1207, in http_open
> return self.do_open(httplib.HTTPConnection, req) File "C:\Python27\lib\urllib2.py", line 1177, in do_open
> raise URLError(err) URLError: <urlopen error [Errno 10060] A connection attempt failed because the connected party did not properly
> respond after a period of time, or established connection failed
> because connected host has failed to respond>
I tried bypassing the g variable and trying to urlopen("http://www.google.com/") with no success either (it generates the same error after the same length of time).
The error code 10060 means it cannot connect to the remote peer. It might be because of the network problem or mostly your setting issues, such as proxy setting.
You could try to connect the same host with other tools(such as ncat) and/or with another PC within your same local network to find out where the problem is occuring.
For proxy issue, there are some material here:
Using an HTTP PROXY - Python
Why can't I get Python's urlopen() method to work on Windows?
Hope it helps!
Answer (Basic is advance!):
Error: 10060
Adding a timeout parameter to request solved the issue for me.
Example 1
import urllib
import urllib2
g = "http://www.google.com/"
read = urllib2.urlopen(g, timeout=20)
Example 2
A similar error also occurred while I was making a GET request. Again, passing a timeout parameter solved the 10060 Error.
response = requests.get(param_url, timeout=20)
This is because of the proxy settings.
I also had the same problem, under which I could not use any of the modules which were fetching data from the internet.
There are simple steps to follow:
1. open the control panel
2. open internet options
3. under connection tab open LAN settings
4. go to advance settings and unmark everything, delete every proxy in there. Or u can just unmark the checkbox in proxy server this will also do the same
5. save all the settings by clicking ok.
you are done.
try to run the programme again, it must work
it worked for me at least
just change your internet connection it is going to work..
This work fine:
import urllib2
opener = urllib2.build_opener(
urllib2.HTTPHandler(),
urllib2.HTTPSHandler(),
urllib2.ProxyHandler({'http': 'http://user:pass#proxy:3128'}))
urllib2.install_opener(opener)
print urllib2.urlopen('http://www.google.com').read()
But, if http change to https:
...
print urllib2.urlopen('https://www.google.com').read()
There are errors:
Traceback (most recent call last):
File "D:\Temp\6\tmp.py", line 13, in <module>
print urllib2.urlopen('https://www.google.com').read()
File "C:\Python26\lib\urllib2.py", line 124, in urlopen
return _opener.open(url, data, timeout)
File "C:\Python26\lib\urllib2.py", line 389, in open
response = self._open(req, data)
File "C:\Python26\lib\urllib2.py", line 407, in _open
'_open', req)
File "C:\Python26\lib\urllib2.py", line 367, in _call_chain
result = func(*args)
File "C:\Python26\lib\urllib2.py", line 1154, in https_open
return self.do_open(httplib.HTTPSConnection, req)
File "C:\Python26\lib\urllib2.py", line 1121, in do_open
raise URLError(err)
URLError: <urlopen error [Errno 10060]
Why and how solve this problem?
Change this line:
urllib2.ProxyHandler({'http': 'http://user:pass#proxy:3128'}))
to this:
urllib2.ProxyHandler({'https': 'http://user:pass#proxy:3128'}))
It works fine for me.
On Windows, errno 10060 is a winsock error meaning the connection timed out. Are you able to reach https://www.google.com from the same machine using a web browser with a proxy set to http://user:pass#proxy:3128 ? Are you sure your proxy server can handle both https and http on the same port?
The documentation for urllib2 says the following:
Note: Currently urllib2 does not support fetching of https locations
through a proxy. However, this can be enabled by extending urllib2 as
shown in this recipe.
I must admit above recipe didn't work right away for Jython 2.5.3, but I'm still trying.
UPDATE: I applied this patch to Jython 2.5.3, and it worked for me. I can fetch HTTPS resources over a proxy server now.
UPDATE2: Here is the code to query HTTPS resources with Basic authentication over HTTP Proxy (DON'T FORGET TO INSTALL PATCH FIRST (see previous update)):
from suds.client import Client
from suds.transport.https import HttpAuthenticated
credentials = dict(username='...', password='...', proxy={'https': 'host:port', 'http': 'host:port'})
t = HttpAuthenticated(**credentials)
url = 'https://example.com/service?wsdl'
client = Client(url, transport=t)
print client.service.getFoo()