I'm trying to send an HTTPS request through an HTTPS tunnel. That is, my proxy expects HTTPS for the CONNECT. It also expects a client certificate.
I'm using Requests' proxy features.
import requests
url = "https://some.external.com/endpoint"
with requests.Session() as session:
response = session.get(
url,
proxies={"https": "https://proxy.host:4443"},
# client certificates expected by proxy
cert=(cert_path, key_path),
verify="/home/savior/proxy-ca-bundle.pem",
)
with response:
...
This works, but with some limitations:
I can only set client certificates for the TLS connection with the proxy, not for the external endpoint.
The proxy-ca-bundle.pem only verifies the server certificates in the TLS connection with the proxy. The server certificates from the external endpoint are seemingly ignored.
Is there any way to use requests to address these two issues? I'd like to set a different set of CAs for the external endpoint.
I also tried using http.client and HTTPSConnection.set_tunnel but, as far as I can tell, its tunnel is done through HTTP and I need HTTPS.
Looking at the source code, it doesn't seem like requests currently supports this "TLS in TLS", ie. providing two sets of clients/CA bundles for a proxied requests.
We can use PycURL which simply wraps libcurl
from io import BytesIO
import pycurl
url = "https://some.external.com/endpoint"
buffer = BytesIO()
curl = pycurl.Curl()
curl.setopt(curl.URL, url)
curl.setopt(curl.WRITEDATA, buffer)
# proxy settings
curl.setopt(curl.HTTPPROXYTUNNEL, 1)
curl.setopt(curl.PROXY, "https://proxy.host")
curl.setopt(curl.PROXYPORT, 4443)
curl.setopt(curl.PROXY_SSLCERT, cert_path)
curl.setopt(curl.PROXY_SSLKEY, key_path)
curl.setopt(curl.PROXY_CAINFO, "/home/savior/proxy-ca-bundle.pem")
# endpoint verification
curl.setopt(curl.CAINFO, "/home/savior/external-ca-bundle.pem")
try:
curl.perform()
except pycurl.error:
pass # log or re-raise
else:
status_code = curl.getinfo(curl.RESPONSE_CODE)
PycURL will use the PROXY_ settings to establish a TLS connection to the proxy, send it an HTTP CONNECT request. Then it'll establish a new TLS session through the proxy connection to the external endpoint and use the CAINFO bundle to verify those server certificates.
Related
I'm trying to connect to a websocket server that protected with CloudFlare through upgrade: websocket header. Expected result is 101 Switching Protocol. Using a raw Socket, I was able to connect into the server but with several issues such as SSLv3 Handshake Failure or the server doesn't give any response; sometimes occur.
import ssl
import socket
socketch = ssl._create_unverified_context().wrap_socket(socket.socket(), server_hostname='unpkg.com')
socketch.connect(('unpkg.com', 443))
socketch.sendall(b'''GET / HTTP/1.1\r
Host: identity.o2.co.uk.zainvps.tk\r
User-Agent: cpprestsdk/2.9.0\r
Upgrade: websocket\r
Connection: Upgrade\r
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==\r
Sec-WebSocket-Version: 13\r\n\r
''')
print(socketch.recv(10000))
print('')
Using a raw socket is unstable, so I think it's better to use requests module.
import requests
heading = {'Host':'identity.o2.co.uk.zainvps.tk','Connection':'upgrade','Upgrade':'websocket','Sec-Websocket-Version':'13','Sec-Websocket-Key':'dGhlIHNhbXBsZSBub25jZQ=='}
r = requests.get('https://unpkg.com', headers=heading)
print(r.status_code)
Using requests; the server responded with 403 status codes which means it's rejected by the CloudFlare protection but when using Socket, it gives the correct 101 status code. I'm assuming that it is because of wrapped socket gives an expected SSL Hostname through server_hostname.
Is this idea can also be implemented inside requests.Session()?
UPDATE 1:
Someone mentioning about the use of CloudScraper module to bypass the CloudFlare protection. Using CloudScraper still returns in 403 status code with Custom Headers.
import cloudscraper
scraper = cloudscraper.create_scraper()
url = 'https://unpkg.com'
sc = scraper.get(url, headers={"Host": "usaws1.sshstores.vip", "Connection": "upgrade", "Upgrade": "websocket","Sec-WebSocket-Key": "dGhlIHNhbXBsZSBub25jZQ==", "Sec-WebSocket-Version": "13"})
print(sc.status_code)
when using Python 2.7s urllib2 I do not seem to be able to retrieve a resource from a HTTPS server while using a SSL secured proxy server, i.e. to following:
CLIENT ---- (HTTPS) ---> PROXY ---- (https) --- > SERVER
Of cause to get through the proxy server one uses CONNECT. Any ideas?
Alternative question: when using CONNECT one needs to setup a completly independent 2. SSL session inside the tunnel, right? How could one do that in python as simply calling ssl.wrap_socket does not do the trick...?
I have a Tornado HTTPServer initialized like so:
ssl_options = {
"certfile": "mycert.crt",
"keyfile": "mykey.key"
}
server = tornado.httpserver.HTTPServer(application, xheaders=True,
ssl_options=ssl_options)
It has a WebSocketHandler with an open() method.
When I attempt to open a secure websocket connection via Javascript from the Chrome 30 console, like
var sock = new WebSocket("wss://localhost:9001/mywebsocket");
the connection does not open successfully, there is no log output, and the WebSocketHandler's open() is not called.
How can I open and maintain a secure (SSL) WebSocket connection to a Javascript client in Tornado?
Attempting an insecure connection, like
var sock = new WebSocket("ws://localhost:9001/mywebsocket");
results in the following error output in the Tornado log:
2013-10-08 13:59:55,305 tornado.general 820 : SSL Error on 8 ('192.168.149.27', 62851): [Errno 1] _ssl.c:490: error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request
Also in this case, the connection is not opened successfully and open() isn't called.
Additional info: The number after the IP in the error message (62851 in the above example) increases with every request, secure or otherwise. I don't know what that number is but it does indicate that the request is at least getting to the server.
Also, removing ssl_options from the constructor and making insecure (ws://) requests to the server fixes the issue.
The certificate being used is self-signed. To communicate with an HTTP endpoint of the server via curl, I had to use the --insecure flag.
If you're starting the connection from the javascript console, the browser doesn't have a chance to show you the self-signed certificate warning and give you a chance to accept it. If you go to https://localhost:9001 first and accept the certificate there, does it work?
I am creating a proxy server in python, which is based on BaseHTTPServer.
What it does is create a connection to a squid proxy, identifies the browser request(GET, CONNECT, POST etc) and adds a proxy-authorization header to it, and then forwards this request to the squid proxy.
Problem is, as I understand, when I send a connect request, I should relay all the corresponding traffic to the squid proxy. But, as I can see in wireshark, the squid proxy doesn't reply to the 'Client Hello' part of the handshake, which I think is due to squid proxy not understanding binary data of SSL that I am just forwarding to it.
How do I process HTTPS requests in this case?
The code is more or less similar to TinyHTTPProxy : http://www.oki-osk.jp/esc/python/proxy/
RFC 2817 defines the CONNECT method. It is different from other HTTP methods in that the receiving proxy (your Python proxy) is directed to establish a raw TCP tunnel directly to the destination host (called the authority in the RFC).
A proxy can make no assumptions about the data that will be sent over that tunnel; it will not necessarily be HTTP – the client can use the tunnel to speak any protocol it likes. Indeed, SSL ≠ HTTP.
You have two options:
Open a TCP connection directly to the requested destination host.
Make a CONNECT request to your upstream proxy (Squid). This is within spec:
It may be the case that the proxy itself can only reach the
requested origin server through another proxy. In this case, the
first proxy SHOULD make a CONNECT request of that next proxy,
requesting a tunnel to the authority. A proxy MUST NOT respond
with any 2xx status code unless it has either a direct or tunnel
connection established to the authority.
Make sure that your request includes the required Host header.
CONNECT www.google.com:443 HTTP/1.1
Host: www.google.com:443
Proxy-Authorization: ...
I am using SSL tunneling with a proxy server to connect to a target server. I use http to connect to the proxy server and HTTPS to connect to the target server. The SSL tunneling works as it should and I can exchange HTTPS messages with the remote server, but there is a problem. The proxy server returns a header in its reply to urllib2's request to establish the SSL tunnel that I need to see, but I don't see a way to get access to it using urllib2 (Python 2.7.3).
I suppose I could theoretically implement the SSL tunneling handshake myself, but that would get me way deeper into the protocol than I want to be (or with which I feel comfortable).
Is there a way to get access to the reply using urllib2 when establishing the SSL tunnel?
UPDATE:
Here is the code that uses the proxy server to connect to the target server (the proxy server and the target server's URLs are not the actual ones):
proxy_handler = urllib2.ProxyHandler({'https': 'http://proxy.com'})
url_opener = urllib2.build_opener (proxy_handler)
request = urllib2.Request ('https://target_server.com/')
response = url_opener.open (request)
print response.headers.dict
I used WireShark to look at the message traffic. WireShark won't show me the bodies of the messages exchanged with the target server because they are encrypted, but I can see the body of the SSL Tunnel handshake. I can see the header that I'm interested coming back from the proxy server.
How are you calling the https page.
are you using
resp = urllib2.urlopen('https')
resp.info().headers