I'm trying to write a twisted https client, that uses a client certificate for authentication. This is the example from the twisted docs for making a simple https request:
from __future__ import print_function
from twisted.internet import reactor
from twisted.web.client import Agent
from twisted.web.http_headers import Headers
agent = Agent(reactor)
d = agent.request(
b'GET',
b'https://127.0.0.1:8880/test',
Headers({'User-Agent': ['Twisted Web Client Example']}),
None)
def cbResponse(ignored):
print('Response received')
d.addCallback(cbResponse)
def cbShutdown(ignored):
reactor.stop()
d.addBoth(cbShutdown)
reactor.run()
I've found this explanation on how to use a client with ssl certificates. I have no idea though, how I can tell the agent to use a certificate for authentication. Any ideas?
Construct Agent with an IPolicyForHTTPS which can create a contextFactory to provide a client certificate to the TLS connection.
certData = getModule(__name__).filePath.sibling('public.pem').getContent()
authData = getModule(__name__).filePath.sibling('server.pem').getContent()
clientCertificate = ssl.PrivateCertificate.loadPEM(authData)
authority = ssl.Certificate.loadPEM(certData)
options = ssl.optionsForClientTLS(u'example.com', authority,
clientCertificate)
class SinglePolicy(object):
def creatorForNetloc(self, hostname, port):
return options
agent = Agent(reactor, SinglePolicy())
Related
I'm trying to connect to a websocket server that protected with CloudFlare through upgrade: websocket header. Expected result is 101 Switching Protocol. Using a raw Socket, I was able to connect into the server but with several issues such as SSLv3 Handshake Failure or the server doesn't give any response; sometimes occur.
import ssl
import socket
socketch = ssl._create_unverified_context().wrap_socket(socket.socket(), server_hostname='unpkg.com')
socketch.connect(('unpkg.com', 443))
socketch.sendall(b'''GET / HTTP/1.1\r
Host: identity.o2.co.uk.zainvps.tk\r
User-Agent: cpprestsdk/2.9.0\r
Upgrade: websocket\r
Connection: Upgrade\r
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==\r
Sec-WebSocket-Version: 13\r\n\r
''')
print(socketch.recv(10000))
print('')
Using a raw socket is unstable, so I think it's better to use requests module.
import requests
heading = {'Host':'identity.o2.co.uk.zainvps.tk','Connection':'upgrade','Upgrade':'websocket','Sec-Websocket-Version':'13','Sec-Websocket-Key':'dGhlIHNhbXBsZSBub25jZQ=='}
r = requests.get('https://unpkg.com', headers=heading)
print(r.status_code)
Using requests; the server responded with 403 status codes which means it's rejected by the CloudFlare protection but when using Socket, it gives the correct 101 status code. I'm assuming that it is because of wrapped socket gives an expected SSL Hostname through server_hostname.
Is this idea can also be implemented inside requests.Session()?
UPDATE 1:
Someone mentioning about the use of CloudScraper module to bypass the CloudFlare protection. Using CloudScraper still returns in 403 status code with Custom Headers.
import cloudscraper
scraper = cloudscraper.create_scraper()
url = 'https://unpkg.com'
sc = scraper.get(url, headers={"Host": "usaws1.sshstores.vip", "Connection": "upgrade", "Upgrade": "websocket","Sec-WebSocket-Key": "dGhlIHNhbXBsZSBub25jZQ==", "Sec-WebSocket-Version": "13"})
print(sc.status_code)
As the title states, I'd like to figure out how to use SNI (server name indication) with the tornado webserver.
I'd like to be able to present a particular certificate based on the hostname of the given request.
After looking more into this I found the solution using a sni_callback from ssl.SSLContext.
That will give you a method that contains the hostname before a TLS handshake is established.
Within the servername_callback method you can then chose what certificate to load based on the hostname.
Working solution
import tornado.ioloop
import tornado.web
import tornado.httpserver
import ssl
class MainHandler(tornado.web.RequestHandler):
def get(self):
self.write('Hello, world')
application = tornado.web.Application([
(r'/', MainHandler)
])
def servername_callback(sock, hostname, cb_context):
# hostname contains the hostname that the client is requesting
print("hostname", hostname)
# now that we have the hostname we can dynamically pick the correct certificate
ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
# this part is up to you to store via a config file or even in a database
ssl_context.load_cert_chain(certfile="/path/to/cert", keyfile="/path/to/key")
sock.context = ssl_context
if __name__ == '__main__':
ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
ssl_context.sni_callback = servername_callback
http_server = tornado.httpserver.HTTPServer(application, ssl_options=ssl_context)
http_server.listen(443)
tornado.ioloop.IOLoop.instance().start()
I'm trying to send an HTTPS request through an HTTPS tunnel. That is, my proxy expects HTTPS for the CONNECT. It also expects a client certificate.
I'm using Requests' proxy features.
import requests
url = "https://some.external.com/endpoint"
with requests.Session() as session:
response = session.get(
url,
proxies={"https": "https://proxy.host:4443"},
# client certificates expected by proxy
cert=(cert_path, key_path),
verify="/home/savior/proxy-ca-bundle.pem",
)
with response:
...
This works, but with some limitations:
I can only set client certificates for the TLS connection with the proxy, not for the external endpoint.
The proxy-ca-bundle.pem only verifies the server certificates in the TLS connection with the proxy. The server certificates from the external endpoint are seemingly ignored.
Is there any way to use requests to address these two issues? I'd like to set a different set of CAs for the external endpoint.
I also tried using http.client and HTTPSConnection.set_tunnel but, as far as I can tell, its tunnel is done through HTTP and I need HTTPS.
Looking at the source code, it doesn't seem like requests currently supports this "TLS in TLS", ie. providing two sets of clients/CA bundles for a proxied requests.
We can use PycURL which simply wraps libcurl
from io import BytesIO
import pycurl
url = "https://some.external.com/endpoint"
buffer = BytesIO()
curl = pycurl.Curl()
curl.setopt(curl.URL, url)
curl.setopt(curl.WRITEDATA, buffer)
# proxy settings
curl.setopt(curl.HTTPPROXYTUNNEL, 1)
curl.setopt(curl.PROXY, "https://proxy.host")
curl.setopt(curl.PROXYPORT, 4443)
curl.setopt(curl.PROXY_SSLCERT, cert_path)
curl.setopt(curl.PROXY_SSLKEY, key_path)
curl.setopt(curl.PROXY_CAINFO, "/home/savior/proxy-ca-bundle.pem")
# endpoint verification
curl.setopt(curl.CAINFO, "/home/savior/external-ca-bundle.pem")
try:
curl.perform()
except pycurl.error:
pass # log or re-raise
else:
status_code = curl.getinfo(curl.RESPONSE_CODE)
PycURL will use the PROXY_ settings to establish a TLS connection to the proxy, send it an HTTP CONNECT request. Then it'll establish a new TLS session through the proxy connection to the external endpoint and use the CAINFO bundle to verify those server certificates.
I have the following code for a portforward proxy. How can I add
ssl support so that the proxy can connect to a server listening with ssl.
here is the code:
from twisted.internet import reactor
from twisted.protocols import portforward
class ProxyServer(portforward.ProxyServer):
def dataReceived(self, data)
portforward.ProxyServer.dataReceived(self, data)
class ProxyFactory(portforward.ProxyFactory):
protocol = ProxyServer
reactor.listenTCP(8080,ProxyFactory("127.0.0.1",443) )
reactor.run()
Check this link out. This excerpt is probably relevant to what you are looking for .
https://twistedmatrix.com/documents/13.2.0/core/howto/ssl.html
with open('server.pem') as keyAndCert:
cert = ssl.PrivateCertificate.loadPEM(keyAndCert.read())
log.startLogging(sys.stdout)
factory = Factory()
factory.protocol = echoserv.Echo
reactor.listenSSL(8000, factory, cert.options())
reactor.run()
"Notice how all of the protocol code from the TCP version of the echo client and server examples is the same (imported or repeated) in these SSL versions - only the reactor method used to initiate a network action is different."
How do i write a simple http web server using twisted framework?
I want a web server that can receive http request and return a response to the client.
Am going through the twisted documentation and am kinda confused (maybe am just lazy), but it does not look very direct on how to do this, especially how does the twisted server receive the request parameters?
This is my effort...
from twisted.web import proxy, http
from twisted.internet import reactor
from twisted.python import log
from twisted.protocols import basic
import sys
log.startLogging(sys.stdout)
class InformixProtocol(basic.LineReceiver):
def lineReceived(self, user):
self.transport.write("Hello this is twisted web server!")
self.transport.loseConnection()
class ProxyFactory(http.HTTPFactory):
#protocol = proxy.Proxy
protocol = InformixProtocol
reactor.listenTCP(8080, ProxyFactory())
reactor.run()
Thanks
Gath