I'm using a library (sentry.io observer) that should connect to a remote server via https and upload some data. I do not control the server, but I can see that no data is uploaded. I set the urllib logger level to debug and I see two log messages
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): <server_url>:443
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (2): <server_url>:443
but no exception is thrown. I used wireshark to sniff packets and I see two SSL handshakes are executed, but the FIN packet is sent right after server finishes the handshake. Packets exchange looks like:
< - client sends message
> - server sends message
< TCP handshake [syn, syn ack, ack]
< Client hello
> Server hello, certificate, server key exchange, server hello done
< Client key exchange, change cipher spec, finished
> New session ticket, change cipher spec, finished
< TCP connection termination [fin ack, fin ack, ack]
This packet exchange is done twice, as urllib tries to connect to the remote server twice. The server certificate is valid, but the connection is cancelled by client. I set the library and urllib loggers to debug, but no error messages or anything that could help me narrow the issue down appears.
The issue only appears when requests are done from docker (based on centos 7), but when launching the app on ubuntu host it works fine, connection is established and data is uploaded. What could be the cause of the issue?
Related
Client side:
data = b'\xff' * 1000000
ssock = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0)
#context is created by ssl.create_default_context(ssl.Purpose.SERVER_AUTH)
ssock = context.wrap_socket(ssock, server_hostname='xd1337sv')
ssock.connect((SERVERADDR, SERVERPORT))
ssock.sendall(data)
#time.sleep(3)
ssock.close()
If I just use regular non-SSL socket, everything works correctly with the server receiving exact amount of data. If I use TLS socket, the behavior then depends on the version.
If I run either the server or client on Python 3.6 and therefore the TLSv1.2 will be used, there's no problem.
Problem arises only when TLSv1.3 is used and depends on the size of data and how soon client ssocket.close() line is executed.
If I put a right amount of time.sleep before ssocket.close() depending on the size of data, then I get no error. Otherwise, the server will get ConnectionResetError [WinError 10054] An existing connection was forcibly closed by the remote host and receive only part of the data, or throw ConnectionAbortedError [WinError 10053] An established connection was aborted by the software in your host machine and receive no data.
I'm testing both the server and client on my local machine with local address 192.168.1.2.
The difference is caused by TLS 1.3 sending a session ticket after the TLS handshake while with previous TLS versions the session ticket is send inside the TLS handshake. Thus, with TLS 1.3 data from the server (the session ticket) will arrive after the ssock.connect(...) is done. Since your application does not read any data after the connect it closes the socket while unread data are still inside the socket buffer of the underlying TCP socket. This will cause RST send to the server and cause there the connection reset error.
This is a known problems with applications which never attempt to read from the server. If the application would expect a response from the server and use recv to get it this would implicitly also read the session ticket.
To fix this situation when you don't expect the server to return any application data do a proper SSL shutdown of the socket before closing it. Since this will read the servers SSL shutdown message it will also implicitly read the session ticket send before by the server.
try:
ssock = ssock.unwrap()
except:
True
ssock.close()
For more information see also this issue and this documentation.
I was getting a similar problem when the application was running through gunicorn with certificates. The jsondecodeerror problem randomly came to the client, i.e. the response was empty. The only thing that TLS 1.2 was used.
The solution was simple, I deployed the application on uwsgi and the problem went away
I have developed a desktop client using PyQt4, it connect to my web service by requests lib. You know, requests maybe one of the most useful http client, I think it should be no problem. My desktop client works all right until something strange happened.
I use the following code to send request to my server.
response = requests.get(url, headers = self.getHeaders(), timeout=600, proxies = {}, verify = False)
where header only includes auth token.
def getHeaders(self, additional = None):
headers = {
'Auth-Token' : HttpBasicClient.UserAuthToken,
}
if additional is not None:
headers.update(additional)
return headers
I cannot connect to my web service, all the http request pop the same error "'Cannot connect to proxy.', error(10061, '')". For example:
GET Url: http:// api.fangcloud.com/api/v1/user/timestamp
HTTPSConnectionPool(host='api.fangcloud.com', port=443): Max retries exceeded with url: /api/v1/user/timestamp (Caused by ProxyError('Cannot connect to proxy.', error(10061, '')))
this API does nothing but return the timestamp of my server. When I copy the url into Chrome in same machine with same environment, it returns correct response. However, my desktop client can only returns error. Is it anything wrong with requests lib?
I googled this problem of connection error 10061 ("No connection could be made because the target machine actively refused it"). This maybe caused by TCP connect rejection of web server.
The client sends a SYN packet to the server targeting the port (80 for HTTP). A server that is running a service on port 80 will respond with a SYN ACK, but if it is not, it will respond with a RST ACK. Your client reaches the server, but not the intended service. This is one way a server could “actively refuse” a connection attempt.
But why? My client works all right before and Chrome still works. I use no proxy on my machine. Is there anything I miss?
I notice there is a white space in URL, is that correct?
I tested in my ipython with requests.. that the response was:
{
"timestamp": 1472760770,
"success": true
}
For HTTP and HTTPS.
I have a Tornado HTTPServer initialized like so:
ssl_options = {
"certfile": "mycert.crt",
"keyfile": "mykey.key"
}
server = tornado.httpserver.HTTPServer(application, xheaders=True,
ssl_options=ssl_options)
It has a WebSocketHandler with an open() method.
When I attempt to open a secure websocket connection via Javascript from the Chrome 30 console, like
var sock = new WebSocket("wss://localhost:9001/mywebsocket");
the connection does not open successfully, there is no log output, and the WebSocketHandler's open() is not called.
How can I open and maintain a secure (SSL) WebSocket connection to a Javascript client in Tornado?
Attempting an insecure connection, like
var sock = new WebSocket("ws://localhost:9001/mywebsocket");
results in the following error output in the Tornado log:
2013-10-08 13:59:55,305 tornado.general 820 : SSL Error on 8 ('192.168.149.27', 62851): [Errno 1] _ssl.c:490: error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request
Also in this case, the connection is not opened successfully and open() isn't called.
Additional info: The number after the IP in the error message (62851 in the above example) increases with every request, secure or otherwise. I don't know what that number is but it does indicate that the request is at least getting to the server.
Also, removing ssl_options from the constructor and making insecure (ws://) requests to the server fixes the issue.
The certificate being used is self-signed. To communicate with an HTTP endpoint of the server via curl, I had to use the --insecure flag.
If you're starting the connection from the javascript console, the browser doesn't have a chance to show you the self-signed certificate warning and give you a chance to accept it. If you go to https://localhost:9001 first and accept the certificate there, does it work?
I am creating a proxy server in python, which is based on BaseHTTPServer.
What it does is create a connection to a squid proxy, identifies the browser request(GET, CONNECT, POST etc) and adds a proxy-authorization header to it, and then forwards this request to the squid proxy.
Problem is, as I understand, when I send a connect request, I should relay all the corresponding traffic to the squid proxy. But, as I can see in wireshark, the squid proxy doesn't reply to the 'Client Hello' part of the handshake, which I think is due to squid proxy not understanding binary data of SSL that I am just forwarding to it.
How do I process HTTPS requests in this case?
The code is more or less similar to TinyHTTPProxy : http://www.oki-osk.jp/esc/python/proxy/
RFC 2817 defines the CONNECT method. It is different from other HTTP methods in that the receiving proxy (your Python proxy) is directed to establish a raw TCP tunnel directly to the destination host (called the authority in the RFC).
A proxy can make no assumptions about the data that will be sent over that tunnel; it will not necessarily be HTTP – the client can use the tunnel to speak any protocol it likes. Indeed, SSL ≠ HTTP.
You have two options:
Open a TCP connection directly to the requested destination host.
Make a CONNECT request to your upstream proxy (Squid). This is within spec:
It may be the case that the proxy itself can only reach the
requested origin server through another proxy. In this case, the
first proxy SHOULD make a CONNECT request of that next proxy,
requesting a tunnel to the authority. A proxy MUST NOT respond
with any 2xx status code unless it has either a direct or tunnel
connection established to the authority.
Make sure that your request includes the required Host header.
CONNECT www.google.com:443 HTTP/1.1
Host: www.google.com:443
Proxy-Authorization: ...
I am using SSL tunneling with a proxy server to connect to a target server. I use http to connect to the proxy server and HTTPS to connect to the target server. The SSL tunneling works as it should and I can exchange HTTPS messages with the remote server, but there is a problem. The proxy server returns a header in its reply to urllib2's request to establish the SSL tunnel that I need to see, but I don't see a way to get access to it using urllib2 (Python 2.7.3).
I suppose I could theoretically implement the SSL tunneling handshake myself, but that would get me way deeper into the protocol than I want to be (or with which I feel comfortable).
Is there a way to get access to the reply using urllib2 when establishing the SSL tunnel?
UPDATE:
Here is the code that uses the proxy server to connect to the target server (the proxy server and the target server's URLs are not the actual ones):
proxy_handler = urllib2.ProxyHandler({'https': 'http://proxy.com'})
url_opener = urllib2.build_opener (proxy_handler)
request = urllib2.Request ('https://target_server.com/')
response = url_opener.open (request)
print response.headers.dict
I used WireShark to look at the message traffic. WireShark won't show me the bodies of the messages exchanged with the target server because they are encrypted, but I can see the body of the SSL Tunnel handshake. I can see the header that I'm interested coming back from the proxy server.
How are you calling the https page.
are you using
resp = urllib2.urlopen('https')
resp.info().headers