We Keep Coding

When should I close the socket, if the request contains Connection: keep-alive?

When I am using this Python script to send a request to my server:
import requests as r
url = "http://localhost:8070/"
response = r.get(url=url)
It sends the following request:
GET / HTTP/1.1
Host: localhost:8070
User-Agent: python-requests/2.27.1
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
If I understand it correctly, Connection: keep-alive means that I should not close the client-socket, because the client could use it again.
But if I do not close the client-socket, the Python script just gets stuck until I close the socket. Is there another way to indicate that the request is complete so pythons requests understands it?
If I try to send the request to any other server, the script almost immediately finishes. My guess would be to timeout the client after a few milliseconds e.g. by using select like this:
struct timeval timeout;
timeout.tv_sec = 0;
timeout.tv_usec = 1000;
select_ret = select(this->_maxfds + 1, &this->_readfds, &this->_writefds, NULL, &timeout);
And now I would just close the client-socket after select returns 0:
if (select_ret == 0) {
close(client_socket);
}
Is that a valid approach, or am I missing something?
I am sending the response like this:
char *response = "HTTP/1.1 200 Ok\r\n\r\n";
send(this->_client_socket, response, strlen(response), 0)
BUT this does not terminate the python script. The python script still hangs after I execute this line of code. It only finishes when I close the socket on my side.
So how would I determine if I should close it or not? As I already said my approach was to use a timeout in case no data is getting written in to the socket from the client side.

Your server's response is incomplete.
Your understanding of Connection: keep-alive in the request is correct. However, there is no Content-Length or Transfer-Encoding: chunked header present in your response, so the only way the client has to know when the response is finished is to wait for the socket connection to be closed on the server side. Read the rules outlined in RFC 2616 Section 4.4 and RFC 7230 Section 3.3.3 of the HTTP 1.1 protocol spec.
Try something more like this instead:
const char *response = "HTTP/1.1 200 Ok\r\nContent-Length: 0\r\n\r\n";
send(this->_client_socket, response, strlen(response), 0)

Requests Library Force Use of HTTP/1.1 On HTTPS Proxy CONNECT

I am having a problem with a misbehaving HTTP Proxy server. I have no control over the proxy server, unfortunately -- it's an 'enterprise' product from IBM. The proxy server is part of a service virtualization solution being leveraged for software testing.
The fundamental issue (I think*) is that the proxy server sends back HTTP/1.0 responses. I can get it to work fine from SOAP UI ( A Java application) and curl from the command line, but Python refuses to connect. From what I can tell, Python is behaving correctly, and the other two are not, as the server expects HTTP/1.1 responses (it wants Host headers, at the very least, to route the service request to a given stub).
Is there a way to get Requests, or the underlying urllib3, or the even farther down http lib to always use http1.1, even if the other end appears to be using 1.0?
Here is a sample program (unfortunately, it requires you to have an IBM Ration Integration Tester installation with RTCP to really replicate) to reproduce the problem:
import http.client as http_client
http_client.HTTPConnection.debuglevel = 1
import logging
import requests
logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True
requests.post("https://host:8443/axl",
headers={"soapAction": '"CUCM:DB ver=9.1 updateSipTrunk"'},
data='<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:tns="http://www.cisco.com/AXL/API/9.1"><soapenv:Header/><soapenv:Body><tns:updateSipTrunk><name>PLACEHOLDER</name><newName>PLACEHOLDER</newName><destinations><destination><addressIpv4>10.10.1.5</addressIpv4><sortOrder>1</sortOrder></destination></destinations></tns:updateSipTrunk></soapenv:Body></soapenv:Envelope>',
verify=False)
(Proxy is configured via HTTPS_PROXY environment variable)
Debug output before the error, note the HTTP/1.0:
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): host.com
send: b'CONNECT host.com:8443 HTTP/1.0\r\n'
send: b'\r\n'
header: Host: host.com:8443
header: Proxy-agent: Green Hat HTTPS Proxy/1.0
The exact error text that occurs in RHEL 6 is:
requests.exceptions.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:646)
Even though the Host header is shown here, it does NOT show up on the wire. I confirmed this with a tcpdump:
14:03:14.315049 IP sourcehost.53214 > desthost.com: Flags [P.], seq 0:32, ack 1, win 115, options [nop,nop,TS val 2743933964 ecr 4116114841], length 32
0x0000: 0000 0c07 ac00 0050 56b5 4044 0800 4500 .......PV.#D..E.
0x0010: 0054 3404 4000 4006 2ca0 0af8 3f15 0afb .T4.#.#.,...?...
0x0020: 84f8 cfde 0c7f a4f8 280a 4ebd b425 8018 ........(.N..%..
0x0030: 0073 da46 0000 0101 080a a38d 1c0c f556 .s.F...........V
0x0040: XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX ..CONNECT.host
0x0050: XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX xx:8443.HTTP/1.0
0x0060: 0d0a
When I curl it with verbose, this is what the output looks like:
* About to connect() to proxy proxy-host.com port 3199 (#0)
* Trying 10.**.**.** ... connected
* Connected to proxy-host.com (10.**.**.**) port 3199 (#0)
* Establish HTTP proxy tunnel to host.com:8443
> CONNECT host.com:8443 HTTP/1.1
> Host: host.com:8443
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Proxy-Connection: Keep-Alive
> soapAction: "CUCM:DB ver=9.1 updateSipTrunk"
>
< HTTP/1.0 200 OK
< Host: host.com:8443
< Proxy-agent: Green Hat HTTPS Proxy/1.0
<
* Proxy replied OK to CONNECT request
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /path/to/store/ca-bundle.crt
CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
Truncated after this point. You can see the HTTP/1.0 response from the proxy after connecting. The curl's tcpdump also clearly shows the host header, as well as HTTP 1.1.
*I can't be entirely sure this is the fundamental issue, as I can't test it. I do see HTTP/1.0 responses, and can tell that my non-working Python code sends CONNECT HTTP/1.0 messages, while the working Java sends HTTP/1.1 messages, as does Curl. It's possible the problem is unrelated (although I find that unlikely) or that Python is misbehaving, and not Java/curl. I simply don't know enough to know for sure.
So, is there a way to force urllib3/requests to use HTTP v1.1 at all times?

httplib (which requests relies upon for HTTP(S) heavy lifting) always uses HTTP/1.0 with CONNECT:
Lib/httplib.py:788:
def _tunnel(self):
self.send("CONNECT %s:%d HTTP/1.0\r\n" % (self._tunnel_host,
self._tunnel_port))
for header, value in self._tunnel_headers.iteritems():
self.send("%s: %s\r\n" % (header, value))
self.send("\r\n")
<...>
So you can't "force" it to use "HTTP/1.1" here other than by editing the subroutine.
This MAY be the problem if the proxy doesn't support HTTP/1.0 - in particular, 1.0 does not require a Host: header, and indeed, as you can see by comparing your log output with the code above, httplib does not send it. While, in verity, a proxy may expect it regardless. But if this is the case, you should've gotten an error from the proxy or something in response to CONNECT -- unless the proxy is so borken that it substitutes some default (or garbage) for Host:, returns 200 anyway and tries to connect God-knows-where, at which point you're getting timeouts.
You can make httplib add the Host: header to CONNECT by adding it to _tunnel_headers (indirectly):
s=requests.Session()
proxy_url=os.environ['HTTPS_PROXY']
s.proxies["https"]=proxy_url
# have to specify proxy here because env variable is only detected by httplib code
#while we need to trigger requests' proxy logic that acts earlier
# "https" means any https host. Since a Session persists cookies,
#it's meaningless to make requests to multiple hosts through it anyway.
pm=s.get_adapter("https://").proxy_manager_for(proxy_url)
pm.proxy_headers['Host']="host.com"
del pm,proxy_url
<...>
s.get('https://host.com')

If you do not depend on the requests library you may find the following snippet useful:
import http.client
conn = http.client.HTTPSConnection("proxy.domain.lu", 8080)
conn.set_tunnel("www.domain.org", 443, headers={'User-Agent': 'curl/7.56.0'})
conn.request("GET", "/api")
response = conn.getresponse()
print( response.read() )

Python: Tor not connecting with a certain website

I am trying to access a specific website with Python and Tor, but somehow it just loads forever and no response comes.
import requests
import socks
import socket
url = 'http://www.ryanair.com'
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, '127.0.0.1', 9150)
socket.socket = socks.socksocket
response = requests.get(url)
#no response comes
Some weeks ago it was working. In addition, the same code works still fine for other sites.
My guess would be that it is not a problem on my side, but on the website's side, but I have no clue what can be wrong.
Any ideas of what can be and how it could be solved?
Edit:
If I run the command:
curl --proxy socks5h://127.0.0.1:9150 -vvv http://www.ryanair.com
I get:
* Rebuilt URL to: http://www.ryanair.com/
* Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 9150 (#0)
> GET / HTTP/1.1
> Host: www.ryanair.com
> User-Agent: curl/7.49.1
> Accept: */*
>
< HTTP/1.1 302 Found : Moved Temporarily
< Location: https://www.ryanair.com/
< Connection: close
< Cache-Control: no-cache
< Pragma: no-cache
<
* Closing connection 0
Trying with https:
$ curl --proxy socks5h://127.0.0.1:9150 -vvv https://www.ryanair.com
* Rebuilt URL to: https://www.ryanair.com/
* Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 9150 (#0)
* TLS 1.2 connection using TLS_RSA_WITH_3DES_EDE_CBC_SHA
* Server certificate: www.ryanair.com
* Server certificate: GeoTrust SSL CA - G3
* Server certificate: GeoTrust Global CA
> GET / HTTP/1.1
> Host: www.ryanair.com
> User-Agent: curl/7.49.1
> Accept: */*
>
#...and it remains here
Edit 2:
I just tried to load this website directly at the Tor browser and it will also keep loading without end.

It will be difficult to run diagnostics without access to your proxy server, but in general, if you're debugging a request, curl is your most invaluable tool.
This should help:
curl --proxy socks5h://127.0.0.1:9150 -vvv http://www.ryanair.com
The -vvv will show you information about the request and the response. That will show you whether your request is well structured or whether the server response is at fault.

How to connect a client to a server through an HTTP proxy in Python?

I'm really new to coding using sockets.
I like the socket library, I get to understand a big part of what's happening in my program, so i you don't mind i would like to stick with it.
So as the title says, I have a socket based client and server and I would like to exchange content through an HTTP proxy(I'm using a Squid proxy). This little piece of code is supposed to bypass the proxy in my campus to simulate a chat over the campus network. This is totally legal since I asked the IT guys that work there.
Here's the deal, I am able to send a POST request through the proxy to my server which receives it and sends it back to client 1, but when I try to send more requests to the proxy none of them gets to the server so I think to my self the connection died but here's the thing, when I send messages from client 2 which is connected directly to the server, the server AND client 1 receive them.
import socket
from _thread import *
def sender(server,h):
b=input("<<--Send--")
b=h
server.send(b.encode())
PROXY_IP="127.0.0.1"
PROXY_PORT=3128
server=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
server.settimeout(0.05)
server.connect((PROXY_IP,PROXY_PORT))
header="""POST http://127.0.0.1:3001 HTTP/1.1\r\n
Host: 127.0.0.1:3001\r\n
Proxy-Connection: keep-alive\r\n
Content-Length: 5 \r\n\r\n
hello\r\n"""
server.send(header.encode())
while 1:
try:
start_new_thread(sender,(server,header))
a=server.recv(1024)
print("-->>{}".format(a.decode()))
except KeyboardInterrupt:
break
except:
pass
server.close()
I already tried the CONNECT method which works perfectly, but it's not allowed in my campus network proxy.
What am I doing wrong ?
Is there something I should know about how to re-send content through a proxy ?
Thank you for your time and please bear with me..
Here's what I get on the client that sends a request to the proxy:
~#Sent : POST http://127.0.0.1:3001 HTTP/1.1
Host: 127.0.0.1:3001
Proxy-Connection: keep-alive
Content-Length: 5
hello
#Received : HTTP/1.1 200 OK
Server: squid/3.5.19
Mime-Version: 1.0
Date: Mon, 10 Oct 2016 00:46:39 GMT
X-Transformed-From: HTTP/0.9
X-Cache: MISS from kali
X-Cache-Lookup: MISS from kali:3128
Transfer-Encoding: chunked
Via: 1.1 kali (squid/3.5.19)
Connection: keep-alive
#Received : B2
POST / HTTP/1.1
Content-Length: 5
Host: 127.0.0.1:3001
Via: 1.1 kali (squid/3.5.19)
X-Forwarded-For: 127.0.0.1
Cache-Control: max-age=259200
Connection: keep-alive
hello
#Sent : POST http://127.0.0.1:3001 HTTP/1.1
Host: 127.0.0.1:3001
Proxy-Connection: keep-alive
Content-Length: 5
hello
Nothing is received after this..

POST http://127.0.0.1:3001 HTTP/1.1\r\n
Host: 127.0.0.1:3001\r\n
Proxy-Connection: keep-alive\r\n
Content-Length: 5 \r\n\r\n
hello\r\n
The body of your HTTP response consists of 7 bytes not 5 as you've stated in your Content-length. The \r\n after the 5 byte still belong to the response body. Giving the wrong size might mixup request handling, i.e. the proxy is expecting a new request but is actually getting \r\n, i.e. the 2 bytes after your 5 bytes Content-length.
Apart from that both path and Host header must include the name of the target from the perspective of the proxy. Using 127.0.0.1. like in your example would mean that you try to access a server at the same host of the proxy, i.e. localhost from the view of the proxy. This is probably not what you've intended.
...
X-Transformed-From: HTTP/0.9
This header in the response of the proxy indicates that your server does not properly speak HTTP/1.x. Instead of sending HTTP header and body it just sends the payload back without any HTTP header, like done in the HTTP 0.9 protocol which was obsoleted 20 years ago. With HTTP 0.9 the response will always end only at the end of the TCP connection. This means that you cannot have multiple requests within the same TCP connection.
I'm really new to coding using sockets.
The problem is not caused by the wrong use of sockets but due the wrong implementation of the application protocol, i.e. the data send over the socket. If you really need to implement HTTP please study the standards, i.e. RFC 7230 and following. If you don't want to do this use existing and tested HTTP libraries instead of writing your own.

Calling a Flask REST service method in different OS with curl

I wrote the following POST method for my REST API, which was built using Flask. The method receives one parameter, which is a radio station url.
#app.route('/todo/api/v1.0/predvajaj', methods=['POST'])
def create_task():
print "Expression value: " + str(not request.json or not 'title' in request.json)
if not request.json or not 'title' in request.json:
abort(400)
link=request.json['title']
print "Link value: " + link
cmd = "pkill sox"
os.system(cmd)
time.sleep(2)
#link = "http://www.radiostationurl.m3u"
cmd = "sox -t mp3 " + link + " -t wav -r 22050 -c 1 - | sudo ../pifm - 90.5 &"
os.system(cmd)
return jsonify({'status': "ok"}), 201
The API runs on a Raspberry Pi with the ip address: 192.168.0.200. I tried testing the method locally (on the Pi), using the curl tool. This worked fine:
curl -i -H "Content-Type: application/json" -X POST -d '{"title":"http://www.radiostationurl.m3u"}' http://192.168.0.200:5000/todo/api/v1.0/predvajaj
Then I tried testing testing it with a computer (running Windows) in the same LAN with the same command and tool, but I get the following error:
HTTP/1.0 400 BAD REQUEST
Content-Type: text/html
Content-Length: 192
Server: Werkzeug/0.10.4 Python/2.7.3
Date: Wed, 05 Aug 2015 11:06:05 GMT
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>400 Bad Request</title>
<h1>Bad Request</h1>
<p>The browser (or proxy) sent a request that this server could not understand.</p>
Webserver output (two requests - one from the pi and from the other pc):
Expression value: False
192.168.0.200 - - [05/Aug/2015 11:05:56] "POST /todo/api/v1.0/predvajaj HTTP/1.1" 201 -
sox WARN wav: Length in output .wav header will be wrong since can't seek to fix it
192.168.0.103 - - [05/Aug/2015 11:06:05] "POST /todo/api/v1.0/predvajaj HTTP/1.1" 400
So the problem is in the evaluation of the if expression. Can anybody tell me why is it failing to evaluate?
EDIT: Tried running curl with the -v switch as #meuh suggested. The content-length is different.
Pi:
* About to connect() to 192.168.0.200 port 5000 (#0)
* Trying 192.168.0.200...
* connected
* Connected to 192.168.0.200 (192.168.0.200) port 5000 (#0)
> POST /todo/api/v1.0/predvajaj HTTP/1.1
> User-Agent: curl/7.26.0
> Host: 192.168.0.200:5000
> Accept: */*
> Content-Type: application/json
> Content-Length: 51
>
* upload completely sent off: 51 out of 51 bytes
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.0, assume close after body
< HTTP/1.0 201 CREATED
< Content-Type: application/json
< Content-Length: 27
< Server: Werkzeug/0.10.4 Python/2.7.3
< Date: Wed, 05 Aug 2015 13:49:01 GMT
<
{
"status": "predvajam"
* Closing connection #0
}
Windows:
* About to connect() to 192.168.0.200 port 5000 (#0)
* Trying 192.168.0.200...
* Connected to 192.168.0.200 (192.168.0.200) port 5000 (#0)
> POST /todo/api/v1.0/predvajaj HTTP/1.1
> Host: 192.168.0.200:5000
> User-Agent: curl/7.43.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 49
>
* upload completely sent off: 49 out of 49 bytes
* HTTP 1.0, assume close after body
< HTTP/1.0 400 BAD REQUEST
< Content-Type: text/html
< Content-Length: 192
< Server: Werkzeug/0.10.4 Python/2.7.3
< Date: Wed, 05 Aug 2015 13:50:51 GMT
<
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>400 Bad Request</title>
<h1>Bad Request</h1>
<p>The browser (or proxy) sent a request that this server could not understand.</p>
* Closing connection 0

I overlooked a note about this issue, on the site with the tutorial, which says:
Note: if you are on Windows and use the Cygwin version of curl from bash then the above command will work just fine. However, if you are using the native version of curl from the regular command prompt there is a little dance that needs to be done to send double quotes inside the body of a request. Essentially on Windows you have to use double quotes to enclose the body of the request, and then inside it you escape a double quote by writing three of them in sequence.
The correct command, in my case, is:
curl -i -H "Content-Type: application/json" -X POST -d "{"""title""":"""http://www.radiostationurl.m3u"""}" http://192.168.0.200:5000/todo/api/v1.0/predvajaj