We Keep Coding

How exactly should I properly interrupt HTTP upload requests?

I developed a server app which, among others, handles uploading of large chunks of data. When the upload request starts, before even receiving the chunk, the server app performs a few checks in order for the client to abort the operation if something goes wrong, instead of finding that there is an issue only after he sends gigabytes of data.
When playing with the server app using curl, I discovered a strange behavior.
curl starts the request, being ready to stream the data.
The server responds immediately with a HTTP 403 to signal a problem and provides a JSON response with the details of the problem.
curl fails with exit code 18 and the following output:
curl: (18) transfer closed with 30 bytes remaining to read
When enabling verbose output, here's what I see:
$ curl -X PUT --limit-rate 2M http://127.0.0.1/blob -F files[]=#/tmp/tmp75hw30vc -v
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
> PUT /blob HTTP/1.1
> Host: 127.0.0.1
> User-Agent: curl/7.58.0
> Accept: */*
> Content-Length: 2439352842
> Content-Type: multipart/form-data; boundary=------------------------32c442f4cf8abe0c
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< HTTP/1.1 403 FORBIDDEN
< Server: nginx/1.10.3
< Date: Sat, 29 Sep 2018 22:03:16 GMT
< Content-Type: application/json
< Content-Length: 30
< Connection: keep-alive
* HTTP error before end of send, stop sending
<
* transfer closed with 30 bytes remaining to read
* stopped the pause stream!
* Closing connection 0
curl: (18) transfer closed with 30 bytes remaining to read
On server side, the code (using Flask) is the following:
def receive_blob():
if _can_upload():
return flask.jsonify({"error": "already-uploading"}), 403
...
I'm not sure I understand if the problem is related to my way to use Flask, or to curl options I'm using.
What should I do to avoid this situation, i.e. to make curl display the JSON error message returned by the server?
The question is not a duplicate of How to handle "100 continue" HTTP message? since mine explicitly asks how to make curl display the JSON error message. The linked question invites to add --fail which would lead instead to the output “curl: (22) The requested URL returned error: 403 FORBIDDEN”

Python: Tor not connecting with a certain website

I am trying to access a specific website with Python and Tor, but somehow it just loads forever and no response comes.
import requests
import socks
import socket
url = 'http://www.ryanair.com'
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, '127.0.0.1', 9150)
socket.socket = socks.socksocket
response = requests.get(url)
#no response comes
Some weeks ago it was working. In addition, the same code works still fine for other sites.
My guess would be that it is not a problem on my side, but on the website's side, but I have no clue what can be wrong.
Any ideas of what can be and how it could be solved?
Edit:
If I run the command:
curl --proxy socks5h://127.0.0.1:9150 -vvv http://www.ryanair.com
I get:
* Rebuilt URL to: http://www.ryanair.com/
* Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 9150 (#0)
> GET / HTTP/1.1
> Host: www.ryanair.com
> User-Agent: curl/7.49.1
> Accept: */*
>
< HTTP/1.1 302 Found : Moved Temporarily
< Location: https://www.ryanair.com/
< Connection: close
< Cache-Control: no-cache
< Pragma: no-cache
<
* Closing connection 0
Trying with https:
$ curl --proxy socks5h://127.0.0.1:9150 -vvv https://www.ryanair.com
* Rebuilt URL to: https://www.ryanair.com/
* Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 9150 (#0)
* TLS 1.2 connection using TLS_RSA_WITH_3DES_EDE_CBC_SHA
* Server certificate: www.ryanair.com
* Server certificate: GeoTrust SSL CA - G3
* Server certificate: GeoTrust Global CA
> GET / HTTP/1.1
> Host: www.ryanair.com
> User-Agent: curl/7.49.1
> Accept: */*
>
#...and it remains here
Edit 2:
I just tried to load this website directly at the Tor browser and it will also keep loading without end.

It will be difficult to run diagnostics without access to your proxy server, but in general, if you're debugging a request, curl is your most invaluable tool.
This should help:
curl --proxy socks5h://127.0.0.1:9150 -vvv http://www.ryanair.com
The -vvv will show you information about the request and the response. That will show you whether your request is well structured or whether the server response is at fault.

Calling a Flask REST service method in different OS with curl

I wrote the following POST method for my REST API, which was built using Flask. The method receives one parameter, which is a radio station url.
#app.route('/todo/api/v1.0/predvajaj', methods=['POST'])
def create_task():
print "Expression value: " + str(not request.json or not 'title' in request.json)
if not request.json or not 'title' in request.json:
abort(400)
link=request.json['title']
print "Link value: " + link
cmd = "pkill sox"
os.system(cmd)
time.sleep(2)
#link = "http://www.radiostationurl.m3u"
cmd = "sox -t mp3 " + link + " -t wav -r 22050 -c 1 - | sudo ../pifm - 90.5 &"
os.system(cmd)
return jsonify({'status': "ok"}), 201
The API runs on a Raspberry Pi with the ip address: 192.168.0.200. I tried testing the method locally (on the Pi), using the curl tool. This worked fine:
curl -i -H "Content-Type: application/json" -X POST -d '{"title":"http://www.radiostationurl.m3u"}' http://192.168.0.200:5000/todo/api/v1.0/predvajaj
Then I tried testing testing it with a computer (running Windows) in the same LAN with the same command and tool, but I get the following error:
HTTP/1.0 400 BAD REQUEST
Content-Type: text/html
Content-Length: 192
Server: Werkzeug/0.10.4 Python/2.7.3
Date: Wed, 05 Aug 2015 11:06:05 GMT
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>400 Bad Request</title>
<h1>Bad Request</h1>
<p>The browser (or proxy) sent a request that this server could not understand.</p>
Webserver output (two requests - one from the pi and from the other pc):
Expression value: False
192.168.0.200 - - [05/Aug/2015 11:05:56] "POST /todo/api/v1.0/predvajaj HTTP/1.1" 201 -
sox WARN wav: Length in output .wav header will be wrong since can't seek to fix it
192.168.0.103 - - [05/Aug/2015 11:06:05] "POST /todo/api/v1.0/predvajaj HTTP/1.1" 400
So the problem is in the evaluation of the if expression. Can anybody tell me why is it failing to evaluate?
EDIT: Tried running curl with the -v switch as #meuh suggested. The content-length is different.
Pi:
* About to connect() to 192.168.0.200 port 5000 (#0)
* Trying 192.168.0.200...
* connected
* Connected to 192.168.0.200 (192.168.0.200) port 5000 (#0)
> POST /todo/api/v1.0/predvajaj HTTP/1.1
> User-Agent: curl/7.26.0
> Host: 192.168.0.200:5000
> Accept: */*
> Content-Type: application/json
> Content-Length: 51
>
* upload completely sent off: 51 out of 51 bytes
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.0, assume close after body
< HTTP/1.0 201 CREATED
< Content-Type: application/json
< Content-Length: 27
< Server: Werkzeug/0.10.4 Python/2.7.3
< Date: Wed, 05 Aug 2015 13:49:01 GMT
<
{
"status": "predvajam"
* Closing connection #0
}
Windows:
* About to connect() to 192.168.0.200 port 5000 (#0)
* Trying 192.168.0.200...
* Connected to 192.168.0.200 (192.168.0.200) port 5000 (#0)
> POST /todo/api/v1.0/predvajaj HTTP/1.1
> Host: 192.168.0.200:5000
> User-Agent: curl/7.43.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 49
>
* upload completely sent off: 49 out of 49 bytes
* HTTP 1.0, assume close after body
< HTTP/1.0 400 BAD REQUEST
< Content-Type: text/html
< Content-Length: 192
< Server: Werkzeug/0.10.4 Python/2.7.3
< Date: Wed, 05 Aug 2015 13:50:51 GMT
<
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>400 Bad Request</title>
<h1>Bad Request</h1>
<p>The browser (or proxy) sent a request that this server could not understand.</p>
* Closing connection 0

I overlooked a note about this issue, on the site with the tutorial, which says:
Note: if you are on Windows and use the Cygwin version of curl from bash then the above command will work just fine. However, if you are using the native version of curl from the regular command prompt there is a little dance that needs to be done to send double quotes inside the body of a request. Essentially on Windows you have to use double quotes to enclose the body of the request, and then inside it you escape a double quote by writing three of them in sequence.
The correct command, in my case, is:
curl -i -H "Content-Type: application/json" -X POST -d "{"""title""":"""http://www.radiostationurl.m3u"""}" http://192.168.0.200:5000/todo/api/v1.0/predvajaj

Why can't I POST to Django with pyCurl?

I've hit something truly strange with a pyCurl script hitting a local Django-Tastypie REST webserver.
Issuing HTTP PUT requests to the server succeeds when I use everything but pycurl (including curl), and fails with error 400 in pycurl.
After much googling and experimentation, I'm stumped. What's wrong here?
Curl call that works:
curl --verbose -X PUT -H 'Content-Type: application/json' -d '{"first_name": "Gaius","id": 1,"last_name": "Balthazar","login": "gbalthazar"}' http://localhost:8000/api/person/1/
PyCurl call that DOESN'T work (error 400):
import pycurl
import StringIO
curl = pycurl.Curl()
url = 'http://localhost:8000/api/person/1/'
curl.setopt(pycurl.URL,url)
curl.setopt(pycurl.VERBOSE, 1)
body = '{"first_name": "Gaius","id": 1,"last_name": "Baltar","login": "gbaltar"}'
curl.setopt(pycurl.READFUNCTION, StringIO.StringIO(body).read)
curl.setopt(pycurl.UPLOAD, 1)
curl.setopt(pycurl.HTTPHEADER,['Content-Type: application/json','Expect:'])
curl.setopt(curl.TIMEOUT, 5)
curl.perform()
(I've tried removing the Expects header as well, I see the header set to 100-Continue in the pycurl call, but same result.)
Unfortunately this project really does need pycurl's low-level access to HTTP timing stats to measure performance, so I can't do it with another HTTP/REST library.
Output of Curl Call:
* About to connect() to localhost port 8000 (#0)
* Trying 127.0.0.1...
* connected
* Connected to localhost (127.0.0.1) port 8000 (#0)
> PUT /api/person/1/ HTTP/1.1
> User-Agent: curl/7.27.0
> Host: localhost:8000
> Accept: */*
> Content-Type: application/json
> Content-Length: 78
>
* upload completely sent off: 78 out of 78 bytes
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Date: Thu, 05 Jun 2014 23:45:26 GMT
< Server: WSGIServer/0.1 Python/2.7.3
< Vary: Accept
< X-Frame-Options: SAMEORIGIN
< Content-Type: application/json
<
* Closing connection #0
{"first_name": "Gaius", "id": 1, "last_name": "Balthazar", "login": "gbalthazar", "pk": "1", "resource_uri": "/api/person/1/"}
Output of PyCurl Verbose Call:
* About to connect() to localhost port 8000 (#0)
* Trying 127.0.0.1...
* connected
* Connected to localhost (127.0.0.1) port 8000 (#0)
> PUT /api/person/1/ HTTP/1.1
User-Agent: PycURL/7.27.0
Host: localhost:8000
Accept: */*
Transfer-Encoding: chunked
Content-Type: application/json
* HTTP 1.0, assume close after body
< HTTP/1.0 400 BAD REQUEST
< Date: Thu, 05 Jun 2014 23:44:25 GMT
< Server: WSGIServer/0.1 Python/2.7.3
< X-Frame-Options: SAMEORIGIN
< Content-Type: application/json
<
* Closing connection #0
{"error": ""}
What am I missing here?

Found the answer:
It needs length of the request body to handle correctly
For POST:
curl.setopt(pycurl.POSTFIELDSIZE, len(body))
For PUT:
curl.setopt(pycurl.INFILESIZE, len(body))
(Yes, it's a different option for different HTTP calls... that's libcurl for you)
Not completely sure what triggers this behaviour, but the above fixes it and the tests work now.
EDIT: Adding verbose pycurl output from this:
* About to connect() to localhost port 8000 (#0)
* Trying 127.0.0.1...
* connected
* Connected to localhost (127.0.0.1) port 8000 (#0)
> PUT /api/person/1/ HTTP/1.1
User-Agent: PycURL/7.27.0
Host: localhost:8000
Accept: */*
Content-Type: application/json
Content-Length: 72
* We are completely uploaded and fine
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Date: Fri, 06 Jun 2014 17:41:38 GMT
< Server: WSGIServer/0.1 Python/2.7.3
< Vary: Accept
< X-Frame-Options: SAMEORIGIN
< Content-Type: application/json
<
* Closing connection #0
{"first_name": "Gaius", "id": 1, "last_name": "Baltar", "login": "gbaltar", "pk": "1", "resource_uri": "/api/person/1/"}

HTTP telnet POST/GAE server question (SIMPLE STUFF)

I am playing with HTTP transfers, just trying to make something work. I have a GAE server and I'm pretty sure it's working properly because it renders when I go to it with my browser, but here is the python code anyway:
import sys
print 'Content-Type: text/html'
print ''
print '<pre>'
number = -1
data = sys.stdin.read()
try:
number = int(data[data.find('=')+1:])
except:
number = -1
print 'If your number was', number, ', then you are awesome!!!'
print '</pre>'
I am just learning the whole HTTP POST vs GET vs Response process, but this is what I have been doing from the terminal:
$ telnet localhost 8080
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET http://localhost:8080/?number=28 HTTP/1.0
HTTP/1.0 200 Good to go
Server: Development/1.0
Date: Thu, 07 Jul 2011 21:29:28 GMT
Content-Type: text/html
Cache-Control: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Content-Length: 61
<pre>
If your number was -1 , then you are awesome!!!
</pre>
Connection closed by foreign host.
I am using a GET here because I stumbled around for about 40 minutes trying to make a telnet POST work - with no success :(
I would appreciate any help on how to get this GET and/or the POST to work. Thanks in advance!!!!

when using GET, no data will be present in the request body, so sys.stdin.read() is bound to fail. instead, you might want to look at the environment, specifically os.environ['QUERY_STRING']
Another thing you're doing a little strangely is you are not using the correct request format. The second part of the request should not include the url scheme, host or port, it should look like:
GET /?number=28 HTTP/1.0
specify the host in a seperate Host: header; the server will determine the scheme on it's own.
When using POST, most servers won't read past the amount of data in the Content-Length header, which if you don't supply one, may be assumed to be zero bytes. The server may try to read any bytes after the point specified by the content-length to be the next request in a persistent connection, and when it doesn't begin with a valid request, it closes the connection. So basically:
POST / HTTP/1.0
Host: localhost: 8080
Content-Length: 2
Content-Type: text/plain
28
But why are you testing this in telnet? How about curl?
$ curl -vs -d'28' -H'Content-Type: text/plain' http://localhost:8004/
* About to connect() to localhost port 8004 (#0)
* Trying ::1... Connection refused
* Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 8004 (#0)
> POST / HTTP/1.1
> User-Agent: curl/7.20.1 (x86_64-redhat-linux-gnu) libcurl/7.20.1 NSS/3.12.6.2 zlib/1.2.3 libidn/1.16 libssh2/1.2.4
> Host: localhost:8004
> Accept: */*
> Content-Type: text/plain
> Content-Length: 2
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Date: Thu, 07 Jul 2011 22:09:17 GMT
< Server: WSGIServer/0.1 Python/2.6.4
< Content-Type: text/html; charset=UTF-8
< Content-Length: 45
<
* Closing connection #0
{'body': '28', 'method': 'POST', 'query': []}
or better yet, in python:
>>> import httplib
>>> headers = {"Content-type": "text/plain",
... "Accept": "text/plain"}
>>>
>>> conn = httplib.HTTPConnection("localhost:8004")
>>> conn.request("POST", "/", "28", headers)
>>> response = conn.getresponse()
>>> print response.read()
{'body': '28', 'method': 'POST', 'query': []}
>>>