We Keep Coding

Can I use Python requests library to send inconsistent requests

I try to write a python script containing a somewhat unusual HTTP request as part of learning about web attacks and solving the lab at
https://portswigger.net/web-security/request-smuggling/lab-basic-cl-te.
There, I need to issue a request containing both a Content-Length and a Transfer-Encoding header that are in disagreement.
My basic and still unmanipulated request looks like this and works as expected:
with requests.Session() as client:
client.verify = False
client.proxies = proxies
[...]
data = '0\r\n\r\nX'
req = requests.Request('POST', host, data=data)
prep = client.prepare_request(req)
client.send(prep)
[...]
Content-Length: 6\r\n
\r\n
0\r\n
\r\n
X
However, as soon as I add the Transfer-Encoding header, the request itself gets modified.
data = '0\r\n\r\nX'
req = requests.Request('POST', host, data=data)
prep = client.prepare_request(req)
prep.headers['Transfer-Encoding'] = 'chunked'
client.send(prep)
The request that is actually send down the wire is
[...]
Content-Length: 0\r\n
\r\n
whereas the expected request would be
[...]
Content-Length: 6\r\n
Transfer-Encoding: chunked\r\n
\r\n
0\r\n
\r\n
X
The same thing happens if I flip things around, prepare a chunked request and modify the Content-Length header afterwards:
def gen():
yield b'0\r\n'
yield b'\r\n'
yield b'X'
req = requests.Request('POST', host, data=gen())
prep = client.prepare_request(req)
prep.headers['Content-Length'] = '6'
client.send(prep)
Basically, the Transfer-Encoding header gets removed completely, the data is reinterpreted according to the chunking and the Content-Length header gets recalculated to match.
I was under the impression that preparing a request and manipulating its content before sending should send the modified content, but either this is a wrong assumption or I do things horribly wrong.
Is sending such a request possible this way or do I have to go onto a lower level to put arbitrary data on the wire?

requests is a good HTTP client, and as such will prevent you from generating bad HTTP queries. As writing bad HTTP queries will result in 400 errors in a lot of cases.
To generate syntax errors in HTTP queries you need to avoid using high level http clients (like a browser, but also like an http library). Instead you need togo down to the tcp/ip socket management (and maybe ssl also) and start writing the full HTTP protocol with your own code, no library.

Parse Error: The server returned a malformed response - Error: Parse Error: Expected HTTP/ - when trying to access one of my app URLs with postman

I am managing an app built by third parts in python.
I have this url dispatcher
urls += [(r'/path/objectAlpha/(.*)', objectAlphaHandler)] # this was made by third parts, it is expected to work
and this class
class objectAlphaHandler(BaseHandler):
def __init__(self, application, request, **kwargs):
super(objectAlphaHandler, self).__init__(application, request, **kwargs) # do the init of the basehandler
#gen.coroutine
def post(self, action=''):
response = {}
...
response = yield self.my_method(json_data)
...
self.write(json.dumps(response))
def my_method(self, json_data)
...
I want to check that the app correctly receives the request and returns some response.
So I try to access that url with Postman
request type:
POST
URL:
http://<machine_ip>:<machine_port>/path/objectAlpha/
I get this error from Postman response box
Parse Error: The server returned a malformed response
and when I click on "view in console" I see
POST http://machine_ip>:<machine_port>/path/objectAlpha/
Error: Parse Error: Expected HTTP/
Request Headers
Content-Type: application/json
User-Agent: PostmanRuntime/7.28.4
Accept: */*
Postman-Token: d644d7dd-699b-4d77-b32f-46a575ae31fc
Host: xx.xxx.x.xx:22
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Request Body
What does Error: Parse Error: Expected HTTP/ mean?
I checked my app logs but it seems it is not handling any request, even if Postman indicates that the server is returning a (malformed) response.
I also tryed to chenge the target url to:
https...
but it returns
Error: write EPROTO 28427890592840:error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER:../../third_party/boringssl/src/ssl/tls_record.cc:242:
which I found out it indicates I should stick with HTTP
Then I tried also:
http://<machine_ip>/path/objectAlpha/
and
<machine_ip>/path/objectAlpha/
which generically return:
Error: connect ECONNREFUSED <machine_ip>:80
I also tryed to substitute line
urls += [(r'/path/objectAlpha/(.*)', objectAlphaHandler)]
with
urls += [(r'/path/objectAlpha/', objectAlphaHandler)]
and
urls += [(r'/path/objectAlpha', objectAlphaHandler)]
but none of these worked.
What is wrong? How can I fix it?
UPDATE
Apparently, according to this thread on Postman Github, the problem happens only on Postman Desktop and not on Postman on browser.
So I tryed to send the request form Postman on my browser but I get
Cloud Agent Error: Can not send requests to reserved address. Make sure address is publicly accessible or select a different agent.
because, according to this other thread,
Postman Website cannot send a request to your computer's localhost. It first needs to connect to your PC with the Postman desktop client
and even if I follow the indications in that answer
Run it [ndr. Postman desktop], then go to the Postman workspace in your browser -> send the request and it will work.
I still get the same
Error: Parse Error: Expected HTTP/
on both Postman Desktop and Postman on browser.
UPDATE
Going on debugging, I tryed to cast a curl on that URL from my terminal:
myuser#mymachine-VirtualBox:~$ curl --verbose "http://<target_machine_ip>:<target_machine_port>/path/objectAlpha"
and I got:
* Trying <target_machine_ip>:<target_machine_port>...
* Connected to <target_machine_ip> (<target_machine_ip>) port <target_machine_port> (#0)
> GET /orkpos5/receipt HTTP/1.1
> Host: <target_machine_ip>:<target_machine_port>
> User-Agent: curl/7.74.0
> Accept: */*
>
* Received HTTP/0.9 when not allowed
* Closing connection 0
curl: (1) Received HTTP/0.9 when not allowed

SOLVED
Since the exposed API has been built by third parts for internal use, it is not exposed to the public.
I did not know that, so in the request URL I had put the well-known port number for HTTP requests, the number 22 (see the list of well-known port numbers.).
To solve the problem, in the request URL, I changed the <target_machine_port> to the port on which the API is actually exposed.

Postman POST request works but doesnt when exported to cURL / request / http.request

I am completly lost because I got no idea anymore what I am doing wrong.
I want to make a simple POST request to a certain address. I visited the website using firefox, opened it's terminal, copied the POST request as cURL and executed in the terminal. The recieved response has status 200 but it's body is unreadable like "������q9i".
But when I use Postman->Import->cURL and execute the request it works?! Also status 200 but this time the body contains a properly readable html code just as expected.
So I though maybe it's because Postman is adjusted the request. So I opened the code panel to the right side of the program and exported Postman's request again as cURL, python - http.client and python - request, but none of them are working?! Again I just recieve an unreadable body. How on earth can this happen?
I'm using the same machine for all requests, there is no VPN or something so it cannot be related to IP address. There is no authentification or anything.
There is just maybe one hint I noticed: The response recieved in Postman is exactly one byte shorter then the one recieved in cURL or python. Could this be the problem? Is Postman handling the response's body differently?
I appreciate any help a lot!

cURL is displaying the raw response of the body, while Postman and Firefox process the response. In your case, I suspect that you request a zipped response with a header like Accept-Encoding: gzip, deflate. If you remove that header, you will get the uncompressed response.
If there is no such header in your request, it would be good to see the request you are trying to execute.

Can HTTP 200 response have incomplete data?

I am requesting pdf binary content from a Tomcat Webserice from a Python Web Application Server.
We have implemented a 2 times retry like this in Python. Once in a while we get an HTTP 500 Response. This issue is being investigated however it is very likely to be an environment issue related to insufficient resources like maximum no: of process reached etc. In the next retry, more often than not, we get an HTTP 200 with partial blob content (i.e with EOF Marker in PDF). How is that possible?
Is there any flaws in this retry logic? How can HTTP 200 response have incomplete data is beyond my understanding. Is the HTTP 200 sent first and then the real data (which means there is a possibility of server dying after it sent HTTP 200)? The only other explanation is that server is sending the entire content but the program that is generating the data is sending incomplete data due to some resource issues which might have also caused HTTP 500.
# There is a unique id as well to make it new request. (retries is 2 by default)
while retries:
try:
req = urllib2.Request(url, data=input_html)
req.add_header('Accept', 'application/pdf')
req.add_header('Content-Type', 'text/html')
handle = urllib2.urlopen(req)
pdf_blob = handle.read()
except:
log(traceback)
retries = retries - 1
if not retries:
raise
Architecture is as follows:
Web Application -> Calls Tomcat -> Gets PDF -> Stores To DB.

Should a webserver ignore the request body when request method is not POST or PUT?

I was working on a simple API server using tornado and all requests require the parameter access_token. I was playing with curl, and was surprised to find that DELETE and GET requests will not extract this value from the request body--they only allow this param to be passed via the query string.
ie, when I do
curl -i -X DELETE -d access_token=1234 http://localhost:8888/
In the delete method of my web handler, this returns None:
self.get_argument('access_token', None)
However, when I do
curl -i -X DELETE http://localhost:8888/?access_token=1234
This yields "1234" as expected:
self.get_argument('access_token', None)
I examined the tornado source, and found that the body is only parsed for POST and PUT requests: https://github.com/facebook/tornado/blob/4b346bdde80c1e677ca0e235e04654f8d64b365c/tornado/httpserver.py#L258
Is it correct to ignore the request body for GET, HEAD, and DELETE requests, or is this a choice made by the authors of tornado?

This is correct per the HTTP/1.1 protocol specification.
DELETE and GET requests do not accept entity data enclosed in the request.
According to the definition, get requests retrieve their entity data from the request URI.
HEAD requests are defined as identical to GET requests except that the server should not return a message body in the response.
Therefore the authors of tornado were correct to ignore the "post" data for GET, HEAD, and DELETE.
See HTTP/1.1 Method Definitions

It is a good idea to not to accept requests with the payload if they are not POST or PUT. Just because of security reasons. Some servers, e.g. lighttpd, return server error in this case.