We Keep Coding

HTTP request with browser "redirects" request without 302, but with Python Request library it 404's

I'm trying to fetch a page made with react with Python's request.get that returns 404.
import requests
page=requests.get("https://example.com/foo", allow_redirects=True)
print(page.status_code)
results in 404. I see that requests support HTTP 1.1 only.
With curl the URL returns 404 but then the server responds with a different page anyways. The server is using HTTP2. Here are some hints from curl -vv that seem relevant:
$ curl -v https://example.com/foo
* Trying 10.0.0.1
* TCP_NODELAY set
* Connected to example.com (10.0.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
[snip]
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
> GET /foo HTTP/2
> Host: example.com
> User-Agent: curl/7.61.1
> Accept: */*
[snip]
< HTTP/2 404
< date: Fri, 08 Apr 2022 08:42:34 GMT
< content-type: text/html; charset=utf-8
< cache-control: public, max-age=0, s-maxage=300
< etag: W/"a568501bae2318d9d0ca13a89359638e"
< last-modified: Fri, 10 Sep 2021 17:30:40 UTC
< strict-transport-security: max-age=315360000; includeSubdomains; preload
< vary: Accept-Encoding
< x-content-type-options: nosniff
< cf-cache-status: MISS
[snip some cloudflare stuff]
Then finally followed by the content of https://example.com/bar,
The headers of the response seem to indicate it "offers" http 1.1, so how do I ask for it it with the request library?
Searching, I see httpx as a http2 compliant library but their examples have snippets of async but assuming background knowledge of the same. Do I have to use httpx for http2 on Python 3.8 or greater? Is there a way to do this without async/await?

You can use httpx for http/2, you have a specific part of the doc explaining how to activate it, you need at least python 3.6, the doc suggest the use of async by it's examples.
pip install httpx[http2]
From the doc:
client = httpx.AsyncClient(http2=True)
but you can use it with the classic Client:
import httpx
client = httpx.Client(http2=True)
if __name__ == "__main__":
resp = client.get('https://example.com/foo')
print(resp.content)

Replacing Python requests with tornado HTTPclient doesn't work with basic auth

There is a simple code on Python using requests module to make 1 get request with basic authorization. It works.
Then raised a need to rewrite this code to use tornado HTTP client. It doesn't work. Server returns 500 code. I tried HTTP client and AsyncHTTP client, both doesn't work.
Also I have other request, it need authorization token instead of basic auth and it works with Tornado AsyncHTTP client
Here is working and non-working examples with logs. Could you please help to find the reason of error? HTTP logs are quite similar, I don't see any substant difference. I tried to check non-working example without headers Connection and Content-Length, nothing changed.
I deleted from examples password, user name and host name, but I ckecked a lot of times they are identical in working and non-working examples.
Working example:
import requests
from requests.auth import HTTPBasicAuth
url = "http://myhost:81/oapi/GetDate/4rnm"
details_ = requests.get(url, headers={'x-app-id': "kav_test"}, timeout=30, auth=HTTPBasicAuth("user", "login"))
Log from working example:
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): myhost
send: b'GET /oapi/GetDate/4rnm HTTP/1.1\r\nHost: myhost:81\r\nUser-Agent: python-requests/2.18.4\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nx-app-id: kav_test\r\nAuthorization: Basic foo==\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
DEBUG:urllib3.connectionpool:myhost:81 "GET /oapi/GetDate/4rnm HTTP/1.1" 200 None
header: Transfer-Encoding header: Content-Type header: Server header: Access-Control-Allow-Origin header: Access-Control-Allow-Headers header: Access-Control-Allow-Credentials header: Access-Control-Allow-Methods header: Access-Control-Max-Age header: Set-Cookie header: Date {"date":"2021-01-01","hasErrors":false,"errors":[]}
non-working example:
from tornado import httpclient, ioloop
http_client_ = httpclient.AsyncHTTPClient()
...
url = "http://myhost:81/oapi/GetDate/4rnm"
response = await http_client_.fetch(url, method='GET', auth_username="user", auth_password="login", headers={'x-app-id': "test", "Connection": "keep-alive", "Content-Length": "0"}
Log from non-working example:
There is an error Host not found in the end, but I don't understand why, because on the top of log there is a connection with the the host.
DEBUG:asyncio:Using selector: SelectSelector
DEBUG:tornado.curl_httpclient:GET myhost:81/oapi/GetDate/4rnm (username: 'foo')
DEBUG:tornado.curl_httpclient:Trying myip...
DEBUG:tornado.curl_httpclient:TCP_NODELAY set
DEBUG:tornado.curl_httpclient:Connected to myhost (myip) port 81 (#0)
DEBUG:tornado.curl_httpclient:Server auth using Basic with user 'foo'
DEBUG:tornado.curl_httpclient:> GET /oapi/GetDate/4rnm HTTP/1.1
DEBUG:tornado.curl_httpclient:> Host: myhost:81
DEBUG:tornado.curl_httpclient:> Authorization: Basic foo==
DEBUG:tornado.curl_httpclient:> User-Agent: Mozilla/5.0 (compatible; pycurl)
DEBUG:tornado.curl_httpclient:> Accept: */*
DEBUG:tornado.curl_httpclient:> Accept-Encoding: gzip,deflate
DEBUG:tornado.curl_httpclient:> X-App-Id: kav_test
DEBUG:tornado.curl_httpclient:> Connection: keep-alive
DEBUG:tornado.curl_httpclient:> Content-Length: 0
DEBUG:tornado.curl_httpclient:>
Traceback (most recent call last):
DEBUG:tornado.curl_httpclient:< HTTP/1.1 500 Internal Server Error
DEBUG:tornado.curl_httpclient:< Transfer-Encoding: chunked
DEBUG:tornado.curl_httpclient:< Content-Type: application/json
tornado.httpclient.HTTPClientError: HTTP 500: Internal Server Error
DEBUG:tornado.curl_httpclient:< X-Error-Msg:
Host (myhost:81) not found.
(x-local://wtv/wtv_tools_web.xml, line 5689)
(x-local://wtv/wtv_tools_web.xml, line 6149)
(user_init(), wt\web\openapi.html, line 976)
(Authorize(), wt\web\openapi.html, line 896)
(wt\web\openapi.html, line 1079)

bad request socket python

I'm using socket to build a simple "web browser" but I'm getting stuck at the start, whit a bad request result, here is my code:
import socket
mysocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
URI = 'data.pr4e.org'
mysocket.connect((URI, 80))
cmd = "GET http://{0}/romeo.txt HTTP/1.0\n\n".format(URI).encode()
mysocket.send(cmd) # send a request
while True:
data = mysocket.recv(512) # recieve 512 bites at time
# if there is no more information to recive, then, close the loop
if (len(data) < 1):
break
print(data.decode())
pass
mysocket.close() # close connection
here is the output
HTTP/1.1 400 Bad Request
Date: Mon, 15 Feb 2021 14:36:06 GMT
Server: Apache/2.4.18 (Ubuntu)
Content-Length: 308
Connection: close
Content-Type: text/html; charset=iso-8859-1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</
h1>
<p>Your browser sent a request that this server could not understand.<br />
</p>
<hr>
<address>Apache/2.4.18 (Ubuntu) Server at do1.dr-chuck.com Port 80</address>
what I'm doing wrong? also, I tryed replacing data.pr4e.org by facebook.com and youtube.com and I get this output:
HTTP/1.1 301 Moved Permanently
Vary: Accept-Encoding
Location: https://facebook.com/
Content-Type: text/html; charset="utf-8"
X-FB-Debug: LPmWQm0VVptVpi8QX8/SxymrJg9ZoL/mL+W+G4pZA4HGj5WI5YIG1s8sgqwp6TIleGvUg3U1eDNEhGoCsaJG5g==
Date: Mon, 15 Feb 2021 14:52:43 GMT
Alt-Svc: h3-29=":443"; ma=3600,h3-27=":443"; ma=3600
Connection: close
Content-Length: 0
thank you

Here the problem is just that you used \n when the server expected \r\n for end of line.
Anyway, as you directly connect to the HTTP host, you should not put the full URI in the request line. This would be better on a HTTP 1.0 conformance point:
cmd = "GET /romeo.txt HTTP/1.0\r\n\r\n".encode()
But if the server could accept more that one virtual server, you should pass the name in a Host header:
cmd = "GET /romeo.txt HTTP/1.0\r\nHost: {}\r\n\r\n".format(URI).encode()

Python Socket Connect poloniex

Hi I Want Python Socket Connect Poloniex API.
I ran the code. But I can not get the results I want.
I Made Code:
===================================================================
import requests
import socket
s=socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("poloniex.com",443))
message="GET /public?command=returnTicker HTTP/1.1\r\nHost: poloniex.com\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: python-requests/2.18.1\r\n\r\n"
s.send(message)
print s.recv(5000)
===================================================================
Response Text:
HTTP/1.1 400 Bad Request
Server: cloudflare-nginx
Date: Tue, 20 Jun 2017 02:52:22 GMT
Content-Type: text/html
Content-Length: 275
Connection: close CF-RAY: - 400 The plain HTTP request was sent to HTTPS port
===================================================================

The error message is right - you're sending an HTTP request to port 443 which is the HTTPS port. If you want to send an HTTP request, use port 80. I have just tried to send a request to port 80, and the response says I should be using HTTPS from now on (see Location: https:// part):
HTTP/1.1 301 Moved Permanently
Date: Tue, 20 Jun 2017 13:40:52 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=d28a8f446379618a093014a5f13bbcb141497966052; expires=Wed, 20-Jun-18 13:40:52 GMT; path=/; domain=.poloniex.com; HttpOnly
Location: https://poloniex.com/public?command=returnTicker
Server: cloudflare-nginx
CF-RAY: 371f2473b09f5a7a-BOS
In this case you should use whether ssl module instead of socket, or just use requests since it is a simpler option.

HTTP telnet POST/GAE server question (SIMPLE STUFF)

I am playing with HTTP transfers, just trying to make something work. I have a GAE server and I'm pretty sure it's working properly because it renders when I go to it with my browser, but here is the python code anyway:
import sys
print 'Content-Type: text/html'
print ''
print '<pre>'
number = -1
data = sys.stdin.read()
try:
number = int(data[data.find('=')+1:])
except:
number = -1
print 'If your number was', number, ', then you are awesome!!!'
print '</pre>'
I am just learning the whole HTTP POST vs GET vs Response process, but this is what I have been doing from the terminal:
$ telnet localhost 8080
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET http://localhost:8080/?number=28 HTTP/1.0
HTTP/1.0 200 Good to go
Server: Development/1.0
Date: Thu, 07 Jul 2011 21:29:28 GMT
Content-Type: text/html
Cache-Control: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Content-Length: 61
<pre>
If your number was -1 , then you are awesome!!!
</pre>
Connection closed by foreign host.
I am using a GET here because I stumbled around for about 40 minutes trying to make a telnet POST work - with no success :(
I would appreciate any help on how to get this GET and/or the POST to work. Thanks in advance!!!!

when using GET, no data will be present in the request body, so sys.stdin.read() is bound to fail. instead, you might want to look at the environment, specifically os.environ['QUERY_STRING']
Another thing you're doing a little strangely is you are not using the correct request format. The second part of the request should not include the url scheme, host or port, it should look like:
GET /?number=28 HTTP/1.0
specify the host in a seperate Host: header; the server will determine the scheme on it's own.
When using POST, most servers won't read past the amount of data in the Content-Length header, which if you don't supply one, may be assumed to be zero bytes. The server may try to read any bytes after the point specified by the content-length to be the next request in a persistent connection, and when it doesn't begin with a valid request, it closes the connection. So basically:
POST / HTTP/1.0
Host: localhost: 8080
Content-Length: 2
Content-Type: text/plain
28
But why are you testing this in telnet? How about curl?
$ curl -vs -d'28' -H'Content-Type: text/plain' http://localhost:8004/
* About to connect() to localhost port 8004 (#0)
* Trying ::1... Connection refused
* Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 8004 (#0)
> POST / HTTP/1.1
> User-Agent: curl/7.20.1 (x86_64-redhat-linux-gnu) libcurl/7.20.1 NSS/3.12.6.2 zlib/1.2.3 libidn/1.16 libssh2/1.2.4
> Host: localhost:8004
> Accept: */*
> Content-Type: text/plain
> Content-Length: 2
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Date: Thu, 07 Jul 2011 22:09:17 GMT
< Server: WSGIServer/0.1 Python/2.6.4
< Content-Type: text/html; charset=UTF-8
< Content-Length: 45
<
* Closing connection #0
{'body': '28', 'method': 'POST', 'query': []}
or better yet, in python:
>>> import httplib
>>> headers = {"Content-type": "text/plain",
... "Accept": "text/plain"}
>>>
>>> conn = httplib.HTTPConnection("localhost:8004")
>>> conn.request("POST", "/", "28", headers)
>>> response = conn.getresponse()
>>> print response.read()
{'body': '28', 'method': 'POST', 'query': []}
>>>