We Keep Coding

When should I close the socket, if the request contains Connection: keep-alive?

When I am using this Python script to send a request to my server:
import requests as r
url = "http://localhost:8070/"
response = r.get(url=url)
It sends the following request:
GET / HTTP/1.1
Host: localhost:8070
User-Agent: python-requests/2.27.1
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
If I understand it correctly, Connection: keep-alive means that I should not close the client-socket, because the client could use it again.
But if I do not close the client-socket, the Python script just gets stuck until I close the socket. Is there another way to indicate that the request is complete so pythons requests understands it?
If I try to send the request to any other server, the script almost immediately finishes. My guess would be to timeout the client after a few milliseconds e.g. by using select like this:
struct timeval timeout;
timeout.tv_sec = 0;
timeout.tv_usec = 1000;
select_ret = select(this->_maxfds + 1, &this->_readfds, &this->_writefds, NULL, &timeout);
And now I would just close the client-socket after select returns 0:
if (select_ret == 0) {
close(client_socket);
}
Is that a valid approach, or am I missing something?
I am sending the response like this:
char *response = "HTTP/1.1 200 Ok\r\n\r\n";
send(this->_client_socket, response, strlen(response), 0)
BUT this does not terminate the python script. The python script still hangs after I execute this line of code. It only finishes when I close the socket on my side.
So how would I determine if I should close it or not? As I already said my approach was to use a timeout in case no data is getting written in to the socket from the client side.

Your server's response is incomplete.
Your understanding of Connection: keep-alive in the request is correct. However, there is no Content-Length or Transfer-Encoding: chunked header present in your response, so the only way the client has to know when the response is finished is to wait for the socket connection to be closed on the server side. Read the rules outlined in RFC 2616 Section 4.4 and RFC 7230 Section 3.3.3 of the HTTP 1.1 protocol spec.
Try something more like this instead:
const char *response = "HTTP/1.1 200 Ok\r\nContent-Length: 0\r\n\r\n";
send(this->_client_socket, response, strlen(response), 0)

how to get html file into code in python?

how to get html file into python code using socket. I was able to implement using the requests library. However, it needs to be rewritten to sockets. I don’t understand how. The implementation code through requests will be below. I will also leave pathetic attempts to implement via a socket using Google. However, the decision is not at all correct. ! (Help implement using sockets.
import requests
reg_get = requests.get("https://stackoverflow.blog/")
text = reg_get.text
print(text)
import socket
request = b"GET / HTTP/1.1\nHost: https://stackoverflow.blog/\n\n"
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("https://stackoverflow.blog/", 80))
s.send(request)
result = s.recv(10000)
while (len(result) > 0):
print(result)
result = s.recv(10000)
After seeing the comments and listening to you. I have rewritten the following code. However, I never got the html. And I received information about the site. How do I get html structure in python
import socket
import ssl
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
request = "GET /r/AccidentalRenaissance/comments/8ciibe/mr_fluffies_betrayal/ HTTP/1.1\r\nHost: www.reddit.com\r\n\r\n"
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
s = context.wrap_socket(sock, server_hostname = "www.reddit.com")
s.connect(("www.reddit.com", 443))
s.sendall(request.encode())
contest = s.recv(1024).decode()
s.close()
print(contest)
result
HTTP/1.1 200 OK
Connection: keep-alive
Cache-control: private, s-maxage=0, max-age=0, must-revalidate, no-store
Content-Type: text/html; charset=utf-8
X-Frame-Options: SAMEORIGIN
Accept-Ranges: bytes
Date: Sun, 03 Oct 2021 03:34:25 GMT
Via: 1.1 varnish
Vary: Accept-Encoding, Accept-Encoding

A URL is composed of a protocol, a hostname, an optional port, and an optional path. In the URL http://stackoverflow.blog/ , https is the protocol, stackoverflow.blog is the hostname, and no port or path is provided. For http, the port defaults to 80 and the path defaults to /. When using sockets, first establish a connection to the host at the port using connect then send an HTTP command to retrieve the page on the path. The HTTP command to retrieve the page is "GET /" and receive the response from the server.
Note that I used http instead of https because https adds security set up and negotiation to the above that occurs once the connect is done but before the "GET /" is done. It is quite complicated and a good reason to use Requests instead of trying to implement it yourself. If you don't want to use Requests but don't want to go down to the level of sockets, take a look at urllib3

How to connect a client to a server through an HTTP proxy in Python?

I'm really new to coding using sockets.
I like the socket library, I get to understand a big part of what's happening in my program, so i you don't mind i would like to stick with it.
So as the title says, I have a socket based client and server and I would like to exchange content through an HTTP proxy(I'm using a Squid proxy). This little piece of code is supposed to bypass the proxy in my campus to simulate a chat over the campus network. This is totally legal since I asked the IT guys that work there.
Here's the deal, I am able to send a POST request through the proxy to my server which receives it and sends it back to client 1, but when I try to send more requests to the proxy none of them gets to the server so I think to my self the connection died but here's the thing, when I send messages from client 2 which is connected directly to the server, the server AND client 1 receive them.
import socket
from _thread import *
def sender(server,h):
b=input("<<--Send--")
b=h
server.send(b.encode())
PROXY_IP="127.0.0.1"
PROXY_PORT=3128
server=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
server.settimeout(0.05)
server.connect((PROXY_IP,PROXY_PORT))
header="""POST http://127.0.0.1:3001 HTTP/1.1\r\n
Host: 127.0.0.1:3001\r\n
Proxy-Connection: keep-alive\r\n
Content-Length: 5 \r\n\r\n
hello\r\n"""
server.send(header.encode())
while 1:
try:
start_new_thread(sender,(server,header))
a=server.recv(1024)
print("-->>{}".format(a.decode()))
except KeyboardInterrupt:
break
except:
pass
server.close()
I already tried the CONNECT method which works perfectly, but it's not allowed in my campus network proxy.
What am I doing wrong ?
Is there something I should know about how to re-send content through a proxy ?
Thank you for your time and please bear with me..
Here's what I get on the client that sends a request to the proxy:
~#Sent : POST http://127.0.0.1:3001 HTTP/1.1
Host: 127.0.0.1:3001
Proxy-Connection: keep-alive
Content-Length: 5
hello
#Received : HTTP/1.1 200 OK
Server: squid/3.5.19
Mime-Version: 1.0
Date: Mon, 10 Oct 2016 00:46:39 GMT
X-Transformed-From: HTTP/0.9
X-Cache: MISS from kali
X-Cache-Lookup: MISS from kali:3128
Transfer-Encoding: chunked
Via: 1.1 kali (squid/3.5.19)
Connection: keep-alive
#Received : B2
POST / HTTP/1.1
Content-Length: 5
Host: 127.0.0.1:3001
Via: 1.1 kali (squid/3.5.19)
X-Forwarded-For: 127.0.0.1
Cache-Control: max-age=259200
Connection: keep-alive
hello
#Sent : POST http://127.0.0.1:3001 HTTP/1.1
Host: 127.0.0.1:3001
Proxy-Connection: keep-alive
Content-Length: 5
hello
Nothing is received after this..

POST http://127.0.0.1:3001 HTTP/1.1\r\n
Host: 127.0.0.1:3001\r\n
Proxy-Connection: keep-alive\r\n
Content-Length: 5 \r\n\r\n
hello\r\n
The body of your HTTP response consists of 7 bytes not 5 as you've stated in your Content-length. The \r\n after the 5 byte still belong to the response body. Giving the wrong size might mixup request handling, i.e. the proxy is expecting a new request but is actually getting \r\n, i.e. the 2 bytes after your 5 bytes Content-length.
Apart from that both path and Host header must include the name of the target from the perspective of the proxy. Using 127.0.0.1. like in your example would mean that you try to access a server at the same host of the proxy, i.e. localhost from the view of the proxy. This is probably not what you've intended.
...
X-Transformed-From: HTTP/0.9
This header in the response of the proxy indicates that your server does not properly speak HTTP/1.x. Instead of sending HTTP header and body it just sends the payload back without any HTTP header, like done in the HTTP 0.9 protocol which was obsoleted 20 years ago. With HTTP 0.9 the response will always end only at the end of the TCP connection. This means that you cannot have multiple requests within the same TCP connection.
I'm really new to coding using sockets.
The problem is not caused by the wrong use of sockets but due the wrong implementation of the application protocol, i.e. the data send over the socket. If you really need to implement HTTP please study the standards, i.e. RFC 7230 and following. If you don't want to do this use existing and tested HTTP libraries instead of writing your own.

How to get port number from an HTTP response, using Python?

I am trying to simulate Network Address Translation for some test code. I am mapping virtual users to high port numbers, then, when I want to test a request from user 1, I send it from port 6000 (user 2, from 6001, etc).
However, I can't see the port number in the response.
connection = httplib.HTTPConnection("the.company.lan", port=80, strict=False,
timeout=10, source_address=("10.129.38.51", 6000))
connection.request("GET", "/portal/index.html")
httpResponse = connection.getresponse()
connection.close()
httpResponse.status is 200, but I don't see the port number anywhere in the response headers.
Maybe I should be using some lower level socket functionality? If so, which is simplest and supports both HTTP and FTP? Btw, I only want to use built-in modules, nothing which I have to install.
[Update] I should have made it clearer; I really do need to get the actual port number received in the response, not just remember it.

To complete #TimSpence answer, you can use a socket object as an interface for your connection and then treat with some API your data as an HTTP object.
host = 'xxx.xxx.xxx.xxx'
port = 80
address = (host, port)
## socket object interface for a TCP connection
listener_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM,
socket.IPPROTO_TCP)
listener_socket.bind(address)
listener_socker.listen(MAX_CONNECTIONS)
## new_connection is the connection object you use to handle the data echange
## source_address (source_host, source_port) is the address object
## source_port is what you're looking for
new_connection, source_address = listener_socket.accept()
data = new_connection.recv(65536)
## handle data as an HTTP object(for example as an HTTP request)
new_connection.close()

HTTP messages do not contain anything about ports so the httpResponse will not have that information.
However, you will need a different connection object (which will map to a different underlying socket) for each request anyway so you can get that information from the HTTPconnection object.
_, port = connection.source_address
Does that help?

Considring your comments, I had to provide a new answer.
I though you can also put a non standard header host in your HTTPRespose, 'Host: domain/IP:port', so that your client can read it when it receives a response.
Server Response:
HTTP/1.1 200 OK
Date: Day, DD Month YYYY HH:MM:SS GMT
Content-Type: text/html; charset=UTF-8
Content-Encoding: UTF-8
Content-Length: LENGTH
Last-Modified: Day, DD Month YYYY HH:MM:SS GMT
Server: Name/Version (Platform)
Accept-Ranges: bytes
Connection: close
Host: domain/IP:port #exapmple: the.company.lan:80
<html>
<head>
<title>Example Response</title>
</head>
<body>
Hello World!
</body>
</html>
Client:
connection = httplib.HTTPConnection("the.company.lan", port=80,
strict=False, timeout=10,
source_address=("10.129.38.51", 6000))
connection.request("GET", "/portal/index.html")
httpResponse = connection.getresponse()
## store a dict with the response headers
## extract your custom header 'host'
res_headers = dict(httpResponse.getheaders());
server_address = tuple(headers['host'].split(':'))
## read the response content
HTMLData = httpResponse.read(CONTENT_LENGTH_HEADER)
connection.close()
This way you got server_address as a tuple (domain, port).

Python socket module. Connecting to an HTTP proxy then performing a GET request on an external resource

To begin with, I understand there are other modules such as Requests that would be better suited and simpler to use, but I want to use the socket module to better understand HTTP.
I have a simple script that does the following:
Client ---> HTTP Proxy ---> External Resource (GET Google.com)
I am able to connect to the HTTP proxy alright, but when I send the GET request headers for google.com to the proxy, it doesn't serve me any response at all.
#!/usr/bin/python
import socket
import sys
headers = """GET / HTTP/1.1\r\n
Host: google.com\r\n\r\n"""
socket = socket
host = "165.139.179.225" #proxy server IP
port = 8080 #proxy server port
try:
s = socket.socket()
s.connect((host,port))
s.send(("CONNECT {0}:{1} HTTP/1.1\r\n" + "Host: {2}: {3}\r\n\r\n").format(socket.gethostbyname(socket.gethostname()),1000,port,host))
print s.recv(1096)
s.send(headers)
response = s.recv(1096)
print response
s.close()
except socket.error,m:
print str(m)
s.close()
sys.exit(1)

To make a HTTP request to a proxy open a connection to the proxy server and then send a HTTP-proxy request. This request is mostly the same as the normal HTTP request, but contains the absolute URL instead of the relative URL, e.g.
> GET http://www.google.com HTTP/1.1
> Host: www.google.com
> ...
< HTTP response
To make a HTTPS request open a tunnel using the CONNECT method and then proceed inside this tunnel normally, that is do the SSL handshake and then a normal non-proxy request inside the tunnel, e.g.
> CONNECT www.google.com:443 HTTP/1.1
>
< .. read response to CONNECT request, must be 200 ...
.. establish the TLS connection inside the tunnel
> GET / HTTP/1.1
> Host: www.google.com

Python 3 requires the request to be encoded. Thus, expanding on David's original code, combined with Steffens answer, here is the solution written for Python 3:
def connectThroughProxy():
headers = """GET http://www.example.org HTTP/1.1
Host: www.example.org\r\n\r\n"""
host = "192.97.215.348" #proxy server IP
port = 8080 #proxy server port
try:
s = socket.socket()
s.connect((host,port))
s.send(headers.encode('utf-8'))
response = s.recv(3000)
print (response)
s.close()
except socket.error as m:
print (str(m))
s.close()
sys.exit(1)
This allows me to connect to the example.org host through my corporate proxy (at least for non SSL/TLS connections).