We Keep Coding

Python Socket HTTP 1.1 keep-alive for a HEAD request

For the HTTP 1.1 protocol, the connections are persistent (keep-alive).
The client should send Connection:close header attribute to close the connection.
In a Python program, this is the case for a GET request. However, a connection for a HEAD request is closed without the Connection:close header.
What is the issue?
I have also tested a Java version of a HEAD request, and the connection is persistent there.
Python program for a HEAD request:
#!/usr/bin/env python
import socket
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.connect(("webcode.me" , 80))
s.sendall(b"HEAD / HTTP/1.1\r\nHost: webcode.me\r\nAccept: text/html\r\n\r\n")
print(str(s.recv(1024), 'utf-8'))
Python program for a GET request:
#!/usr/bin/env python
import socket
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.connect(("webcode.me" , 80))
s.sendall(b"GET / HTTP/1.1\r\nHost: webcode.me\r\nAccept: text/html\r\nConnection: close\r\n\r\n")
# s.sendall(b"GET / HTTP/1.0\r\nHost: webcode.me\r\nAccept: text/html\r\n\r\n")
while True:
data = s.recv(512)
if not data:
break
print(data.decode())

For the HTTP 1.1 protocol, the connections are persistent (keep-alive)
No, the connections can be persistent if the server also wants them to be persistent. The server might decide to close the connection immediately, 5 seconds after ... or even never by its own if the client signals support for persistence.
However, a connection for a HEAD request is closed without the Connection:close header.
It is your client which is closing the connection, not the server. Your client does a single recv and then it is done with the socket and the program. If one would modify the code to continue with recv until no more data can be read then (similar to your second program) then the client would hang since the server is waiting for the new request from the client.

PUT request blocking too long with simple custom HTTP client and server

I have implemented a simple HTTP server and a client. The latter issues a PUT request using the requests library, sending some arbitrary JSON and then exits.
When I start the server, and then run the client, both the server and the client block. The server however appears to not have gone through the entire handler function yet.
This is what I get on server side:
$ python3 server.py
PUT / HTTP/1.1
That is, after printing the request line, the content JSON string is not printed. At this point both client and server block for some reason.
Interestingly, when I trigger a KeyboardInterrupt to the client, the server proceeds:
$ python3 server.py
PUT / HTTP/1.1
b'{"content": "Hello World"}'
127.0.0.1 - - [25/Feb/2016 11:52:54] "PUT / HTTP/1.1" 200 -
My questions:
Why is it necessary to kill the client to let the server proceed?
Am I using any of these components the wrong way?
How can I make client and server to operate (nearly) instantaneously?
This is the code of the HTTP server. It only handles PUT requests. It prints the request line and the content data and responds using the success code to the client:
import http.server
class PrintPUTRequestHandler(http.server.BaseHTTPRequestHandler):
def do_PUT(self):
print(self.requestline)
print(self.rfile.read())
self.send_response(200)
self.end_headers()
server_address = ('', 8000)
httpd = http.server.HTTPServer(server_address, PrintHTTPRequestHandler)
httpd.serve_forever()
This is the HTTP client. It is intended to connect to the server, write the request and return as soon as possible (but it doesn't):
import requests
server_address = "http://127.1:8000"
data = '{"content": "Hello World"}'
requests.put(server_address, data, headers={"Content-type": "application/json"})
This is how I run it after the server has started (no output observable):
python client.py

The server blocks both itself and the client on this line:
print(self.rfile.read())
That happens because you didn't specify the amount of data to be read so the server reads the input stream until it is closed. And in your case the input stream is closed once you kill the client.
Remember that the server doesn't know a priori when the streaming of data ends because you may want to send data chunk by chunk (for example when you send big files).
The size of request should be passed in Content-Length header so this is what you should do:
length = int(self.headers['Content-Length'])
print(self.rfile.read(length))
That's assuming that the length is small enough to fit in your memory (and in your case it is).

In python : How to perform a HTTP request, and connect with a websocket, through the same connection?

I am trying to connect to a website through a program, whose authentification protocol appears to be the following :
Connect to the websocket.
Receive crucial data for logging in.
Make a HTTP request with username/password/some of the datas THROUGH THIS CONNECTION.
This means that if i open another connection, the data linked to this connection, according to the server, will be different.
For exemple, receiving the data through the websocket, and doing a HTTP request with a browser will not work, as the browser is another connection.
By connection, i mean what is created at the start of the program, for exemple :
ws2= websocket.create_connection("ws://sim.smogon.com:8000/showdown/websocket") for the websocket module
h1 = httplib.HTTPConnection('www.cwi.nl') for the http module.
My problem is that i have to create 2 different connections to connect to websocket and make a http request. This means that the http request will not work with the websocket received data.
How to make these 2 connections the same ? Or, simplier said : how to solve the problem ?

How do I send and receive HTTP POST requests in Python?

I have these two Python scripts I'm using to attempt to work out how to send and receive POST requests in Python:
The Client:
import httplib
conn = httplib.HTTPConnection("localhost:8000")
conn.request("POST", "/testurl")
conn.send("clientdata")
response = conn.getresponse()
conn.close()
print(response.read())
The Server:
from BaseHTTPServer import BaseHTTPRequestHandler,HTTPServer
ADDR = "localhost"
PORT = 8000
class RequestHandler(BaseHTTPRequestHandler):
def do_POST(self):
print(self.path)
print(self.rfile.read())
self.send_response(200, "OK")
self.end_headers()
self.wfile.write("serverdata")
httpd = HTTPServer((ADDR, PORT), RequestHandler)
httpd.serve_forever()
The problem is that the server hangs on self.rfile.read() until conn.close() has been called on the client but if conn.close() is called on the client the client cannot receive a response from the server. This creates a situation where one can either get a response from the server or read the POST data but never both. I assume there is something I'm missing here that will fix this problem.
Additional information:
conn.getresponse() causes the client to hang until the response is received from the server. The response doesn't appear to be received until the function on the server has finished execution.

There are a couple of issues with your original example. The first is that if you use the request method, you should include the message body you want to send in that call, rather than calling send separately. The documentation notes send() can be used as an alternative to request:
As an alternative to using the request() method described above, you
can also send your request step by step, by using the four functions
below.
You just want conn.request("POST", "/testurl", "clientdata").
The second issue is the way you're trying to read what's sent to the server. self.rfile.read() attempts to read the entire input stream coming from the client, which means it will block until the stream is closed. The stream won't be closed until connection is closed. What you want to do is read exactly how many bytes were sent from the client, and that's it. How do you know how many bytes that is? The headers, of course:
length = int(self.headers['Content-length'])
print(self.rfile.read(length))
I do highly recommend the python-requests library if you're going to do more than very basic tests. I also recommend using a better HTTP framework/server than BaseHTTPServer for more than very basic tests (flask, bottle, tornado, etc.).

Long time answered but came up during a search so I bring another piece of answer. To prevent the server to keep the stream open (resulting in the response never being sent), you should use self.rfile.read1() instead of self.rfile.read()

How to connect a Python socket on client-side to Node.js/socket.io?

I want to connect Blender (v2.55) to a webpage through sockets.
For the web part, I can use Node.js & socket.io. I've already used a little node.js/socket.io, it's not a problem I think.
Now for Blender, it runs on Python 3.1, so I've already sockets and I can add libraries if needed. I'm new to Python sockets, can I connect a client to node.js/socket.io directly ?
I tried with the basic code from the Python doc:
import socket
import sys
HOST, PORT = "127.0.0.1", 8080
data = "Hello from Blender"
# Create a socket (SOCK_STREAM means a TCP socket)
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# Connect to server and send data
sock.connect((HOST, PORT))
sock.send(bytes(data + "\n","utf8"))
# Receive data from the server and shut down
received = sock.recv(1024)
sock.close()
print("Sent: %s" % data)
print("Received: %s" % received)
It results by:
Sent: Hello from Blender
Received: b''
It seems that Blender is connected, but doesn't receive data. Also Node shows no new client connected…
Do I need something else ? If somebody can help me out…

You are missing a protocol/handshake. What you have there is a bare TCP socket connection. node.js/socket.io lives on top of a TCP socket. Basically when you open a connection to a socket.io server, it's expecting you to use some protocol for communication (websockets, longpolling, htmlfile, whatever). The initial handshake defines what that protocol will be. Websockets is one of the supported protocols. This blog post should help you. It doesn't look all that hard to get websockets implemented.

you can try the form of loop to receive valid data.
import socket
host="127.0.0.1"
port=8088
web=socket.socket()
web.bind((host,port))
web.listen(5)
print("recycle")
while True:
conn,addr=web.accept()
data=conn.recv(8)
print(data)
conn.sendall(b'HTTP/1.1 200 OK\r\n\r\nHello world')
conn.close()
and use your browser to visit the host and port for a check

I understand this thread is extremely old. But I faced the same problem recently and couldn't find an answer or any similar questions. So here is my answer.
Answer: Use socket.io for python python-socketio
The reason why built-in sockets or any other websocket library in python won't work is explained in the socket.io website socket.io
Socketio is simply just not a websoket connection. Although they say, it uses websockets for transport internally, the connection is established with HTTP protocol http:// as opposed to the WEBSOCKET protocol ws://. This results in the failure of handshake and the connection fails to be established.