We Keep Coding

http.server rfile read blocking in python

Trying to build a simple python3 HTTP Server with http.server:
import http.server
from http.server import HTTPServer, BaseHTTPRequestHandler
import socketserver
class Handler(BaseHTTPRequestHandler):
def do_PATCH(self):
print ("do_Patch called")
contentLength = int(self.headers['content-length'])
res = self.rfile.read(contentLength)
self.send_response(200)
self.send_header('Content-type', 'text/plain')
self.end_headers()
self.wfile.write('Something brilliant'.encode())
httpd = HTTPServer(("127.0.0.1",33181),Handler)
httpd.serve_forever()
Problem is that self.rfile.read() is blocking, so if I don't get the content-length exactly right I either get too little data or I hang.. Docs I've found says it's supposed to throw an exception in non-blocking mode but I've not found out how to set non-blocking mode.
I can get around this by setting the content-length as above, but I can also corrupt the content-length which hangs the thread:
import http.client
myValueableString = "Shorter than 300"
for mylen in [len(myValueableString), 300]:
conn = http.client.HTTPConnection("127.0.0.1",33181)
conn.request("PATCH","/test",myValueableString ,{'Content-length': mylen})
print(conn.getresponse().read().decode())
Is there any good way to 1) Force finding the "true" length of rfile stream, or 2) Force a timeout or 3) Enable "non-blocking" mode? I realize the code isn't recommended for production but this is the only real stumbling block I've encountered.

Read HTTP request data in Python 3?

I am trying to write a basic "echo" HTTP server that writes back the raw data it receives in the request. How can I get the request data as a string?
This is my program:
#!/usr/bin/env python
from http.server import HTTPServer, BaseHTTPRequestHandler
class RequestHandler(BaseHTTPRequestHandler):
def do_GET(self):
print('data', self.rfile.readall())
self.send_response(200)
self.send_header('Content-Type', 'text/html')
self.end_headers()
message = 'Hello Client!'
self.wfile.write(bytes(message, 'utf8'))
return
def server_start():
address = ('', 1992)
httpd = HTTPServer(address, RequestHandler)
httpd.serve_forever()
server_start()
Error:
self.rfile.readall(): '_io.BufferedReader' object has no attribute 'readall'

If it's a get request there won't be a body, so the data is going to be sent in the url.
from http.server import HTTPServer, BaseHTTPRequestHandler
import urlparse
class RequestHandler(BaseHTTPRequestHandler):
def do_GET(self):
parsed_path = urlparse.urlparse(self.path)
print(parsed_path.query)
...
Otherwise, you should implement a POST method if you want to send any more complex object as data (take a look at different HTTP methods if you're not familiar with them).
A post method would be something like:
def do_POST(self):
post_body = self.rfile.readall()
Note that here you can use the method rfile.readall(). Here you have a nice gist with some examples!

SimpleHTTPServer Custom Headers

By default SimpleHTTPServer sends it's own headers.
I've been trying to figure out how to send my own headers and found this solution. I tried adapting it to my (very) simple proxy:
import SocketServer
import SimpleHTTPServer
import urllib
class Proxy(SimpleHTTPServer.SimpleHTTPRequestHandler):
headers = ['Date: Wed, 29 Oct 2014 15:54:43 GMT', 'Server: Apache', 'Accept-Ranges: bytes', 'X-Mod-Pagespeed: 1.6.29.7-3566', 'Vary: Accept-Encoding', 'Cache-Control: max-age=0, no-cache', 'Content-Length: 204', 'Connection: close', 'Content-Type: text/html']
def end_headers(self):
print "Setting custom headers"
self.custom_headers()
SimpleHTTPServer.SimpleHTTPRequestHandler.end_headers(self)
def custom_headers(self):
for i in self.headers:
key, value = i.split(":", 1)
self.send_header(key, value)
def do_GET(self):
self.copyfile(urllib.urlopen(self.path), self.wfile)
httpd = SocketServer.ForkingTCPServer(('', PORT), Proxy)
httpd.serve_forever()
But end_headers() doesn't set the custom headers (confirmed on Wireshark).
Given a list of headers like the one in my little snippet, how I can overwrite SimpleHTTPServer's default headers and server my own?

I think you miss something in do_GET().
SimpleHTTPServer also calls
self.send_response(200)
See the following code or better the module SimpleHTTPServer
self.send_response(200)
self.send_header("Content-type", ctype)
fs = os.fstat(f.fileno())
self.send_header("Content-Length", str(fs[6]))
self.send_header("Last-Modified", self.date_time_string(fs.st_mtime))
self.end_headers()
I think you should override the send_head() method for what you want to do and read the source of SimpleHTTPServer.

Setting up an HTTP server that listens over a file-socket

How can I use HTTPServer (or some other class) to set up an HTTP server that listens to a filesystem socket instead of an actual network socket? By "filesystem socket" I mean sockets of the AF_UNIX type.

HTTPServer inherits from SocketServer.TCPServer, so I think it's fair to say that it isn't intended for that use-case, and even if you try to work around it, you may run into problems since you are kind of "abusing" it.
That being said, however, it would be possible per se to define a subclass of HTTPServer that creates and binds Unix sockets quite simply, as such:
class UnixHTTPServer(HTTPServer):
address_family = socket.AF_UNIX
def server_bind(self):
SocketServer.TCPServer.server_bind(self)
self.server_name = "foo"
self.server_port = 0
Then, just pass the path you want to bind to by the server_address argument to the constructor:
server = UnixHTTPServer("/tmp/http.socket", ...)
Again, though, I can't guarantee that it will actually work well. You may have to implement your own HTTP server instead.

I followed the example from #Dolda2000 above in Python 3.5 and ran into an issue with the HTTP handler falling over with an invalid client address. You don't have a client address with Unix sockets in the same way that you do with TCP, so the code below fakes it.
import socketserver
...
class UnixSocketHttpServer(socketserver.UnixStreamServer):
def get_request(self):
request, client_address = super(UnixSocketHttpServer, self).get_request()
return (request, ["local", 0])
...
server = UnixSocketHttpServer((sock_file), YourHttpHandler)
server.serve_forever()
With these changes, you can perform an HTTP request against the Unix socket with tools such as cURL.
curl --unix-socket /run/test.sock http:/test

Overview
In case it help anyone else, I have created a complete example (made for Python 3.8) based on Roger Lucas's example:
Server
import socketserver
from http.server import BaseHTTPRequestHandler
class myHandler(BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(200)
self.send_header('Content-type','text/html')
self.end_headers()
self.wfile.write(b"Hello world!")
return
class UnixSocketHttpServer(socketserver.UnixStreamServer):
def get_request(self):
request, client_address = super(UnixSocketHttpServer, self).get_request()
return (request, ["local", 0])
server = UnixSocketHttpServer(("/tmp/http.socket"), myHandler)
server.serve_forever()
This will listen on the unix socket and respond with "Hello World!" for all GET requests.
Client Request
You can send a request with:
curl --unix-socket /tmp/http.socket http://any_path/abc/123
Troubleshooting
If you run into this error:
OSError: [Errno 98] Address already in use
Then delete the socket file:
rm /tmp/http.socket

Why does my HTTP response using Python Sockets fail?

Code:
from socket import *
sP = 14000
servSock = socket(AF_INET,SOCK_STREAM)
servSock.bind(('',sP))
servSock.listen(1)
while 1:
connSock, addr = servSock.accept()
connSock.send('HTTP/1.0 200 OK\nContent-Type:text/html\nConnection:close\n<html>...</html>')
connSock.close()
When I go to the browser and type in localhost:14000, I get an error 101- ERR_CONNECTION_RESET The connection was reset? Not sure why! What am I doing wrong

Several bugs, some more severe than others ... as #IanWetherbee already noted, you need an empty line before the body. You also should send \r\n not just \n. You should use sendall to avoid short sends. Last, you need to close the connection once you're done sending.
Here's a slightly modified version of the above:
from socket import *
sP = 14000
servSock = socket(AF_INET,SOCK_STREAM)
servSock.bind(('',sP))
servSock.listen(1)
while 1:
connSock, addr = servSock.accept()
connSock.sendall('HTTP/1.0 200 OK\r\nContent-Type:text/html\r\nConnection:close\r\n\r\n<html><head>foo</head></html>\r\n')
connSock.close()

Running your code, I have similar errors and am unsure on their origins too. However, rather than rolling your own HTTP server, have you considered a built in one? Check out the sample below. This can also support POST as well (have to add the do_POST method).
Simple HTTP Server
from BaseHTTPServer import HTTPServer, BaseHTTPRequestHandler
class customHTTPServer(BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(200)
self.send_header('Content-type', 'text/html')
self.end_headers()
self.wfile.write('<HTML><body>Hello World!</body></HTML>')
return
def main():
try:
server = HTTPServer(('',14000),customHTTPServer)
print 'server started at port 14000'
server.serve_forever()
except KeyboardInterrupt:
server.socket.close()
if __name__=='__main__':
main()