I have a server, which takes few minutes to process a specific request and then responds to it.
The client has to keep waiting for the response without knowing when it will complete.
Is there a way to let the client know about the processing status? (say 50% completed, 80% completed), without the client having to poll for the status.
Without using any of the newer techniques (websockets, webpush/http2, ...), I've previously used a simplified Pushlet or Long polling solution for HTTP 1.1 and various javascript or own client implementation. If my solution doesn't fit in your use case, you can always google those two names for further possible ways.
Client
sends a request, reads 17 bytes (Inital http response) and then reads 2 bytes at a time getting processing status.
Server
sends a valid HTTP response and during request progress sends 2 bytes of percentage completed, until last 2 bytes are "ok" and closes connection.
UPDATED: Example uwsgi server.py
from time import sleep
def application(env, start_response):
start_response('200 OK', [])
def working():
yield b'00'
sleep(1)
yield b'36'
sleep(1)
yield b'ok'
return working()
UPDATED: Example requests client.py
import requests
response = requests.get('http://localhost:8080/', stream=True)
for r in response.iter_content(chunk_size=2):
print(r)
Example server (only use for testing :)
import socket
from time import sleep
HOST, PORT = '', 8888
listen_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
listen_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
listen_socket.bind((HOST, PORT))
listen_socket.listen(1)
while True:
client_connection, client_address = listen_socket.accept()
request = client_connection.recv(1024)
client_connection.send('HTTP/1.1 200 OK\n\n')
client_connection.send('00') # 0%
sleep(2) # Your work is done here
client_connection.send('36') # 36%
sleep(2) # Your work is done here
client_connection.sendall('ok') # done
client_connection.close()
If the last 2 bytes aren't "ok", handle error someway else. This isn't beautiful HTTP status code compliance but more of a workaround that did work for me many years ago.
telnet client example
$ telnet localhost 8888
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET / HTTP/1.1
HTTP/1.1 200 OK
0036okConnection closed by foreign host.
This answer probably won’t help in your particular case, but it might help in other cases.
The HTTP protocol supports informational (1xx) responses:
indicates an interim
response for communicating connection status or request progress
prior to completing the requested action and sending a final
response
There is even a status code precisely for your use case, 102 (Processing):
interim response used to
inform the client that the server has accepted the complete request,
but has not yet completed it
Status code 102 was removed from further editions of that standard due to lack of implementations, but it is still registered and could be used.
So, it might look like this (HTTP/2 has an equivalent binary form):
HTTP/1.1 102 Processing
Progress: 50%
HTTP/1.1 102 Processing
Progress: 80%
HTTP/1.1 200 OK
Date: Sat, 05 Aug 2017 11:53:14 GMT
Content-Type: text/plain
All done!
Unfortunately, this is not widely supported. In particular, WSGI does not provide a way to send arbitrary 1xx responses. Clients support 1xx responses in the sense that they are required to parse and tolerate them, but they usually don’t give programmatic access to them: in this example, the Progress header would not be available to the client application.
However, 1xx responses may still be useful (if the server can send them) because they have the effect of resetting the client’s socket read timeout, which is one of the main problems with slow responses.
Use Chunked Transfer Encoding, which is a standard technique to transmit streams of unknown length.
See: Wikipedia - Chunked Transfer Encoding
Here a python server implementation available as a gist on GitHub:
https://gist.github.com/josiahcarlson/3250376
It sends content using chunked transfer encoding using standard library modules
In the client if chunked transfer encoding has been notified by the server, you'd only need to:
import requests
response = requests.get('http://server.fqdn:port/', stream=True)
for r in response.iter_content(chunk_size=None):
print(r)
chunk_size=None, because the chunks are dynamic and will be determined by the information in the simple conventions of the chunked transfer semantics.
See: http://docs.python-requests.org/en/master/user/advanced/#chunk-encoded-requests
When you see for example 100 in the content of response r, you know that the next chunk will be the actual content after processing the 100.
Related
I'm trying to get 101 Switching Protocol response by upgrading the connnection to HTTP/2 via Socket. I've tried big web that support HTTP/2 such as google.com and twitter.com, None of them that give a proper response, here's the approach:
import socket
import ssl
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
context = ssl.create_default_context()
context.set_ciphers(cipher)
sock.connect(('twitter.com', 443))
sock = context.wrap_socket(sock, server_hostname='twitter.com')
sock.sendall(bytes('GET h2://twitter.com/ HTTP/1.1\r\nHost: twitter.com\r\nConnection: Upgrade, HTTP2-Settings\r\nUpgrade: h2\r\nHTTP2-Settings: \r\nAlt-Svc: h2=":443"\r\n\r\n',encoding='utf-8'))
sock.settimeout(5)
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
line = str(sock.recv(10000))
print(line)
Upgrading the HTTP/1 connection to HTTP/2; giving 400 bad request, not what I'm expecting 101.
sock.connect(('www.google.com', 443))
sock = context.wrap_socket(sock, server_hostname='www.google.com')
sock.sendall(bytes('GET / HTTP/1.1\r\nHost: www.google.com\r\nConnection: Upgrade, HTTP2-Settings\r\nUpgrade: h2\r\nHTTP2-Settings: \r\n\r\n',encoding='utf-8'))
I've also tried to set ALPN protocol manually but google.com and twitter.com give different response:
context.set_alpn_protocols(['h2'])
for google.com there's frame response:
b'\x00\x00\x12\x04\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00d\x00\x04\x00\x10\x00\x00\x00\x06\x00\x01\x00\x00\x00\x00\x04\x08\x00\x00\x00\x00\x00\x00\x0f\x00\x01'
but twitter.com giving empty response frame:
b''
The reason for this is, i had to perform checks based on wordlist that contains list of website. How can i properly check if website supports for HTTP/2 in Python?
Attachment:
Twitter
yes, you do need to set the context.set_alpn_protocols(['h2'])
and you can check if you successfully negotiated with sock.selected_alpn_protocol().
Unfortunately, if you want to actually test an http call, you need to encode the header (get, host, etc.). rfc7540 section 4 and section 6.2 detail the requirement. a python library like h2 can help with the encoding as well as properly encode and decode the other sadly necessary http2 chatter.
The whole connection upgrade thing is not used by web browsers, so it is not unexpected that websites don't support it. Once you have an alpn h2 negotiated tls stream, a properly encoded
GET / HTTP/2
Host: twitter.com
should be all you need. That is the reason twitter just closed the socket. I suspect the google frame sent back (if you used hpack to decode) would say something like "I don't understand".
I use https://gist.github.com/bradmontgomery/2219997 python code to setup a http server.
For my use I just add few lines to the 'do_POST' method:
def _set_headers(self):
self.send_response(200)
self.send_header('Content-type', 'text/html')
self.end_headers()
def do_POST(self):
self._set_headers()
self.wfile.write("POST!")
content_length = int(self.headers['Content-Length'])
print(content_length )
post_data = self.rfile.read(content_length)
print(post_data)
I want to send a file through Curl :
curl -F "file=#file.txt" "http://myServer.noip.me:22222/" --trace-ascii debugdump.txt
Client side : Curl response is:
curl: (52) Empty reply from server
Server side : server prints content_length value and then completely hangs at line "self.rfile.read(content_length)". It does not print 'print(post_data)'.
Firewall has been disabled on both side.
Last lines from debugdump.txt (Client side):
== Info: Empty reply from server
== Info: Connection #0 to host myServer.noip.me left intact
What did I miss?
Empty reply from server means that the server closed the connection without responding anything which is a HTTP protocol violation.
This should never happen with a proper server so it signals something is seriously wrong in the server side.
Sometimes this happens because the server was naively implemented and crashes or misbehaves on an unexpected request coming from a client. The best chance to make this work then, is to try to alter the request in ways to make it more similar to the way the badly written server software might expect it to be.
This is a matter of guessing. Something is wrong in that server of yours.
With Python's http.server package it's your responsibility to explicitly send an HTTP status code as part of your do_POST() override.
In your example code you first reply to the client (i.e. via send_header(), end_headers() and wfile before reading the whole POST request from the client!
See also Python's wfile documentation:
Contains the output stream for writing a response back to the client. Proper adherence to the HTTP protocol must be used when writing to this stream in order to achieve successful interoperation with HTTP clients.
So this looks racy and you probably just need to make sure that you read the complete POST request before you start replying.
IOW, curl just complains that it didn't receive any HTTP status code from your server before the connection was closed. Apparently the connection isn't properly shutdown on both sides such that your server blocks on the read side.
That curl error
curl: (52) Empty reply from server
would also show up if you simply forget to send a status code by omitting
self.send_response(200)
self.end_headers()
at the end of your method - or if your method prematurely exits (e.g. because it raises an exception).
However, the latter issues should show up in stderr ouput of your server.
I need to write simple http caching proxy with python socket class (not using SimpleHTTPServer class). The simple scheme is:
-open socket
-get http message
-check if we have cached resource
-either provide cached resource or re-request it from remote server
-close socket
When getting a message I use synchronous recv method:
38 message = ""
39 BUF_SIZE = 4096
40 while 1:
41 chunk = connection.recv(BUF_SIZE)
42 if None == chunk or 0 == len(chunk):
43 break
44 message += chunk
45 print "rec %i" % (len(message))
And when I connect to my proxy with a browser, the browser will hold the socket open until it either gets response or I manually cancel the wait, which is perfectly by according to how HTTP is intended to work. So, my control is stuck at recv.
-I cannot just read say 1000bytes, coz I don't know how large the actual message is? There doesn't seem to be any message end mark in http?
-the program cannot continue while having open socket, using synchronous read;
-I could use either threads or some kind of async api, to solve control blocking, but this doesn't help with the problem to understand when to stop reading request and start handling it.
I have implemented a simple HTTP server and a client. The latter issues a PUT request using the requests library, sending some arbitrary JSON and then exits.
When I start the server, and then run the client, both the server and the client block. The server however appears to not have gone through the entire handler function yet.
This is what I get on server side:
$ python3 server.py
PUT / HTTP/1.1
That is, after printing the request line, the content JSON string is not printed. At this point both client and server block for some reason.
Interestingly, when I trigger a KeyboardInterrupt to the client, the server proceeds:
$ python3 server.py
PUT / HTTP/1.1
b'{"content": "Hello World"}'
127.0.0.1 - - [25/Feb/2016 11:52:54] "PUT / HTTP/1.1" 200 -
My questions:
Why is it necessary to kill the client to let the server proceed?
Am I using any of these components the wrong way?
How can I make client and server to operate (nearly) instantaneously?
This is the code of the HTTP server. It only handles PUT requests. It prints the request line and the content data and responds using the success code to the client:
import http.server
class PrintPUTRequestHandler(http.server.BaseHTTPRequestHandler):
def do_PUT(self):
print(self.requestline)
print(self.rfile.read())
self.send_response(200)
self.end_headers()
server_address = ('', 8000)
httpd = http.server.HTTPServer(server_address, PrintHTTPRequestHandler)
httpd.serve_forever()
This is the HTTP client. It is intended to connect to the server, write the request and return as soon as possible (but it doesn't):
import requests
server_address = "http://127.1:8000"
data = '{"content": "Hello World"}'
requests.put(server_address, data, headers={"Content-type": "application/json"})
This is how I run it after the server has started (no output observable):
python client.py
The server blocks both itself and the client on this line:
print(self.rfile.read())
That happens because you didn't specify the amount of data to be read so the server reads the input stream until it is closed. And in your case the input stream is closed once you kill the client.
Remember that the server doesn't know a priori when the streaming of data ends because you may want to send data chunk by chunk (for example when you send big files).
The size of request should be passed in Content-Length header so this is what you should do:
length = int(self.headers['Content-Length'])
print(self.rfile.read(length))
That's assuming that the length is small enough to fit in your memory (and in your case it is).
I have these two Python scripts I'm using to attempt to work out how to send and receive POST requests in Python:
The Client:
import httplib
conn = httplib.HTTPConnection("localhost:8000")
conn.request("POST", "/testurl")
conn.send("clientdata")
response = conn.getresponse()
conn.close()
print(response.read())
The Server:
from BaseHTTPServer import BaseHTTPRequestHandler,HTTPServer
ADDR = "localhost"
PORT = 8000
class RequestHandler(BaseHTTPRequestHandler):
def do_POST(self):
print(self.path)
print(self.rfile.read())
self.send_response(200, "OK")
self.end_headers()
self.wfile.write("serverdata")
httpd = HTTPServer((ADDR, PORT), RequestHandler)
httpd.serve_forever()
The problem is that the server hangs on self.rfile.read() until conn.close() has been called on the client but if conn.close() is called on the client the client cannot receive a response from the server. This creates a situation where one can either get a response from the server or read the POST data but never both. I assume there is something I'm missing here that will fix this problem.
Additional information:
conn.getresponse() causes the client to hang until the response is received from the server. The response doesn't appear to be received until the function on the server has finished execution.
There are a couple of issues with your original example. The first is that if you use the request method, you should include the message body you want to send in that call, rather than calling send separately. The documentation notes send() can be used as an alternative to request:
As an alternative to using the request() method described above, you
can also send your request step by step, by using the four functions
below.
You just want conn.request("POST", "/testurl", "clientdata").
The second issue is the way you're trying to read what's sent to the server. self.rfile.read() attempts to read the entire input stream coming from the client, which means it will block until the stream is closed. The stream won't be closed until connection is closed. What you want to do is read exactly how many bytes were sent from the client, and that's it. How do you know how many bytes that is? The headers, of course:
length = int(self.headers['Content-length'])
print(self.rfile.read(length))
I do highly recommend the python-requests library if you're going to do more than very basic tests. I also recommend using a better HTTP framework/server than BaseHTTPServer for more than very basic tests (flask, bottle, tornado, etc.).
Long time answered but came up during a search so I bring another piece of answer. To prevent the server to keep the stream open (resulting in the response never being sent), you should use self.rfile.read1() instead of self.rfile.read()