I am experiencing some performance problems when creating a very simple Python HTTP server. The key issue is that performance is varying depending on which client I use to access it, where the server and all clients are being run on the local machine. For instance, a GET request issued from a Python script (urllib2.urlopen('http://localhost/').read()) takes just over a second to complete, which seems slow considering that the server is under no load. Running the GET request from Excel using MSXML2.ServerXMLHTTP also feels slow. However, requesting the data Google Chrome or from RCurl, the curl add-in for R, yields an essentially instantaneous response, which is what I would expect.
Adding further to my confusion is that I do not experience any performance problems for any client when I am on my computer at work (the performance problems are on my home computer). Both systems run Python 2.6, although the work computer runs Windows XP instead of 7.
Below is my very simple server example, which simply returns 'Hello world' for any get request.
from BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer
class MyHandler(BaseHTTPRequestHandler):
def do_GET(self):
print("Just received a GET request")
self.send_response(200)
self.send_header("Content-type", "text/html")
self.end_headers()
self.wfile.write('Hello world')
return
def log_request(self, code=None, size=None):
print('Request')
def log_message(self, format, *args):
print('Message')
if __name__ == "__main__":
try:
server = HTTPServer(('localhost', 80), MyHandler)
print('Started http server')
server.serve_forever()
except KeyboardInterrupt:
print('^C received, shutting down server')
server.socket.close()
Note that in MyHandler I override the log_request() and log_message() functions. The reason is that I read that a fully-qualified domain name lookup performed by one of these functions might be a reason for a slow server. Unfortunately setting them to just print a static message did not solve my problem.
Also, notice that I have put in a print() statement as the first line of the do_GET() routine in MyHandler. The slowness occurs prior to this message being printed, meaning that none of the stuff that comes after it is causing a delay.
The request handler issues a inverse name lookup in order to display the client name in the log. My Windows 7 issues a first DNS lookup that fails with no delay, followed by 2 successive NetBIOS name queries to the HTTP client, and each one run into a 2 sec timeout = 4 seconds delay !!
Have a look at https://bugs.python.org/issue6085
Another fix that worked for me is to override BaseHTTPRequestHandler.address_string() in my request handler with a version that does not perform the name lookup
def address_string(self):
host, port = self.client_address[:2]
#return socket.getfqdn(host)
return host
Philippe
This does not sound like a problem with the code. A nifty way of troubleshooting an HTTP server is to connect to it to telnet to it on port 80. Then you can type something like:
GET /index.html HTTP/1.1
host: www.blah.com
<enter> <enter>
and observe the server's response. See if you get a delay using this approach.
You may also want to turn off any firewalls to see if they are responsible for the slowdown.
Try replacing 127.0.0.1 for localhost. If that solves the problem, then that is a clue that the FQDN lookup may indeed be the possible cause.
Replacing localhost with 127.0.0.1 can solve the problem:)
Related
I am trying to start a Http server writing in python through my java code. Initially it seems to work fine, but after 8 requests it closes again. What am I doing wrong?
the server is started like this:
ProcessBuilder builder = new ProcessBuilder("python", "src\\main\\java\\python\\HttpHandler.py");
Process p = builder.start();
and within the python code the server looks like this:
class HttpHandler(BaseHTTPRequestHandler):
def do_GET(self):
#handle get request
def do_POST(self):
#handle post request
with HTTPServer(('', 8000), HttpHandler) as server:
server.serve_forever()
From the comments, the problem was caused by the server process writing to its standard output or standard error streams, and the Java code not reading from these streams. Eventually a buffer filled up and the server process blocked because it couldn't write to the buffer. Overriding the http log function fixed the problem.
I use https://gist.github.com/bradmontgomery/2219997 python code to setup a http server.
For my use I just add few lines to the 'do_POST' method:
def _set_headers(self):
self.send_response(200)
self.send_header('Content-type', 'text/html')
self.end_headers()
def do_POST(self):
self._set_headers()
self.wfile.write("POST!")
content_length = int(self.headers['Content-Length'])
print(content_length )
post_data = self.rfile.read(content_length)
print(post_data)
I want to send a file through Curl :
curl -F "file=#file.txt" "http://myServer.noip.me:22222/" --trace-ascii debugdump.txt
Client side : Curl response is:
curl: (52) Empty reply from server
Server side : server prints content_length value and then completely hangs at line "self.rfile.read(content_length)". It does not print 'print(post_data)'.
Firewall has been disabled on both side.
Last lines from debugdump.txt (Client side):
== Info: Empty reply from server
== Info: Connection #0 to host myServer.noip.me left intact
What did I miss?
Empty reply from server means that the server closed the connection without responding anything which is a HTTP protocol violation.
This should never happen with a proper server so it signals something is seriously wrong in the server side.
Sometimes this happens because the server was naively implemented and crashes or misbehaves on an unexpected request coming from a client. The best chance to make this work then, is to try to alter the request in ways to make it more similar to the way the badly written server software might expect it to be.
This is a matter of guessing. Something is wrong in that server of yours.
With Python's http.server package it's your responsibility to explicitly send an HTTP status code as part of your do_POST() override.
In your example code you first reply to the client (i.e. via send_header(), end_headers() and wfile before reading the whole POST request from the client!
See also Python's wfile documentation:
Contains the output stream for writing a response back to the client. Proper adherence to the HTTP protocol must be used when writing to this stream in order to achieve successful interoperation with HTTP clients.
So this looks racy and you probably just need to make sure that you read the complete POST request before you start replying.
IOW, curl just complains that it didn't receive any HTTP status code from your server before the connection was closed. Apparently the connection isn't properly shutdown on both sides such that your server blocks on the read side.
That curl error
curl: (52) Empty reply from server
would also show up if you simply forget to send a status code by omitting
self.send_response(200)
self.end_headers()
at the end of your method - or if your method prematurely exits (e.g. because it raises an exception).
However, the latter issues should show up in stderr ouput of your server.
I got a Twisted Game server and I want to make a "ping" command server-side. (The client sends commands to server and the server do thnigs, and answer).
But I can't thing any way to get "Ping Time" of a connection between the server and the client. Is there a way to get it, for example with
self.transport
or other. But I can't find.
Any ideas please?
Thanks for help.
"ping time" is not an inherent property of a connection, but rather, the amount of time it takes for the client to send a do-nothing request to the server and have the server send an answer.
If you were using, for example, AMP, you could do something like this:
def pingTime(self):
then = reactor.seconds()
def pung(ignored):
now = reactor.seconds()
return now - then
return self.callRemote(Ping).addCallback(pung)
I have these two Python scripts I'm using to attempt to work out how to send and receive POST requests in Python:
The Client:
import httplib
conn = httplib.HTTPConnection("localhost:8000")
conn.request("POST", "/testurl")
conn.send("clientdata")
response = conn.getresponse()
conn.close()
print(response.read())
The Server:
from BaseHTTPServer import BaseHTTPRequestHandler,HTTPServer
ADDR = "localhost"
PORT = 8000
class RequestHandler(BaseHTTPRequestHandler):
def do_POST(self):
print(self.path)
print(self.rfile.read())
self.send_response(200, "OK")
self.end_headers()
self.wfile.write("serverdata")
httpd = HTTPServer((ADDR, PORT), RequestHandler)
httpd.serve_forever()
The problem is that the server hangs on self.rfile.read() until conn.close() has been called on the client but if conn.close() is called on the client the client cannot receive a response from the server. This creates a situation where one can either get a response from the server or read the POST data but never both. I assume there is something I'm missing here that will fix this problem.
Additional information:
conn.getresponse() causes the client to hang until the response is received from the server. The response doesn't appear to be received until the function on the server has finished execution.
There are a couple of issues with your original example. The first is that if you use the request method, you should include the message body you want to send in that call, rather than calling send separately. The documentation notes send() can be used as an alternative to request:
As an alternative to using the request() method described above, you
can also send your request step by step, by using the four functions
below.
You just want conn.request("POST", "/testurl", "clientdata").
The second issue is the way you're trying to read what's sent to the server. self.rfile.read() attempts to read the entire input stream coming from the client, which means it will block until the stream is closed. The stream won't be closed until connection is closed. What you want to do is read exactly how many bytes were sent from the client, and that's it. How do you know how many bytes that is? The headers, of course:
length = int(self.headers['Content-length'])
print(self.rfile.read(length))
I do highly recommend the python-requests library if you're going to do more than very basic tests. I also recommend using a better HTTP framework/server than BaseHTTPServer for more than very basic tests (flask, bottle, tornado, etc.).
Long time answered but came up during a search so I bring another piece of answer. To prevent the server to keep the stream open (resulting in the response never being sent), you should use self.rfile.read1() instead of self.rfile.read()
I'm using python2.6 with HTTPServer and the ThreadingMixIn, which will handle each request in a separate thread. I'm also using HTTP1.1 persistent connections ('Connection: keep-alive'), so neither the server or client will close a connection after a request.
Here's roughly what the request handler looks like
request, client_address = sock.accept()
rfile = request.makefile('rb', rbufsize)
wfile = request.makefile('wb', wbufsize)
global server_stopping
while not server_stopping:
request_line = rfile.readline() # 'GET / HTTP/1.1'
# etc - parse the full request, write to wfile with server response, etc
wfile.close()
rfile.close()
request.close()
The problem is that if I stop the server, there will still be a few threads waiting on rfile.readline().
I would put a select([rfile, closefile], [], []) above the readline() and write to closefile when I want to shutdown the server, but I don't think it would work on windows because select only works with sockets.
My other idea is to keep track of all the running requests and rfile.close() but I get Broken pipe errors.
Ideas?
You're almost there—the correct approach is to call rfile.close() and to catch the broken pipe errors and exit your loop when that happens.
If you set daemon_threads to true in your HTTPServer subclass, the activity of the threads will not prevent the server from exiting.
class ThreadedHTTPServer(ThreadingMixIn, HTTPServer):
daemon_threads = True
You could work around the Windows problem by making closefile a socket, too -- after all, since it's presumably something that's opened by your main thread, it's up to you to decide whether to open it as a socket or a file;-).