We Keep Coding

Request timed out: timeout('timed out') in Python's HTTPServer

I am trying to create a simple HTTP server that uses the Python HTTPServer which inherits BaseHTTPServer. [https://github.com/python/cpython/blob/main/Lib/http/server.py][1]
There are numerous examples of this approach online and I don't believe I am doing anything unusual.
I am simply importing the class via:
"from http.server import HTTPServer, BaseHTTPRequestHandler"
in my code.
My code overrides the do_GET() method to parse the path variable to determine what page to show.
However, if I start this server and connect to it locally (ex: http://127.0.0.1:50000) the first page loads fine. If I navigate to another page (via my first page links) that too works fine, however, on occasion (and this is somewhat sporadic), there is a delay and the server log shows a Request timed out: timeout('timed out') error. I have tracked this down to the handle_one_request method in the BaseHTTPServer class:
def handle_one_request(self):
"""Handle a single HTTP request.
You normally don't need to override this method; see the class
__doc__ string for information on how to handle specific HTTP
commands such as GET and POST.
"""
try:
self.raw_requestline = self.rfile.readline(65537)
if len(self.raw_requestline) > 65536:
self.requestline = ''
self.request_version = ''
self.command = ''
self.send_error(HTTPStatus.REQUEST_URI_TOO_LONG)
return
if not self.raw_requestline:
self.close_connection = True
return
if not self.parse_request():
# An error code has been sent, just exit
return
mname = 'do_' + self.command ## the name of the method is created
if not hasattr(self, mname): ## checking that we have that method defined
self.send_error(
HTTPStatus.NOT_IMPLEMENTED,
"Unsupported method (%r)" % self.command)
return
method = getattr(self, mname) ## getting that method
method() ## finally calling it
self.wfile.flush() #actually send the response if not already done.
except socket.timeout as e:
# a read or a write timed out. Discard this connection
self.log_error("Request timed out: %r", e)
self.close_connection = True
return
You can see where the exception is thrown in the "except socket.timeout as e:" clause.
I have tried overriding this method by including it in my code but it is not clear what is causing the error so I run into dead ends. I've tried creating very basic HTML pages to see if there was something in the page itself, but even "blank" pages cause the same sporadic issue.
What's odd is that sometimes a page loads instantly, and almost randomly, it will then timeout. Sometimes the same page, sometimes a different page.
I've played with the http.timeout setting, but it makes no difference. I suspect it's some underlying socket issue, but am unable to diagnose it further.
This is on a Mac running Big Sur 11.3.1, with Python version 3.9.4.
Any ideas on what might be causing this timeout, and in particular any suggestions on a resolution. Any pointers would be appreciated.

After further investigation, this particular appears to be an issue with Safari. Running the exact same code and using Firefox does not show the same issue.

App Engine giving errors about outdated files

Just to be extremely clear, here's some initial information about my setup:
Platform: Google App Engine
Environment Type: Standard
Runtime: Python 3
Python Framework: Flask
Web Server: Gunicorn, 2 workers
It's a flask app running on GAE standard python3 runtime.
Relevant File Structure:
- api.py
- services/
---- HighLow.py
As you can hopefully see from the file structure, the main flask app file is api.py.
api.py then imports services/HighLow.py
Relevant Code From HighLow.py:
def get_today_for_user(self, uid):
#Connect to MySQL
conn = pymysql.connect(self.host, self.username, self.password, self.database, cursorclass=pymysql.cursors.DictCursor)
cursor = conn.cursor()
uid = bleach.clean(uid)
cursor.execute("SELECT * FROM highlows WHERE uid='{}' AND DATE(_timestamp) = CURDATE();")
highlow = cursor.fetchone()
conn.commit()
conn.close()
if highlow == None:
return {
"high":"",
"low":"",
"total_likes": 0,
"high_image": "",
"low_image": ""
}
return highlow
If you want to look at the full code for better context, you can view it on our GitHub repo: https://github.com/highlowapp/highlowbackend
The Problem:
This setup has been working fine for a while now, and I was perfectly happy with it. However, I recently started getting this error from GCP's Error Reporting:
File "/srv/services/HighLow.py", line 380: 'low_image': ""
at <module> (/srv/api.py:8)
at <module> (/srv/wsgi.py:1)
at import_app (/env/lib/python3.7/site-packages/gunicorn/util.py:350)
at load_wsgiapp (/env/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py:41)
at load (/env/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py:52)
at wsgi (/env/lib/python3.7/site-packages/gunicorn/app/base.py:67)
at load_wsgi (/env/lib/python3.7/site-packages/gunicorn/workers/base.py:138)
at init_process (/env/lib/python3.7/site-packages/gunicorn/workers/base.py:129)
at spawn_worker (/env/lib/python3.7/site-packages/gunicorn/arbiter.py:583)
This error confuses me for a couple of reasons:
It doesn't give any kind of exception, it just "announces" that line 380 says 'low_image': "".
When I update the code to change that line, it gives me the same error. In other words, it says that line 380 is: 'low_image': "", when in reality (you can verify this on the GitHub repository), it's: "low_image": "" (with double quotes). I've also tried simply deleting the line, but the error continues to appear.
I've tried deleting app engine's storage bucket and re-running gcloud app deploy, but nothing has changed that error message at all.
My Question
Why am I getting this error message, and how do I fix it?
EDIT:
As I said above, I already tried deleting the staging bucket, and that didn't work.
Also, I used the "source" tool with Stackdriver to verify that the code has been updated.
Finally, I tried creating a new App Engine app in a separate project, and the code worked...except that I had an identical error in my code (maybe I'm just forgetting commas now?), which made the same error occur in a different file.
So I updated the file, redeployed, and voila! my app works!
However, I need it to work in the current project, not the test project. I need a way to clear everything that GAE knows about anything, and start all over. Is there a way to do this? Or does someone know why I'm having problems in the first place?

Ok, I finally figured out a workaround, and thought I'd share for anyone else who encounters this issue.
You know how tech support people always tell you to try turning it off and then back on? Let me tell you, it works.
I went to App Engine -> Settings -> Disable Application, and let it disable, and then re-enabled it with App Engine -> Settings -> Enable Application.
I guess it just needed to "reboot".

How can I trigger an IncompleteRead (on purpose) in a Python web application?

I've got some Python code that makes requests using the requests library and occasionally experiences an IncompleteRead error. I'm trying to update this code to handle this error more gracefully and would like to test that it works, so I'm wondering how to actually trigger the conditions under which IncompleteRead is raised.
I realize I can do some mocking in a unit test; I'd just like to actually reproduce the circumstances (if I can) under which this error was previously occurring and ensure my code is able to deal with it properly.

Adding a second answer, more to the point this time. I took a dive into some source code, and found information that may help
The IncompleteRead exception bubbles up from httplib, part of the python standard library. Most likely, it comes from this function:
def _safe_read(self, amt):
"""
Read the number of bytes requested, compensating for partial reads.
Normally, we have a blocking socket, but a read() can be interrupted
by a signal (resulting in a partial read).
Note that we cannot distinguish between EOF and an interrupt when zero
bytes have been read. IncompleteRead() will be raised in this
situation.
This function should be used when <amt> bytes "should" be present for
reading. If the bytes are truly not available (due to EOF), then the
IncompleteRead exception can be used to detect the problem.
"""
So, either the socket was closed before the HTTP response was consumed, or the reader tried to get too many bytes out of it. Judging by search results (so take this with a grain of salt), there is no other arcane situation that can make this happen.
The first scenario can be debugged with strace. If I'm reading this correctly, the 2nd scenario can be caused by the requests module, if:
A Content-Length header is present that exceeds the actual amount of data sent by the server.
A chunked response is incorrectly assembled (has an erroneous length byte before one of the chunks), or a regular response is being interpreted as chunked.
This function raises the Exception:
def _update_chunk_length(self):
# First, we'll figure out length of a chunk and then
# we'll try to read it from socket.
if self.chunk_left is not None:
return
line = self._fp.fp.readline()
line = line.split(b';', 1)[0]
try:
self.chunk_left = int(line, 16)
except ValueError:
# Invalid chunked protocol response, abort.
self.close()
raise httplib.IncompleteRead(line)
Try checking the Content-Length header of your buffered responses, or the chunk format of your chunked responses.
To produce the error, try:
Forcing an invalid Content-Length
Using the chunked response protocol, with a too-large length byte at the beginning of a chunk
Closing the socket mid-response

By looking at the places where raise IncompleteRead appears at https://github.com/python/cpython/blob/v3.8.0/Lib/http/client.py, I think the standard library's http.client module (named httplib back in Python 2) raises this exception in only the following two circumstances:
When a response's body is shorter than claimed by the response's Content-Length header, or
When a chunked response claims that the next chunk is of length n, but there are fewer than n bytes remaining in the response body.
If you install Flask (pip install Flask), you can paste this into a file to create a test server you can run with endpoints that artificially create both of these circumstances:
from flask import Flask, make_response
app = Flask(__name__)
#app.route('/test')
def send_incomplete_response():
response = make_response('fourteen chars')
response.headers['Content-Length'] = '10000'
return response
#app.route('/test_chunked')
def send_chunked_response_with_wrong_sizes():
# Example response based on
# https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding
# but with the stated size of the second chunk increased to 900
resp_text = """7\r\nMozilla\r\n900\r\nDeveloper\r\n7\r\nNetwork\r\n0\r\n\r\n"""
response = make_response(resp_text)
response.headers['Transfer-Encoding'] = 'chunked'
return response
app.run()
and then test them with http.client:
>>> import http.client
>>>
>>> conn = http.client.HTTPConnection('localhost', 5000)
>>> conn.request('GET', '/test')
>>> response = conn.getresponse()
>>> response.read()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.8/http/client.py", line 467, in read
s = self._safe_read(self.length)
File "/usr/lib/python3.8/http/client.py", line 610, in _safe_read
raise IncompleteRead(data, amt-len(data))
http.client.IncompleteRead: IncompleteRead(14 bytes read, 9986 more expected)
>>>
>>> conn = http.client.HTTPConnection('localhost', 5000)
>>> conn.request('GET', '/test_chunked')
>>> response = conn.getresponse()
>>> response.read()
Traceback (most recent call last):
File "/usr/lib/python3.8/http/client.py", line 571, in _readall_chunked
value.append(self._safe_read(chunk_left))
File "/usr/lib/python3.8/http/client.py", line 610, in _safe_read
raise IncompleteRead(data, amt-len(data))
http.client.IncompleteRead: IncompleteRead(28 bytes read, 2276 more expected)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.8/http/client.py", line 461, in read
return self._readall_chunked()
File "/usr/lib/python3.8/http/client.py", line 575, in _readall_chunked
raise IncompleteRead(b''.join(value))
http.client.IncompleteRead: IncompleteRead(7 bytes read)
In real life, the most likely reason this might happen sporadically is if a connection was closed early by the server. For example, you can also try running this Flask server, which sends a response body very slowly, with a total of 20 seconds of sleeping:
from flask import Flask, make_response, Response
from time import sleep
app = Flask(__name__)
#app.route('/test_generator')
def send_response_with_delays():
def generate():
yield 'foo'
sleep(10)
yield 'bar'
sleep(10)
yield 'baz'
response = Response(generate())
response.headers['Content-Length'] = '9'
return response
app.run()
If you run that server in a terminal, then initiate a request to it and start reading the response like this...
>>> import http.client
>>> conn = http.client.HTTPConnection('localhost', 5000)
>>> conn.request('GET', '/test_generator')
>>> response = conn.getresponse()
>>> response.read()
... and then flick back to the terminal running your server and kill it (e.g. with CTRL-C, on Unix), then you'll see your .read() call error out with a familiar message:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.8/http/client.py", line 467, in read
s = self._safe_read(self.length)
File "/usr/lib/python3.8/http/client.py", line 610, in _safe_read
raise IncompleteRead(data, amt-len(data))
http.client.IncompleteRead: IncompleteRead(6 bytes read, 3 more expected)
Other, less probable causes include your server systematically generating an incorrect Content-Length header (maybe due to some broken handling of Unicode), or your Content-Length header (or the lengths included in a chunked message) being corrupted in transit.
Okay, that covers the standard library. What about Requests? Requests by default defers its work to urllib3 which in turn defers to http.client, so you might expect the exception from http.client to simply bubble up when using Requests. However, life is more complicated than that, for two reasons:
Both urllib3 and requests catch exceptions in the layer beneath them and raise their own versions. For instance, there are urllib3.exceptions.IncompleteRead and requests.exceptions.ChunkedEncodingError.
The current handling of Content-Length checking across all three of these modules is horribly broken, and has been for years. I've done my best to explain it in detail at https://github.com/psf/requests/issues/4956#issuecomment-573325001 if you're interested, but the short version is that http.client won't check Content-Length if you call .read(123) instead of just .read(), that urllib3 may or may not check depending upon various complicated details of how you call it, and that Requests - as a consequence of the previous two issues - currently doesn't check it at all, ever. However, this hasn't always been the case; there have been some attempts to fix it made and unmade, so perhaps at some point in the past - like when this question was asked in 2016 - the state of play was a bit different. Oh, and for extra confusion, while urllib3 has its own version it still sometimes lets the standard library's IncompleteRead exception bubble up, just to mess with you.
Hopefully, point 2 will get fixed in time - I'm having a go right now at nudging it in that direction. Point 1 will remain a complication, but the conditions that trigger these exceptions - whether the underlying http.client.IncompleteRead or the urllib3 or requests alternatives - should remain as I describe at the start of this answer.

When testing code that relies on external behavior (such as server responses, system sensors, etc) the usual approach is to fake the external factors instead of working to produce them.
Create a test version of the function or class you're using to make HTTP requests. If you're using requests directly across your codebase, stop: direct coupling with libraries and external services is very hard to test.
You mention that you want to make sure your code can handle this exception, and you'd rather avoid mocking for this reason. Mocking is just as safe, as long as you're wrapping the modules you need to mock all across your codebase. If you can't mock to test, you're missing layers in your design (or asking too much of your testing suite).
So, for example:
class FooService(object):
def make_request(*args):
# use requests.py to perform HTTP requests
# NOBODY uses requests.py directly without passing through here
class MockFooService(FooService):
def make_request(*args):
raise IncompleteRead()
The 2nd class is a testing utility written solely for the purpose of testing this specific case. As your tests grow in coverage and completeness, you may need more sophisticated language (to avoid incessant subclassing and repetition), but it's usually good to start with the simplest code that will read easily and test the desired cases.

Exceptions in HTTP(WSGI)+JsonDocument Spyne Client

I've programmed a server-side service using Spyne. I want to use the Spyne client code, but I can't do it without having exceptions.
The server side code is something like (I've removed the imports and unified files):
class NotificationsRPC(ServiceBase):
#rpc(Uuid, DateTime, _returns=Integer)
def new_player(ctx, player_uuid, birthday):
# A lot of irrelevant code
return 0
#rpc(Uuid, _returns=Integer)
def deleted_player(ctx, player_uuid):
# A lot of irrelevant code
return 0
# Many other similar methods
radiante_app = Application(
[NotificationsRPC],
tns="radiante.rpc",
in_protocol=JsonDocument(validator="soft"),
out_protocol=JsonDocument()
)
wsgi_app = WsgiApplication(radiante_app)
server = make_server('127.0.0.1', 27182, wsgi_app)
server.serve_forever()
This code runs properly, I can make requests to it through CURL (the real code is implemented using uWSGI, but in this example I'm using the python embedded WSGI server).
The problem appears in the client-side code. It's something like (RadianteRPC is the same class as in server-side, but with pass in the methods body:
radiante_app = Application(
[RadianteRPC],
tns="radiante.rpc",
in_protocol=JsonDocument(validator="soft"),
out_protocol=JsonDocument()
)
rad_client = HttpClient("http://127.0.0.1:27182", radiante_app)
# In the real code, the parameters have more sense.
rad_client.service.new_player(uuid.UUID(), datetime.utcnow())
Then, when the code is executed I have the following error:
File "/vagrant/apps/radsync/signal_hooks/player.py", line 74, in _player_post_save
created
File "/home/vagrant/devenv/local/lib/python2.7/site-packages/spyne/client/http.py", line 64, in __call__
self.get_in_object(self.ctx)
File "/home/vagrant/devenv/local/lib/python2.7/site-packages/spyne/client/_base.py", line 144, in get_in_object
message=self.app.in_protocol.RESPONSE)
File "/home/vagrant/devenv/local/lib/python2.7/site-packages/spyne/protocol/dictdoc.py", line 278, in decompose_incoming_envelope
raise ValidationError("Need a dictionary with exactly one key "
ValidationError: Fault(Client.ValidationError: 'The value "\'Need a dictionary with exactly one key as method name.\'" could not be validated.')
It's worth noting that the client is implemented in Django U_u (not my decision), but I think it hasn't relation with the problem.
I've followed some indications from this question (adapting the example of ZeroMQ transport protocol to HTTP transport protocol): There is an example of Spyne client?
Thank you for your attention.

Python - How to check if Redis server is available

I'm developing a Python Service(Class) for accessing Redis Server. I want to know how to check if Redis Server is running or not. And also if somehow I'm not able to connect to it.
Here is a part of my code
import redis
rs = redis.Redis("localhost")
print rs
It prints the following
<redis.client.Redis object at 0x120ba50>
even if my Redis Server is not running.
As I found that my Python Code connects to the Server only when I do a set() or get() with my redis instance.
So I dont want other services using my class to get an Exception saying
redis.exceptions.ConnectionError: Error 111 connecting localhost:6379. Connection refused.
I want to return proper message/Error code. How can I do that??

If you want to test redis connection once at startup, use the ping() command.
from redis import Redis
redis_host = '127.0.0.1'
r = Redis(redis_host, socket_connect_timeout=1) # short timeout for the test
r.ping()
print('connected to redis "{}"'.format(redis_host))
The command ping() checks the connection and if invalid will raise an exception.
Note - the connection may still fail after you perform the test so this is not going to cover up later timeout exceptions.

The official way to check if redis server availability is ping ( http://redis.io/topics/quickstart ).
One solution is to subclass redis and do 2 things:
check for a connection at instantiation
write an exception handler in the case of no connectivity when making requests

As you said, the connection to the Redis Server is only established when you try to execute a command on the server. If you do not want to go head forward without checking that the server is available, you can just send a random query to the server and check the response. Something like :
try:
response = rs.client_list()
except redis.ConnectionError:
#your error handlig code here

There are already good solutions here, but here's my quick and dirty for django_redis which doesn't seem to include a ping function (though I'm using an older version of django and can't use the newest django_redis).
# assuming rs is your redis connection
def is_redis_available():
# ... get redis connection here, or pass it in. up to you.
try:
rs.get(None) # getting None returns None or throws an exception
except (redis.exceptions.ConnectionError,
redis.exceptions.BusyLoadingError):
return False
return True
This seems to work just fine. Note that if redis is restarting and still loading the .rdb file that holds the cache entries on disk, then it will throw the BusyLoadingError, though it's base class is ConnectionError so it's fine to just catch that.
You can also simply except on redis.exceptions.RedisError which is the base class of all redis exceptions.
Another option, depending on your needs, is to create get and set functions that catch the ConnectionError exceptions when setting/getting values. Then you can continue or wait or whatever you need to do (raise a new exception or just throw out a more useful error message).
This might not work well if you absolutely depend on setting/getting the cache values (for my purposes, if cache is offline for whatever we generally have to "keep going") in which case it might make sense to have the exceptions and let the program/script die and get the redis server/service back to a reachable state.

I have also come across a ConnectionRefusedError from the sockets library, when redis was not running, therefore I had to add that to the availability check.
r = redis.Redis(host='localhost',port=6379,db=0)
def is_redis_available(r):
try:
r.ping()
print("Successfully connected to redis")
except (redis.exceptions.ConnectionError, ConnectionRefusedError):
print("Redis connection error!")
return False
return True
if is_redis_available(r):
print("Yay!")

Redis server connection can be checked by executing ping command to the server.
>>> import redis
>>> r = redis.Redis(host="127.0.0.1", port="6379")
>>> r.ping()
True
using the ping method, we can handle reconnection etc. For knowing the reason for error in connecting, exception handling can be used as suggested in other answers.
try:
is_connected = r.ping()
except redis.ConnectionError:
# handle error

Use ping()
from redis import Redis
conn_pool = Redis(redis_host)
# Connection=Redis<ConnectionPool<Connection<host=localhost,port=6379,db=0>>>
try:
conn_pool.ping()
print('Successfully connected to redis')
except redis.exceptions.ConnectionError as r_con_error:
print('Redis connection error')
# handle exception