I have a python object whose methods are currently being exposed via XML-RPC using the standard xmlrpc.server.SimpleXMLRPCServer (with the ThreadingMixIn, but that should not be relevant).
The server is running on Win64 as are the clients. Some RPC methods return tables of information from a database to the client. I'm finding that even modest blocks of data are overwhelming the OS and I get this kind of error:
Traceback (most recent call last):
File "C:\Python32\lib\wsgiref\handlers.py", line 137, in run
self.result = application(self.environ, self.start_response)
File "U:\Me\src\application\my_xmlrpc_client.py", line 1510, in __call__
body = method(environ, start_response)
File "U:\Me\src\application\my_xmlrpc_client.py", line 305, in q_root
rows = proxy.return_table()
File "C:\Python32\lib\xmlrpc\client.py", line 1095, in __call__
return self.__send(self.__name, args)
File "C:\Python32\lib\xmlrpc\client.py", line 1423, in __request
verbose=self.__verbose
File "C:\Python32\lib\xmlrpc\client.py", line 1136, in request
return self.single_request(host, handler, request_body, verbose)
File "C:\Python32\lib\xmlrpc\client.py", line 1151, in single_request
return self.parse_response(resp)
File "C:\Python32\lib\xmlrpc\client.py", line 1323, in parse_response
return u.close()
File "C:\Python32\lib\xmlrpc\client.py", line 667, in close
raise Fault(**self._stack[0])
xmlrpc.client.Fault: :[Errno 12] Not enough space">
LIBRA.rsvfx.com - - [19/May/2011 15:58:09] "GET / HTTP/1.1" 500 59
Some research into the Errno 12 problem reveals that there's some issue with the underlying MS OS call and not with python itself:
http://bugs.python.org/issue11395
I'm not a very experienced XML-RPC developer; but is there some standard convention I should follow for delivering large payloads which would result in more, smaller writes (as opposed to fewer, large writes)?
And please remember I'm asking about buffer overruns; I don't want to have to debate why I'm using XML-RPC rather than rolling my own RESTful interface... I had to patch my WSGI application for this problem - sending small 1k blocks rather than larger blocks. I'm not sure how to patch the XML-RPC application.
-- edit --
As requested, here is a code sample that reproduces the problem:
import xmlrpc.server
class RPCApp :
def get_page(self):
return ["data" * 64 for i in range(0,1024)]
if __name__ == '__main__' : # important to use this block, for processes to spawn correctly
server = xmlrpc.server.SimpleXMLRPCServer(('127.0.0.1',8989), allow_none=True, logRequests=False)
server.register_instance(RPCApp())
server.serve_forever()
And the client code:
import xmlrpc.client
proxy = xmlrpc.client.ServerProxy('http://127.0.0.1:8989', allow_none=True)
print(proxy.get_page())
If you manipulate the page in the server to be small, then the code works. As it is, the exception is thrown.
-- edit --
Seems to be resolved in python 3.2.1rc1. Looks like we'll have to upgrade our installation....
Related
tl;dr: An app that had been working fine is suddenly throwing a "Bad file descriptor" error with no other changes; I need advice for how to evaluate this.
I inherited an app that had been untouched for years, after the server crashed and I needed to move it to another machine. It's built with Flask, and uses Peewee to talk to a Postgres database over pyscopg2. It has a bunch of other stuff--an Elasticsearch engine for searching, a lot of heavy JS on the front end--but that doesn't seem to be the problem here. The code is moderately complex, and I am not very knowledgeable about all of its pieces.
It took me a while to get it set up using the sketchy deployment instructions that had been left behind, but eventually I got it running, and was able to get a test version running on a clean VM and then deploy it on an actual server, using gunicorn and nginx. It's been working fine in production for a week. I'm using Debian Buster for all versions. I'm using the most recent versions of all software.
I then decided to do some basic code cleanup, and ran the entire app through a linter, before looking at some other changes to make, that the end user had requested. Unfortunately, after this, the app consistently fails at the same point with a "Bad file descriptor" error. This is in a pre-run section, which parses a large XML file and saves the info to the database and to Elasticsearch; the app receives an XML upload, forks a few processes, and runs the parse/index process in the background.
I am subsequently unable to get past this error by any means. I have launched a clean VM and installed everything from scratch; I've reverted the git repo to before I linted the code. Same problem. I don't see how it can be a code issue, as it's now at the same point it was when I started. But I'm at a loss for what to do, and terrified that the production machine will fail.
The errors I get (trimming the first few lines that refer to places in the app itself) are:
[2021-03-14 14:40:11.699837] self.execute()
[2021-03-14 14:40:11.699878] File "/home/deploy/git/myapp/venv/lib/python3.7/site-packages/peewee.py", line 1906, in inner
[2021-03-14 14:40:11.699907] return method(self, database, *args, **kwargs)
[2021-03-14 14:40:11.699946] File "/home/deploy/git/myapp/venv/lib/python3.7/site-packages/peewee.py", line 1977, in execute
[2021-03-14 14:40:11.699976] return self._execute(database)
[2021-03-14 14:40:11.700004] File "/home/deploy/git/myapp/venv/lib/python3.7/site-packages/peewee.py", line 2149, in _execute
[2021-03-14 14:40:11.700032] cursor = database.execute(self)
[2021-03-14 14:40:11.700060] File "/home/deploy/git/myapp/venv/lib/python3.7/site-packages/peewee.py", line 3156, in execute
[2021-03-14 14:40:11.700088] return self.execute_sql(sql, params, commit=commit)
[2021-03-14 14:40:11.700115] File "/home/deploy/git/myapp/venv/lib/python3.7/site-packages/peewee.py", line 3150, in execute_sql
[2021-03-14 14:40:11.700143] self.commit()
[2021-03-14 14:40:11.700171] File "/home/deploy/git/myapp/venv/lib/python3.7/site-packages/peewee.py", line 2916, in __exit__
[2021-03-14 14:40:11.700198] reraise(new_type, new_type(exc_value, *exc_args), traceback)
[2021-03-14 14:40:11.700226] File "/home/deploy/git/myapp/venv/lib/python3.7/site-packages/peewee.py", line 190, in reraise
[2021-03-14 14:40:11.700254] raise value.with_traceback(tb)
[2021-03-14 14:40:11.700282] File "/home/deploy/git/myapp/venv/lib/python3.7/site-packages/peewee.py", line 3143, in execute_sql
[2021-03-14 14:40:11.700309] cursor.execute(sql, params or ())
[2021-03-14 14:40:11.700339] OperationalError('SSL SYSCALL error: Bad file descriptor\n')
127.0.0.1 - - [14/Mar/2021 10:40:11] "POST /manage/versions/upload HTTP/1.1" 500 -
Error on request:
Traceback (most recent call last):
File "/home/deploy/git/myapp/venv/lib/python3.7/site-packages/werkzeug/serving.py", line 323, in run_wsgi
execute(self.server.app)
File "/home/deploy/git/myapp/venv/lib/python3.7/site-packages/werkzeug/serving.py", line 315, in execute
write(data)
File "/home/deploy/git/myapp/venv/lib/python3.7/site-packages/werkzeug/serving.py", line 273, in write
self.send_response(code, msg)
File "/home/deploy/git/myapp/venv/lib/python3.7/site-packages/werkzeug/serving.py", line 388, in send_response
self.wfile.write(hdr.encode("ascii"))
File "/usr/lib/python3.7/socketserver.py", line 799, in write
self._sock.sendall(b)
OSError: [Errno 9] Bad file descriptor
Exception in thread Thread-22:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.7/socketserver.py", line 654, in process_request_thread
self.shutdown_request(request)
File "/usr/lib/python3.7/socketserver.py", line 509, in shutdown_request
self.close_request(request)
File "/usr/lib/python3.7/socketserver.py", line 513, in close_request
request.close()
File "/usr/lib/python3.7/socket.py", line 420, in close
self._real_close()
File "/usr/lib/python3.7/socket.py", line 414, in _real_close
_ss.close(self)
OSError: [Errno 9] Bad file descriptor
I note that the final section ("Exception in thread Thread-22") is showing the system Python, rather than my virtual environment; I don't know if that's relevant, or if that's just what's running some overall process. I didn't get to this point doing anything different, though--the app is running in the virtual environment.
I'd be very grateful for any thoughts here--I'm obviously hoping it's some kind of stupid permission error or something, as I can't easily go into the code because of its complexity.
I've tried to render a animation using Network Render. I connected my PC an my Laptop without further problems. But when I clicked "Render animation on network" after some seconds the following error occurs:
AL lib: (EE) UpdateDeviceParams: Failed to set 44100hz, got 48000hz instead
Traceback (most recent call last):
File "F:\Program Files (x86)\Blender\2.74\scripts\addons\netrender\operat ors.py", line 85, in invoke
return self.execute(context)
File "F:\Program Files (x86)\Blender\2.74\scripts\addons\netrender\operat ors.py", line 77, in execute
scene.network_render.job_id = client.sendJob(conn, scene, True)
File "F:\Program Files (x86)\Blender\2.74\scripts\addons\netrender\client .py", line 121, in sendJob
return sendJobBlender(conn, scene, anim, can_save)
File "F:\Program Files (x86)\Blender\2.74\scripts\addons\netrender\client .py", line 340, in sendJobBlender
response = conn.getresponse()
File "F:\Program Files (x86)\Blender\2.74\python\lib\http\client.py", line 1172, in getresponse
response.begin()
File "F:\Program Files (x86)\Blender\2.74\python\lib\http\client.py", line 351, in begin
version, status, reason = self._read_status()
File "F:\Program Files (x86)\Blender\2.74\python\lib\http\client.py", line 313, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "F:\Program Files (x86)\Blender\2.74\python\lib\socket.py", line 371, inreadinto
return self._sock.recv_into(b)
socket.timeout: timed out
I asked Google: Somebody circumvents the problem by changing "the default timeout to 1000 (instead of 300) (in the socket.py file[...])". I can't find this line, I guess they changed it in the current version. Since I have no experience using python I do not know how I can change it now.
I hope you can help me!
The addon would be a better place to make the change instead of the socket module. If you look in your addons folder you will find netrender/utils.py where you will find a few lines that use socket.setdefaulttimeout and you could make some adjustments there.
An even better solution would be to look at why the connection is timing out, two computers in the same room should not get any timeouts. A common cause of timeouts is the inability to get a connection, firewalls are good at stopping connections, so you may want to check that the port used by network render is allowing incoming connections, and that blender is running with network render turned on to accept the connection. The default port is 8000 which could also be in use by another application, you can configure each computer to use a different port if needed.
I'm getting some weird behavior -- when the application starts up a new instance for the first time, I get a DeadlineExceededError. When I hit refresh in the browser it works just fine And it doesn't matter which page I try. The strange thing is I can see all my debugging code just fine. In fact, I write to the log just prior to calling self.response and it shows up in the console's log. This is pretty hard to troubleshoot, since I'm not having any page load problems in the development environment, and the traceback is a bit opaque to me:
E 2013-09-29 00:10:03.975
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 267, in Handle
for chunk in result:
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/appstats/recording.py", line 1286, in appstats_wsgi_wrapper
end_recording(status)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/appstats/recording.py", line 1410, in end_recording
rec.save()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/appstats/recording.py", line 654, in save
key, len_part, len_full = self._save()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/appstats/recording.py", line 678, in _save
namespace=config.KEY_NAMESPACE)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/memcache/__init__.py", line 1008, in set_multi
namespace=namespace)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/memcache/__init__.py", line 907, in _set_multi_with_policy
status_dict = rpc.get_result()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 612, in get_result
return self.__get_result_hook(self)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/memcache/__init__.py", line 974, in __set_with_policy_hook
rpc.check_success()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 578, in check_success
self.__rpc.CheckSuccess()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_rpc.py", line 133, in CheckSuccess
raise self.exception
DeadlineExceededError
I 2013-09-29 00:10:03.988
This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application.
I'm not sure how to even go about debugging this, since the error seems to be after all my code has already run.
Edit: I should add this:
I 2013-09-29 00:09:06.919
DEBUG: Writing output!
E 2013-09-29 00:10:03.975
You can see there's nearly a full minute between logging "Writing output!" just before self.response is called, and when the error occurs.
Deadlineexceedederror happens in app engine, if any request to a frontend instance does not get a response within 60 seconds. So what is happening in your case must be that when there are no running instance and your app receives a new user request, a new instance is started for processing. This will lead to an overall response time = instance startup time like library loading and initial data access + the time for processing the user request and this causes a deadlineexceeded error. Then when you are accessing your app immediately , there is an already running instance and so response time = the time for processing the user request and you do not get any error.
Please check the suggested approaches for handling deadlineexceedederror including warmup requests, which is like keeping an instance ready before arrival of a live user request.
I'm trying to figure out what causes this error when I run my app using the basic Flask server during development. I start it with this:
from myapp import app
app.run(debug=True, port=5001)
All is well and I'll continue to code and refresh etc, but then after a while I get the recursion error and have to Ctrl-C the server and restart it. Not a big deal, just a little annoying to have to deal with every now and then.
Here's the full traceback, which I tried to use to determine the cause but can't see anything that stands out (possibly something to do with how werkzeug uses Cookie.py?):
Traceback (most recent call last):
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/flask/app.py", line 1701, in __call__
return self.wsgi_app(environ, start_response)
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/werkzeug/wsgi.py", line 411, in __call__
return self.app(environ, start_response)
(last bit repeated a bunch - trimmed to fit in posting size requirements)
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/flask/app.py", line 1685, in wsgi_app
with self.request_context(environ):
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/flask/ctx.py", line 274, in __enter__
self.push()
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/flask/ctx.py", line 238, in push
self.session = self.app.open_session(self.request)
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/flask/app.py", line 792, in open_session
return self.session_interface.open_session(self, request)
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/flask/sessions.py", line 191, in open_session
secret_key=key)
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/werkzeug/contrib/securecookie.py", line 309, in load_cookie
data = request.cookies.get(key)
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/werkzeug/utils.py", line 77, in __get__
value = self.func(obj)
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/werkzeug/wrappers.py", line 418, in cookies
cls=self.dict_storage_class)
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/werkzeug/http.py", line 741, in parse_cookie
cookie.load(header)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Cookie.py", line 632, in load
self.__ParseString(rawdata)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Cookie.py", line 665, in __ParseString
self.__set(K, rval, cval)
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/werkzeug/_internal.py", line 290, in _BaseCookie__set
morsel = self.get(key, _ExtendedMorsel())
File "/Users/jeff/.virtualenvs/fmll/lib/python2.7/site-packages/werkzeug/_internal.py", line 271, in __init__
Morsel.__init__(self)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Cookie.py", line 438, in __init__
dict.__setitem__(self, K, "")
RuntimeError: maximum recursion depth exceeded while calling a Python object
Since it occurs during your developement process, you could increase recursion limit, before starting your server, using :
sys.setrecursionlimit(2000) # Choose the right figure for you here
# the value on my system is 1000 but this is platform-dependant
However, you should use it very carefully and probably not in production unless you have a good knowledge of it's impacts.
Ref : http://docs.python.org/2/library/sys.html#sys.setrecursionlimit
i am using google app engine on my server (listen --address=0.0.0.0 all network) my application a short time correctyl(i get some users input, some process,show the data and insert mysql db) after i take this error message and web browser didn't show anything, my system: google app engine, python 2.7, mysql(via rdbms) i change the port(8080,8091,8090vs...) or restart app engine, my applicayion again work but after then again same error message
Traceback (most recent call last):
File "C:\Python27\lib\SocketServer.py", line 284, in _handle_request_noblock
self.process_request(request, client_address)
File "C:\Python27\lib\SocketServer.py", line 310, in process_request
self.finish_request(request, client_address)
File "C:\Python27\lib\SocketServer.py", line 323, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\dev_appserver.py", line 2734, in __init__
BaseHTTPServer.BaseHTTPRequestHandler.__init__(self, *args, **kwargs)
File "C:\Python27\lib\SocketServer.py", line 639, in __init__
self.handle()
File "C:\Python27\lib\BaseHTTPServer.py", line 343, in handle
self.handle_one_request()
File "C:\Python27\lib\BaseHTTPServer.py", line 313, in handle_one_request
self.raw_requestline = self.rfile.readline(65537)
File "C:\Python27\lib\socket.py", line 476, in readline
data = self._sock.recv(self._rbufsize)
error: [Errno 10054]
Try switching to a more serious networking framework, not based on a toy HTTP server (BaseHTTPServer is a toy HTTP server).
10054 is the Windows error ECONNRESET. This indicates that the connection your HTTP server is trying to read from has been closed. It's not exactly an error condition - connections get closed, it's a normal part of their lifecycle - but the Google AppEngine (development!) server you're using seems to be treating it as an error. Perhaps in doing so it ends up responding incorrectly to other requests as well.
A correct HTTP server will not have a problem dealing with this situation.
The AppEngine development server is indeed pretty limited (single-threaded, HTTP 1.0 only, not robust to connection resets), but it is the only option for emulating in a local dev. environment the behavior of the production server, including stubs to AppEngine services and sandboxing of some modules (including os and socket).
Until Google provide us with a more robust alternative, I added exception handlers for the expected socket errors in the handle_one_request method of BaseHTTPServer.py (located in the Lib subdir of the Python installation directory)
[end of BaseHTTPServer.py::handle_one_request() - Python 2.7]
except socket.error as socket_error:
import errno
if socket_error.errno in (errno.ECONNABORTED, errno.WSAECONNABORTED, errno.ECONNRESET, errno.WSAECONNRESET):
pass
else:
raise
ECONNABORTED/WSAECONNABORTED is happening on remote disconnects - see Python issue 14574 for details.