I wrote a epoll wrapper in python, It works fine but recently I found the performance is not not ideal for large package sending. I look down into the code and found there's actually a LOT of error
Traceback (most recent call last):
File "/Users/dawn/Documents/workspace/work/dev/server/sandbox/single_point/tcp_epoll.py", line 231, in send_now
num_bytes = self.sock.send(self.response)
error: [Errno 35] Resource temporarily unavailable
and previously silent it as the document said, so my sending function was done this way:
def send_now(self):
'''send message at once'''
st = time.time()
times = 0
while self.response != '':
try:
num_bytes = self.sock.send(self.response)
l.info('msg wrote %s %d : %r size %r',self.ip,self.port,self.response[:num_bytes],num_bytes)
self.response = self.response[num_bytes:]
except socket.error,e:
if e[0] in (errno.EWOULDBLOCK,errno.EAGAIN):
#here I printed it, but I silent it in normal days
#print 'would block, again %r',tb.format_exc()
break
else:
l.warning('%r %r socket error %r',self.ip,self.port,tb.format_exc())
#must break or cause dead loop
break
except:
#other exceptions
l.warning('%r %r msg write error %r',self.ip,self.port,tb.format_exc())
break
times += 1
et = time.time()
I googled it, and says it caused by temporarily network buffer run out
So how can I manually and efficiently detect this error instead it goes to exception phase?
Because it cause to much time to rasie/handle the exception.
Use select to see if a socket is ready for reading or writing.
Related
I found a very perplexing issue in aiosmtpd/244, sharing my puzzlement here to help me find inspiration on how to troubleshoot.
Situation
Two servers running Red Hat 7.9
Works on one, doesn't work on the other
Problematic code, simplified:
>>> from aiosmtpd.controller import Controller
>>> from aiosmtpd.handlers import Sink
>>> cont = Controller(Sink())
>>> cont.start()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.6/site-packages/aiosmtpd/controller.py", line 180, in start
raise RuntimeError("Unknown Error, failed to init SMTP server")
RuntimeError: Unknown Error, failed to init SMTP server
The above RuntimeError can only be generated if, at the end of the start() method, self.smtpd is still None.
Expected flow
start() -> _run() -> loop.create_server()
Then upon first connection:
loop.create_server() -> _factory_invoker() -> factory()
The attribute smtpd is set within _factory_invoker in these lines:
try:
self.smtpd = self.factory()
if self.smtpd is None:
raise RuntimeError("factory() returned None")
return self.smtpd
except Exception as err:
self._thread_exception = err
return _FakeServer(self.loop)
self._thread_exception is interpreted in these lines:
if self._thread_exception is not None:
raise self._thread_exception
# Defensive
if self.smtpd is None:
raise RuntimeError("Unknown Error, failed to init SMTP server")
As you can see, if self.smtpd is None, then it would only happen if there's an error in _factory_invoker(). If so, the error should've been caught and recorded in self._thread_exception. If self._thread_exception is None, then _factory_invoker() had succeeded, and thus self.smtpd couldn't be None.
My main problem in troubleshooting this is that on all my test systems (Windows, Ubuntu, OpenSUSE, MacOS, and FreeBSD), I never encountered a similar error.
So I'm stumped. I'd appreciate any ideas on solving this problem.
Okay, so apparently I was sent on a wild goose chase.
The compounding factor is because I have suppressed ALL exceptions in these lines:
try:
self._testconn()
except Exception:
# We totally don't care of exceptions experienced by _testconn,
# which _will_ happen if factory() experienced problems.
pass
As a result, a very helpful exception raised within self._testconn() was swallowed.
Apparently self._testconn() failed in its attempt to connect to the host:port, and thus _factory_invoker() never got called.
Once the try..except block is modified to only swallow socket.socket_timeout, the actual problem reared its head, and we can quickly fix it.
It's all documented in aiosmtpd/244 and thus I won't repeat them here :-)
Thank you all who spent the time & effort to try solving this puzzle!
There is a function in my code (which gets executed in typically 1-2 seconds). For a particular set of arguments, I see that the python gets hung and eventually after few hours the kernel gets killed. For whatever reason (not figured out yet), the same parameter set executes the function just fine. So my question is:
Is there a way to control the amount of time assigned to execute a particular line of code? If it exceeds the time assigned to that particular line, can we
a) Re-run the code automatically or
b) Go back few steps to re-execute that line or
c) Skip the execution of that line
EDIT: This code catches runtime error for some iteration (number of iteration is not fixed) but then gets hung. I need to skip the following line:
obj = package.class0(key='Key0', **p, _verbose=0)
This is causing kernel getting hung
import package
par = [1.294505562707576551e+01, 6.965364410093530800e-01, 5.707082620814450280e+00,
2.537535861030653095e+00,
1.670415320843951779e-01, -1.528978502298340203e+01, 2.379290972641865665e+02,
3.085337613758109665e+00,
3.420212454861922424e+01,
2.943414981550797904e+01,
2.059940694387249849e+01,
-1.147125847273145638e+02,
7.290403481752945059e-01, 3.246951759540853004e+02, 1.179614954727290069e+03,
7.159885607208993363e-01]
m = [1.007070707070707005e+00]
p = collections.OrderedDict()
for idx, key in enumerate(bound_list.keys()):
p[key] = par[idx]
for i in range(1000):
try:
obj = package.class0(key='Key0', **p, _verbose=0)
#res = obj.get_vals(m)
except Exception as ex:
obj = None
#raise
template = "An exception of type {0} occurred. Arguments:\n{1!r}"
message = template.format(type(ex).__name__, ex.args)
print(message)
print("Done! "+str(i))
We use cx_Oracle to connect multiple threads to the database and issue various selects and updates.
However, for unknown reasons, the script is being killed by the system on random database connections. There are no informations in the syslog or messages-files.
Due to the error handling we try to write tracebacks in the logfile. Unfortunately we have no information about the crash of the script in the logfile. Only in stdout is a printout with "PID killed" at the last line.
Could it be a problem to make database connections with multiple threads at the same time? There are also other scripts running at the same time that also talk to the database (not multi-threaded) but access other tables.
This is our Function that is called for every select. For the updates we have other Functions.
def ora_send_sql( logger, statement):
try:
dsn = cx_Oracle.makedsn(SQL_IP, SQL_PORT, SQL_SID)
con = cx_Oracle.connect(SQL_USR, SQL_PWD, dsn)
cur = con.cursor()
cur.execute(statement)
con.commit()
con.close()
return 0
except cx_Oracle.Warning as w:
logger.warning(" Oracle-Warning: "+ str(e).strip())
pass
except cx_Oracle.Error as e:
error, = exc.args
logger.error(" Oracle-Error-Code:", error.code)
logger.error(" Oracle-Error-Message:", error.message)
return -1
except:
exc_type, exc_obj, exc_tb = sys.exc_info()
fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
logger.exception(" Got Traceback in ora_send_sql: " + str(exc_type) + " Fname=" + str(fname) + " Lineno=" + str(exc_tb.tb_lineno))
return -2
I don't suppose you tried turning except: into except Exception as e and then trying to see if the exception is somewhat special. Another possible thing to try perhaps is removing the exception handling completely and letting it crash, then investigating the output. That way you could spot the actual exception that is thrown, because I simply cannot believe it would just "crash". Finally, try investigating dmesg for any segfaults.
I think we have fixed the problem by updating cx_Oracle to the latest version.
They fixed a few memory-leaks in the updates.....
But that still does not explain why we do not see any information about the killing....
You almost certainly need to use threaded=True in the connect() call, see http://cx-oracle.readthedocs.io/en/latest/module.html#cx_Oracle.Connection
#main program
while True:
ReadValue = Func03Modbus(1,70,40);#slave,start,number of registers
x3 = struct.pack('>HH',abs(ReadValue[3]),abs(ReadValue[2]))
pressure = struct.unpack('>f', x3)
print pressure[0]
c3 = struct.pack('>HH',abs(ReadValue[5]),abs(ReadValue[4]))
purity = struct.unpack('>f', c3)
print purity[0]
hrs = int(ReadValue[30])
mins= int(ReadValue[31])
timein =float(str(ReadValue[30])+"."+str(ReadValue[31]))
print timein
r=requests.get("http://api.thingspeak.com/update api_key=5RMT************&field4="+str(pressure[0])+"&field5="+str(purity[0])+"&field1="+str(ReadValue[i])+"&field2="+str(mins)+"&field3="+str(timein)))
print str(ReadValue[30])
time.sleep(15)
While running the above program it stops running with returning following error:
Traceback (most recent call last): File "/home/pi/v1.py", line 123,
in
x3 = struct.pack('>HH',abs(ReadValue[3]),abs(ReadValue[2])); IndexError: tuple index out of range
I want my program to run continuously even when it returns error. I want to skip the error and to run the program continuously. How can I do that ?
In theory you could wrap the code in an exception handler like:
while True:
try:
what you want to do
except Exception as e:
print("Something bad happened:", e)
finally:
# reset device here
time.sleep(15)
But this seems like a really bad idea if you're interacting with hardware, since you can't be sure what state you're leaving it in. Ideally, you'd want to make sure you're doing a proper reset of the device (or reconnect? depends what you're talking to) on every cycle.
Alternatively, if you want to explicitly verify that the values you get back are available, you can do:
ReadValue = Func03Modbus(1,70,40);#slave,start,number of registers
if len(ReadValue) < 32:
print("Got incomplete result")
time.sleep(15)
continue
The language reference/tutorial has more information about handling errors here: https://docs.python.org/3/tutorial/errors.html
In order to continue in the event of this kind of error, simply place the part you wish to ignore exceptions in within an appropriate try: ... except ...
while True:
try:
<body of work>
except IndexError:
<you might want to log the error>
pass
In this case, we continue only in the event of IndexError.
I am a relative python newbie and I am getting confused with how to properly handle exceptions. Apologies for the dumb question.
In my main() I iterate through a list of dates and for each date I call a function, which downloads a csv file from a public web server. I want to properly catch exceptions for obvious reasons but especially because I do not know when the files of interest will be available for download. My program will execute as part of a cron job and will attempt to download these files every 3 hours if available.
What I want is to download the first file in the list of dates and if that results in a 404 then the program shouldn't proceed to the next file because the assumption is if the oldest date in the list is not available then none of the others that come after it will be available either.
I have the following python pseudo code. I have try/except blocks inside the function that attempts to download the files but if an exception occurred inside the function how do I properly handle it in the main() so I can make decisions whether to proceed to the next date or not. The reason why I created a function to perform the download is because I want to re-use that code later on in the same main() block for other file types.
def main():
...
...
# datelist is a list of date objects
for date in datelist:
download_file(date)
def download_file(date):
date_string = str(date.year) + str(date.strftime('%m')) + str(date.strftime('%d'))
request = HTTP_WEB_PREFIX+ date_string + FILE_SUFFIX
try:
response = urllib2.urlopen(request)
except urllib2.HTTPError, e:
print "HTTPError = " + str(e)
except urllib2.URLError, e:
print "URLError = " + str(e)
except httplib.HTTPException, e:
print "HTTPException = " + str(e)
except IOError:
print "IOError = " + str(e)
except Exception:
import traceback
print "Generic exception: " + traceback.format_exc()
else:
print "No problem downloading %s - continue..." % (response)
try:
with open(TMP_DOWNLOAD_DIRECTORY + response, 'wb') as f:
except IOError:
print "IOError = " + str(e)
else:
f.write(response.read())
f.close()
The key concept here is, if you can fix the problem, you should trap the exception; if you can't, it's the caller's problem to deal with. In this case, the downloader can't fix things if the file isn't there, so it should bubble up its exceptions to the caller; the caller should know to stop the loop if there's an exception.
So let's move all the exception handling out of the function into the loop, and fix it so it craps out if there's a failure downloading the file, as the spec requires:
for date in datelist:
date_string = str(date.year) +
str(date.strftime('%m')) +
str(date.strftime('%d'))
try:
download_file(date_string)
except:
e = sys.exc_info()[0]
print ( "Error downloading for date %s: %s" % (date_string, e) )
break
download_file should now, unless you want to put in retries or something like that, simply not trap the exceptions at all. Since you've decoded the date as you like in the caller, that code can come out of download_file as well, giving the much simpler
def download_file(date_string):
request = HTTP_WEB_PREFIX + date_string + FILE_SUFFIX
response = urllib2.urlopen(request)
print "No problem downloading %s - continue..." % (response)
with open(TMP_DOWNLOAD_DIRECTORY + response, 'wb') as f:
f.write(response.read())
f.close()
I would suggest that the print statement is superfluous, but that if you really want it, using logger is a more flexible way forward, as that will allow you to turn it on or off as you prefer later by changing a config file instead of the code.
From my understanding of your question... you should just insert code into the except blocks that you want to execute when you encounter the particular exception. You don't have to print out the encountered error, you can do whatever you feel is necessary when it is raised... provide a popup box with information/options or otherwise direct your program to the next step. Your else section should isolate that portion, so it will only execute if none of your exceptions are raised.