We Keep Coding

TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time Email-Listener

I'm trying to create a Python code for automatically listen any incoming email then download attachment of that email. I found Email-Listener library and practice the documentation. I'm using the exact code from the documentation. This is the code:
import email_listener
# Set your email, password, what folder you want to listen to, and where to save attachments
email = "example#gmail.com"
app_password = "password"
folder = "Inbox"
attachment_dir = "/path/to/attachments"
el = email_listener.EmailListener(email, app_password, folder, attachment_dir)
# Log into the IMAP server
el.login()
# Get the emails currently unread in the inbox
messages = el.scrape()
print(messages)
# Start listening to the inbox and timeout after an hour
timeout = 60
el.listen(timeout)
Well it works, when program running it will listen and download any attachment from unread email, either already exist on inbox or not. But after about 15+ minutes the program will stopped and display this error message:
Traceback (most recent call last):
File "app.py", line 20, in <module>
el.listen(timeout)
File "D:\PreProject\email_auto\emailauto_env\lib\site-packages\email_listener\__init__.py", line 333, in listen
self.__idle(process_func=process_func, **kwargs)
File "D:\PreProject\email_auto\emailauto_env\lib\site-packages\email_listener\__init__.py", line 385, in __idle
self.server.idle_done()
File "D:\PreProject\email_auto\emailauto_env\lib\site-packages\imapclient\imapclient.py", line 175, in wrapper
return func(client, *args, **kwargs)
File "D:\PreProject\email_auto\emailauto_env\lib\site-packages\imapclient\imapclient.py", line 975, in idle_done
return self._consume_until_tagged_response(self._idle_tag, "IDLE")
File "D:\PreProject\email_auto\emailauto_env\lib\site-packages\imapclient\imapclient.py", line 1596, in _consume_until_tagged_response
line = self._imap._get_response()
File "c:\users\0974\appdata\local\programs\python\python37\lib\imaplib.py", line 1047, in _get_response
resp = self._get_line()
File "c:\users\0974\appdata\local\programs\python\python37\lib\imaplib.py", line 1149, in _get_line
line = self.readline()
File "D:\PreProject\email_auto\emailauto_env\lib\site-packages\imapclient\tls.py", line 59, in readline
return self.file.readline()
File "c:\users\0974\appdata\local\programs\python\python37\lib\socket.py", line 589, in readinto
return self._sock.recv_into(b)
File "c:\users\0974\appdata\local\programs\python\python37\lib\ssl.py", line 1071, in recv_into
return self.read(nbytes, buffer)
File "c:\users\0974\appdata\local\programs\python\python37\lib\ssl.py", line 929, in read
return self._sslobj.read(len, buffer)
TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
Is there any solutions? And I don't mind if the solutions use other methods, like not using Email-Listener library

Python XMLRPC: Cannot allow None even after allow_none = True

I'm trying to write a simple application that communicates using RPCs. I'm using python 3.7's xmlrpc.
This is my server code
MY_ADDR = ("localhost", int(sys.argv[1]))
HOST_ADDR = ("localhost", int(sys.argv[2]))
class RpcServer(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.port = MY_ADDR[1]
self.addr = MY_ADDR[0]
# serve other hosts using this
self.server = SimpleXMLRPCServer((self.addr, self.port))
self.server.register_function(self.recv_ops)
def run(self):
self.server.serve_forever()
def recv_ops(self, sender, op):
print("Sender ", sender, " sent: ", op)
pass
And this is what I'm using as my client's code
def send_ops(host_addr, op):
# contact the other host using this
proxy_addr = "http://{addr}:{port}/".format(addr=host_addr[0], port=host_addr[1])
client_proxy = xmlrpc.client.ServerProxy(proxy_addr, allow_none=True)
resp = client_proxy.recv_ops(MY_ADDR, op)
...
send_ops(HOST_ADDR, ("d", ii, last_line[ii])) # THE RPC CALL I MAKE
Despite setting allow_none=True, I keep getting this:
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "nb.py", line 102, in editor
send_ops(HOST_ADDR, ("d", ii, last_line[ii]))
File "nb.py", line 63, in send_ops
resp = client_proxy.recv_ops(MY_ADDR, op)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 1112, in __call__
return self.__send(self.__name, args)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 1452, in __request
verbose=self.__verbose
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 1154, in request
return self.single_request(host, handler, request_body, verbose)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 1170, in single_request
return self.parse_response(resp)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 1342, in parse_response
return u.close()
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 656, in close
raise Fault(**self._stack[0])
xmlrpc.client.Fault: <Fault 1: "<class 'TypeError'>:cannot marshal None unless allow_none is enabled">
What's tripping my is that the server on the other side actually receives the message (without any None)
Sender ['localhost', 8001] sent: ['d', 4, 'o']
What am I missing here? Any help would be appreciated.
Thanks!

In your server class, add allow_none=True to your SimpleXMLRPCServer instantiation.
self.server = SimpleXMLRPCServer((self.addr, self.port), allow_none=True)
The allow_none and encoding parameters are passed on to xmlrpc.client and control the XML-RPC responses that will be returned from the server.

How do you deal with an exception raised by celery (not your code)?

So in my flask app right now I am using Celery to deploy servers on remote machines. Right now, I have an enum, status, which indicates the lifecycle of my deployment process:
#celery.task(bind=True)
def deploy_server(self, server_id):
server = Server.query.get(server_id)
if not server.can_launch():
return
try:
server.status = RemoteStatus.LAUNCHING
db.session.commit()
verify_DNS(server)
host = server.server.ssh_user + '#' + server.server.ip
execute(fabric_deploy_server, self, server, hosts=host)
server.status = RemoteStatus.LAUNCHED
db.session.commit()
except Exception as e:
server.status = RemoteStatus.ERROR
db.session.commit()
traceback.print_exc()
raise e
As you can see, when a server is being deployed, its status is changed to "Launching". If there is an exception, it will be changed to ERROR.
I found one exception which completely bypasses this bloc of code: when I overloaded my celery server with too many requests, I get this exception:
[2017-07-09 18:00:03,127: WARNING/PoolWorker-3] /app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py:542: RuntimeWarning: Exception raised outside body: ConnectionError('max number of clients reached',):
Traceback (most recent call last):
File "/app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py", line 427, in trace_task
uuid, retval, task_request, publish_result,
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 152, in mark_as_done
self.store_result(task_id, result, state, request=request)
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 309, in store_result
request=request, **kwargs)
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 652, in _store_result
self.set(self.get_key_for_task(task_id), self.encode(meta))
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 204, in set
return self.ensure(self._set, (key, value), **retry_policy)
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 194, in ensure
**retry_policy)
File "/app/.heroku/python/lib/python2.7/site-packages/kombu/utils/functional.py", line 333, in retry_over_time
return fun(*args, **kwargs)
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 213, in _set
pipe.execute()
File "/app/.heroku/python/lib/python2.7/site-packages/redis/client.py", line 2641, in execute
return execute(conn, stack, raise_on_error)
File "/app/.heroku/python/lib/python2.7/site-packages/redis/client.py", line 2495, in _execute_transaction
connection.send_packed_command(all_cmds)
File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 538, in send_packed_command
self.connect()
File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 446, in connect
self.on_connect()
File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 514, in on_connect
if nativestr(self.read_response()) != 'OK':
File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 577, in read_response
response = self._parser.read_response()
File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 255, in read_response
raise error
ConnectionError: max number of clients reached
exc, exc_info.traceback)))
My biggest problem with this is that this error is raised somewhere outside of my Try/Catch bloc. Hence, when this exception occurs, all my servers remain in the "Launching" mode rather than "Error".
How can I catch this exception and modify Server.status?

In redis 2.4 there is a hard coded limit of max number of connections which is 10,000. In redis 2.6+ you can specify the max number of clients in redis.conf. also this is not a problem of celery your broker redis refused to accept connections that's the problem.
Set the max number of clients that can be handled by redis simultaneously using redis CLI. Check out redis clients

Python xmlrpclib raise IncompleteRead Error

I have a simple xmlrpc server which is written along the line of
server = SimpleXMLRPCServer(('127.0.0.1', 8000),allow_none=True)
server.register_function(self.fetch_buyer_data,'"fetch_buyer_data")
...
...
server.serve_forever()
This is not the complete code, but you get the idea (hopefully) !
In the same server script, I have a function that reads off the contents of an sqlite3 database and return all the data. Something like this:
def fetch_buyer_data(self, projectname):
conn = sqlite3.connect(...)
# read data from sqlite3 database and save it into list
conn.close()
return datalist
And I use this following code from client to access the above function.
proxy = xmlrpclib.ServerProxy("http://%s:%s/" %(hostip,hostport),allow_none=True)
data = proxy.fetch_buyer_data(SELECTED_PROJECT)
It's all good until the data in sqlite3 database get larger (not very large but something like a few Megabytes!), I keep getting the following error message!
Traceback (most recent call last):
File "C:\Custom\src\Client\client.py", line 178, in show_user_page
userpage = UserPage()
File "C:\Custom\src\Client\client.py", line 2371, in __init__
self.update_buyer_table()
File "C:\Custom\src\Client\client.py", line 6106, in update_buyer_table
data = proxy.fetch_buyer_data(SELECTED_PROJECT)
File "C:\Python27\Lib\xmlrpclib.py", line 1224, in __call__
return self.__send(self.__name, args)
File "C:\Python27\Lib\xmlrpclib.py", line 1578, in __request
verbose=self.__verbose
File "C:\Python27\Lib\xmlrpclib.py", line 1264, in request
return self.single_request(host, handler, request_body, verbose)
File "C:\Python27\Lib\xmlrpclib.py", line 1297, in single_request
return self.parse_response(response)
File "C:\Python27\Lib\xmlrpclib.py", line 1453, in parse_response
stream = GzipDecodedResponse(response)
File "C:\Python27\Lib\xmlrpclib.py", line 1204, in __init__
self.stringio = StringIO.StringIO(response.read())
File "C:\Python27\Lib\httplib.py", line 548, in read
s = self._safe_read(self.length)
File "C:\Python27\Lib\httplib.py", line 649, in _safe_read
raise IncompleteRead(''.join(s), amt)
httplib.IncompleteRead: IncompleteRead(8031 bytes read, 1732 more expected)
NOTE: I have checked that the rest of the registered functions on server are working. So,I can rule out the connection problems(ip,port etc).
What is casuing this error message? How do I overcome the problem?
I am using python 2.7 on windows xp sp3.
UPDATE 1:
I found out that this doesn't entirely depends on the size of the database. Sometimes it gives me error message, sometimes it doesn't. Could anyone tell me what is casuing this IncompleteRead Problem?

Python BaseHTTPServer, how do I catch/trap "broken pipe" errors?

I build a short url translator engine in Python, and I'm seeing a TON of "broken pipe" errors, and I'm curious how to trap it best when using the BaseHTTPServer classes. This isn't the entire code, but gives you an idea of what I'm doing so far:
from BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer
import memcache
class clientThread(BaseHTTPRequestHandler):
def do_GET(self):
content = None
http_code,response_txt,long_url = \
self.ag_trans_url(self.path,content,'GET')
self.http_output( http_code, response_txt, long_url )
return
def http_output(self,http_code,response_txt,long_url):
self.send_response(http_code)
self.send_header('Content-type','text/plain')
if long_url:
self.send_header('Location', long_url)
self.end_headers()
if response_txt:
self.wfile.write(response_txt)
return
def ag_trans_url(self, orig_short_url, post_action, getpost):
short_url = 'http://foo.co' + orig_short_url
# fetch it from memcache
long_url = mc.get(short_url)
# other magic happens to look it up from db if there was nothing
# in memcache, etc
return (302, None, log_url)
def populate_memcache()
# connect to db, do lots of mc.set() calls
def main():
populate_memcache()
try:
port = 8001
if len(sys.argv) > 1:
port = int(sys.argv[1])
server = HTTPServer(('',port), clientThread)
#server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
print '[',str(datetime.datetime.now()),'] short url processing has begun'
server.serve_forever()
except KeyboardInterrupt,SystemExit:
print '^C received, shutting down server'
server.socket.close()
The code itself works great, but started throwing errors almost immediately when in production:
Traceback (most recent call last):
File "/usr/lib/python2.5/SocketServer.py", line 222, in handle_request
self.process_request(request, client_address)
File "/usr/lib/python2.5/SocketServer.py", line 241, in process_request
self.finish_request(request, client_address)
File "/usr/lib/python2.5/SocketServer.py", line 254, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.5/SocketServer.py", line 522, in __init__
self.handle()
File "/usr/lib/python2.5/BaseHTTPServer.py", line 316, in handle
self.handle_one_request()
File "/usr/lib/python2.5/BaseHTTPServer.py", line 310, in handle_one_request
method()
File "/opt/short_url_redirector/shorturl.py", line 38, in do_GET
self.http_output( http_code, response_txt, long_url )
File "/opt/short_url_redirector/shorturl.py", line 52, in http_output
self.send_response(http_code)
File "/usr/lib/python2.5/BaseHTTPServer.py", line 370, in send_response
self.send_header('Server', self.version_string())
File "/usr/lib/python2.5/BaseHTTPServer.py", line 376, in send_header
self.wfile.write("%s: %s\r\n" % (keyword, value))
File "/usr/lib/python2.5/socket.py", line 274, in write
self.flush()
File "/usr/lib/python2.5/socket.py", line 261, in flush
self._sock.sendall(buffer)
error: (32, 'Broken pipe')
The bulk of these errors seem to stem from having a problem calling the send_header() method where all I'm writing out is this:
self.send_header('Location', long_url)
So I'm curious where in my code to try to trap for this IO exception... do I write try/except calls around each of the self.send_header/self.end_headers/self.wfile.write calls? The other error I see from time to time is this one, but not sure which exception to watch to even catch this:
Traceback (most recent call last):
File "/usr/lib/python2.5/SocketServer.py", line 222, in handle_request
self.process_request(request, client_address)
File "/usr/lib/python2.5/SocketServer.py", line 241, in process_request
self.finish_request(request, client_address)
File "/usr/lib/python2.5/SocketServer.py", line 254, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.5/SocketServer.py", line 522, in __init__
self.handle()
File "/usr/lib/python2.5/BaseHTTPServer.py", line 316, in handle
self.handle_one_request()
File "/usr/lib/python2.5/BaseHTTPServer.py", line 299, in handle_one_request
self.raw_requestline = self.rfile.readline()
File "/usr/lib/python2.5/socket.py", line 381, in readline
data = self._sock.recv(self._rbufsize)
error: (104, 'Connection reset by peer')

This appears to be a bug in SocketServer, see this link Python Bug: 14574
A fix (works for me in Python 2.7) is to override the SocketServer.StreamRequestHandler finish() method, something like this:
...
def finish(self,*args,**kw):
try:
if not self.wfile.closed:
self.wfile.flush()
self.wfile.close()
except socket.error:
pass
self.rfile.close()
#Don't call the base class finish() method as it does the above
#return SocketServer.StreamRequestHandler.finish(self)

The "broken pipe" exception means that your code tried to write to a socket/pipe which the other end has closed. If the other end is a web browser, the user could have stopped the request. You can ignore the traceback; it does not indicate a serious problem. If you want to suppress the message, you can put a try ... except block around all of the code in your http_output function, and log the exception if you like.
Additionally, if you want your HTTP server to process more than one request at a time, you need your server class to use one of the SocketServer.ForkingMixIn and SocketServer.ThreadingMixIn classes. Check the documentation of the SocketServer module for details.
Add: The "connection reset by peer" exception means that your code tried to read from a dead socket. If you want to suppress the traceback, you will need to extend the BaseHTTPServer class and override the handle_one_request method to add a try ... except block. You will need a new server class anyway, to implement the earlier suggestion about processing more than one request at a time.

In my application, the error didn't occur in finish(), it occurred in handle(). This fix catches the broken pipe errors:
class MyHTTPRequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
...
def handle(self):
try:
BaseHTTPServer.BaseHTTPRequestHandler.handle(self)
except socket.error:
pass