We Keep Coding

How to detect realtime listener errors in firebase firestore database?

Firestore listeners will randomly close after some length of time (possibly due to inactivity), and in python there is no easy way of catching the errors they throw because they throw them in a separate thread. For my case, I want to maintain a long lasting listener that never closes due to inactivity or server side error.
I've tried wrapping everything in a try - except, and then wrapping that all in the while(True) loop, but that doesn't catch the error because the error is thrown in a separate thread.
The error occurs after 10 minutes - 24 hours of inactivity (I'm not sure inactivity is the case, it could be random, but the shortest interval I ever found was 10 minutes after starting it) on both Linux and windows devices. I haven't tried Mac or any other devices, but I doubt it's device specific.
Looking at gRPC (the thing listeners use to communicate between client and server) spec, there is no default timeout for the python api (and a timeout wouldn't explain why it disconnects after different amounts of time), and no timeout it set anywhere in Firestores listener code.
The specific error that occurs is:
google.api_core.exceptions.InternalServerError: 500 Received RST_STREAM with error code 0
and sometimes
google.api_core.exceptions.InternalServerError: 500 Received RST_STREAM with error code 2
Minimal code to show the problem (left running on a dummy collection called info that only has one document in it for a while):
class TestWatchInfo():
def __init__(self):
self.query_watch = db.collection(u'info').on_snapshot(self.on_snapshot)
def on_snapshot(self, col_snapshot, changes, read_time):
try:
for change in changes:
pass
except Exception as err:
print(err)
print("Error occurred at " + str(time.ctime()))
traceback.print_exc()
if __name__ == '__main__':
try:
test_object = TestWatchInfo()
while(True):
time.sleep(60)
except Exception as err:
print(err)
print("Error occurred at " + str(time.ctime()))
traceback.print_exc()
Ideally, I would be able to catch the actual error that occurs in the main python thread, but as far as I can tell since I am not the one spawning the threads I have no way of adding thread/gRPC specific code to catch that error. Alternatively, I would like to be able to auto-restart the gRPC connection after it gets closed due to the server side.
In actuality, the Firestore listener just raises an error in the thread it created and closes the listener.

I figured out an alternative method to detecting the listener error and restarting the listener after a server side close. I have no idea how to catch the actual error, but I figured out how to detect when Firestore just randomly closes the listener connection.
In the Firebase listener code they keep track of a private variable '_closed' that becomes true if the connection ever gets closed for any reason. Therefore, if we periodically check that, we can restart our listener and be on our merry way.
Using the code from before, I added a new method start_snapshot in order to restart our failed listener expression on error, and in my long running code, I added a check against the listener to see if it is closed, and restart it if it is.
class TestWatchInfo():
def __init__(self):
self.start_snapshot()
def start_snapshot(self):
self.query_watch = db.collection(u'info').on_snapshot(self.on_snapshot)
def on_snapshot(self, col_snapshot, changes, read_time):
try:
for change in changes:
pass
except Exception as err:
print(err)
print("Error occurred at " + str(time.ctime()))
traceback.print_exc()
if __name__ == '__main__':
try:
test_object = TestWatchInfo()
while(True):
if test_object.query_watch._closed:
test_object.start_snapshot()
# code here
except Exception as err:
print(err)
print("Error occurred at " + str(time.ctime()))
traceback.print_exc()

How to avoid program freezing when connecting to server

I have a little script which filters those domain names which are not registred yet. I use pywhois module. The problem is that it suddenly freeze and do nothing after several (sometimes hundreds) of requests. I think it is not a ban because I can run the program right after freeze and it works.
I would like to avoid this freezing. My idea is to count runtime of the function and if the time cross some line (for example 10 seconds) it repeats the code.
Do you have any advice how to avoid the freezing? Or the better way to check domains?
Here is the code:
for keyword in keywords:
try:
details = pythonwhois.get_whois(keyword+'.com')
except Exception as e:
print e
continue
if 'status' not in details.keys():
print 'Free domain!'
print keyword

This method is prone to change (if the underlying library changes), however, you can call internal socket functions to set a timeout for all pythonwhois network calls. For example:
TIMEOUT = 5.0 # timeout in seconds
pythonwhois.net.socket.setdefaulttimeout(TIMEOUT)
pythonwhois.get_whois("example.com")

Maybe you could try dnspython. It looks like you just want to check if a domain name is registered. For example:
import dns.resolver
for keyword in keywords:
try:
dns.resolver.query(keyword+'.com')
except dns.resolver.NXDOMAIN:
print(keyword+'.com is available!')
DNS resolver has a default timeout of 2 seconds. If you want to change that, you can make a new instance of dns.resolver.Resolver with a different timeout.
To make it multithreaded, a thread pool would be the best choice if you can use python3:
from multiprocessing import Pool
def check_keyword(keyword):
try:
dns.resolver.query(keyword+'.com')
except dns.resolver.NXDOMAIN:
# You probably want to change this to a return
print(keyword+'.com is available!')
if __name__ == '__main__':
keywords = [...]
p = Pool(5)
print(p.map(check_keyword, keywords))

Python script interrupts only after buffer receives data using socket and KeyboardInterrupt sent

I'm using Python version 2.7.9 on Windows 8.1 Enterprise 64-bit.
(Python 2.7.9 (default, Dec 10 2014, 12:28:03) [MSC v.1500 64 bit (AMD64)] on win32)
So I am writing a python IRC bot and everything works within the script.
The issue that I am having is if I send a KeyboardInterrupt, the script in the console window shows that it is still running UNTIL the bot receives data.
Situation:
Execute script to connect to IRC server
Logs onto the server no problem
In console window, I send a CTRL + C
Console window hangs making it seem like script is running
Send bot a query message / message is sent to the channel
Console shows interrupt and exit message I designated in the exception
Shouldn't the script immediately quit once a CTRL + C is sent to the console? I do have a part in the script to make it close gracefully if I send a quit message, but this part bothers me.
Here is my code where I believe it may have issues:
def main(NETWORK, NICK, CHAN, PORT):
flag = True
readbuffer = ""
global CURRENTCHANNELS
global MAXCHANNELS
s.connect((NETWORK,PORT))
s.send("NICK %s\r\n" % NICK)
s.send("USER %s %s bla :%s\r\n" % (IDENTD, NETWORK, REALNAME))
while(flag):
try:
readbuffer = readbuffer + s.recv(4096)
except KeyboardInterrupt:
print "Interrupt received"
finally:
s.close()

On Windows, with Python 2.x, Ctrl-C generally does not interrupt socket calls.
In some cases, Ctrl-Break works. If it does, and if that's good enough for you, you're done.
But if Ctrl-Break doesn't work, or if that isn't acceptable as a workaround, the only option is to set your own console ctrl-key handler, with [SetConsoleControlHandler](
https://msdn.microsoft.com/en-us/library/windows/desktop/ms686016%28v=vs.85%29.aspx).
There's a good discussion of this on the PyZMQ issue tracker, including a link to some sample code for working around it.
The code could be simpler if you can use the win32api module from PyWin32, but assuming you can't, I think this is the code you want:
from ctypes import WINFUNCTYPE, windll
from ctypes.wintypes import BOOL, DWORD
kernel32 = windll.LoadLibrary('kernel32')
PHANDLER_ROUTINE = WINFUNCTYPE(BOOL, DWORD)
SetConsoleCtrlHandler = kernel32.SetConsoleCtrlHandler
SetConsoleCtrlHandler.argtypes = (PHANDLER_ROUTINE, BOOL)
SetConsoleCtrlHandler.restype = BOOL
CTRL_C_EVENT = 0
CTRL_BREAK_EVENT = 1
#PHANDLER_ROUTINE
def console_handler(ctrl_type):
if ctrl_type in (CTRL_C_EVENT, CTRL_BREAK_EVENT):
# do something here
return True
return False
if __name__ == '__main__':
if not SetConsoleCtrlHandler(console_handler, True):
raise RuntimeError('SetConsoleCtrlHandler failed.')
The question is what to put in the # do something here. If there were an easy answer, Python would already be doing it. :)
According to the docs, HandlerRoutine functions actually run on a different thread. And, IIRC, closing the socket out from under the main thread will always cause its recv to wake up and raise an exception. (Not the one you want, but still, something you can handle.)
However, I can't find documentation to prove that—it's definitely not recommended (and WinSock2 seems to officially allow the close to fail with WSAEINPROGRESS, even if no Microsoft implementation of WinSock2 actually does that…), but then you're just trying to bail out and quit here.
So, I believe just defining and installing the handler inside main so you can write s.close() as the # do something here will work.
If you want to do this in a way that's guaranteed to be safe and effective, even with some weird WinSock2 implementation you've never heard of being, what you need to do is a bit more complicated. You need to make the recv asynchronous, and use either Windows async I/O (which is very painful from Python 2.x, unless you use a heavy-duty library like Twisted), or write cross-platform select-based code (which isn't very Windows-y, but works—as long as your use an extra socket instead of the usual Unix solution of a pipe). Like this (untested!) example:
# ctypes setup code from above
def main():
extrasock = socket.socket(socket.SOCK_DGRAM)
extrasock.bind(('127.0.0.1', 0))
extrasock.setblocking(False)
#PHANDLER_ROUTINE
def console_handler(ctrl_type):
if ctrl_type in (CTRL_C_EVENT, CTRL_BREAK_EVENT):
killsock = socket.socket(socket.SOCK_DGRAM)
killsock.sendto('DIE', extrasock.getsockname())
killsock.close()
return True
return False
if not SetConsoleCtrlHandler(console_handler, True):
raise RuntimeError('SetConsoleCtrlHandler failed.')
# your existing main code here, up to the try
# except that you need s.setblocking(False) too
try:
r, _, _ = select.select([s, extrasock], [], [])
if extrasock in r:
raise KeyboardInterrupt
if s in r:
readbuffer = readbuffer + s.recv(4096)
except KeyboardInterrupt:
# rest of your code here
If this makes no sense to you, the Sockets HOWTO actually has a pretty good explanation of how to use select, and the pitfalls of using it on Windows.

How to launch win32 applications in separate threads in Python

So, I am having this following snippet which attempts to start Microsoft Powerpoint through the win32api:
import threading
import win32com.client
import sys
class myDemo(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
def run(self):
try:
myObject = win32com.client.Dispatch("Powerpoint.Application")
print "OK"
except:
print "Failed to start Powerpoint!"
sys.exit(1)
print "Now attempting to shutdown..."
try:
myObject.quit()
except:
print "Error"
if __name__ == "__main__":
test = myDemo()
test.start()
The problem is that it fails and I have no clue why.
However, if I change the last line to test.run() it will launch successfully.
So again why is this failing with test.start()?
Why is this happening and how should I solve it considering I need Powerpoint to run on a seperate thread asynchronously?
Thanks in advance.
EDIT: Apparently my question is somehow related to this: http://python.6.x6.nabble.com/Dispatch-error-CoInitialize-has-not-been-called-td1951088.html
However apart from the proposed proper solution no one seems to answer why exactly COM is behaving this way.

I'm afraid your question likely can't be summed up in one or two sentences due to complexities in COM and threading and why they work the way they do. But for starters, here's some good information why COM behaves the way it does under threading:
http://msdn.microsoft.com/en-us/library/ms809971.aspx
Additionally, you should consider reviewing the book Python Programming on Win32. It contains useful information that sheds more light on COM threading. (Despite its age it is still useful.)
Finally, in case it wasn't clear from the reference you provided, whenever your program uses threads and COM, you must indicate in code that you're going to use COM within a thread:
import pythoncom
import win32com.client
### ... inside the thread function ...
x = win32com.client.Dispatch("someCOMobject")
win32com.CoInitialize()
# com calls here
win32com.CoUninitialize()
This type of call uses what's called single-apartment threading. It occurs when the threaded code itself instantiates COM objects.
If you find yourself instantiating a single COM object outside the threaded code (and using the instantiated object in the threaded code e.g. passing access to the COM object between threads), then this type of COM threading is called multithreaded-apartment threading:
import sys
sys.coinit_flags = 0
import pythoncom
import win32com.client
# ... outside the thread function ...
x = win32com.client.Dispatch("someCOMobject")
# ... inside the thread function ...
pythoncom.CoInitialize(pythoncom.COINIT_MULTITHREADED)
# com calls here for x
pythoncom.CoUninitialize()
Hope this helps.

OK, so I think I found an answer but I am not yet sure why it works..
If I cut and paste this line import win32com.client from the top of the page right inside the try block where I dispatch microsoft powerpoint, the app works successfully.
However, I still can't find out why.

There are at least two more ways to solve the issue:
Use run() method instead of start(), i.e. test.run()
Before myObject = win32com.client.Dispatch("Powerpoint.Application") insert the following lines: import pythoncom; CoInitialize()
Notice that using run() instead of start() has been tested in other scripts and it always worked for me!

How to set a timeout in libvirt (using Python)

I would like to set a short connection timeout (only some seconds) when using libvirt in my Python program instead of the long default one.
I found the C function: virEventAddTimeoutFunc() in the C libvirt API here:
http://libvirt.org/html/libvirt-libvirt.html#virEventAddTimeoutFunc
and eventInvokeTimeoutCallback(timer, callback, opaque) in libvirt.py around the line#150 but I do not know how to use it. I did not find any example on the net.
I tried this but I get a segmentation fault: :-(
import libvirt
def timeout_cb_d():
print 'Timeout !'
try:
# try to set the libvirt timeout to 2 seconds:
t = libvirt.eventInvokeTimeoutCallback(2, timeout_cb_d, "from dom0_class")
except:
...
Does anyone can give me a working example please?

We finally found a simple way to proceed using Python alarm & signal handler: http://docs.python.org/library/signal.html#example
Edit:
Here is the idea:
import string, time, sys, signal
class Host:
# (...)
def timeout_handler(self, sig_code, frame):
if 14 == sig_code: sig_code = 'SIGALRM'
print time.strftime('%F %T -'), 'Signal handler called with signal:', sig_code
raise Exception('Timeout!')
def libVirtConnect(self):
try:
# Enable the timeout with an alarm:
signal.signal(signal.SIGALRM, self.timeout_handler)
signal.alarm(self._libvirt_timeout_in_seconds)
self._virt_conn = libvirt.open('xen+tcp://'+self._ip)
signal.alarm(0) # Disable the alarm
except Exception, e:
signal.alarm(0) # Disable the alarm

I'm assuming libvirt communicates over a standard socket. If that's the case, you can set an application-wide timeout using socket.settimeout.
That's not to say the libvirt bindings for python don't call that function themselves, but it's worth a try.

I have often used monkeypatching to change a library so that sockets timeout. Usually you just need to find the method which calls select or poll and monkeypatch in a modified version. Sometimes you need to set up a try-catch which catches socket.timeout and does something to allow it to percolate up to your code without causing another error on the way. In one case I had to create a valid response object instead of None, for instance.