I am trying to establish a long running Pull subscription to a Google Cloud PubSub topic.
I am using a code very similar to the example given in the documentation here, i.e.:
def receive_messages(project, subscription_name):
"""Receives messages from a pull subscription."""
subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(
project, subscription_name)
def callback(message):
print('Received message: {}'.format(message))
message.ack()
subscriber.subscribe(subscription_path, callback=callback)
# The subscriber is non-blocking, so we must keep the main thread from
# exiting to allow it to process messages in the background.
print('Listening for messages on {}'.format(subscription_path))
while True:
time.sleep(60)
The problem is that I'm receiving the following traceback sometimes:
Exception in thread Consumer helper: consume bidirectional stream:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/path/to/google/cloud/pubsub_v1/subscriber/_consumer.py", line 248, in _blocking_consume
self._policy.on_exception(exc)
File "/path/to/google/cloud/pubsub_v1/subscriber/policy/thread.py", line 135, in on_exception
raise exception
File "/path/to/google/cloud/pubsub_v1/subscriber/_consumer.py", line 234, in _blocking_consume
for response in response_generator:
File "/path/to/grpc/_channel.py", line 348, in __next__
return self._next()
File "/path/to/grpc/_channel.py", line 342, in _next
raise self
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, The service was unable to fulfill your request. Please try again. [code=8a75])>
I saw that this was referenced in another question but here I am asking to how to handle it properly in Python. I have tried to wrap the request in an exception but it seems to run in the background and I am not able to retry in case of that error.
A somewhat hacky approach that is working for me is a custom policy_class. The default one has an on_exception function that ignores DEADLINE_EXCEEDED. You can make a class that inherits the default and also ignores UNAVAILABLE. Mine looks like this:
from google.cloud import pubsub
from google.cloud.pubsub_v1.subscriber.policy import thread
import grpc
class AvailablePolicy(thread.Policy):
def on_exception(self, exception):
"""The parent ignores DEADLINE_EXCEEDED. Let's also ignore UNAVAILABLE.
I'm not sure what triggers that error, but if you ignore it, your
subscriber seems to work just fine. It's probably an intermittent
thing and it reconnects later if you just give it a chance.
"""
# If this is UNAVAILABLE, then we want to retry.
# That entails just returning None.
unavailable = grpc.StatusCode.UNAVAILABLE
if getattr(exception, 'code', lambda: None)() == unavailable:
return
# For anything else, fallback on super.
super(AvailablePolicy, self).on_exception(exception)
subscriber = pubsub.SubscriberClient(policy_class=AvailablePolicy)
# Continue to set up as normal.
It looks a lot like the original on_exception just ignores a different error. If you want, you can add some logging whenever the exception is thrown and verify that everything still works. Future messages will still come through.
Related
I'm programming a telegram bot in python that, on command, starts sending a photo of a window on my computer to the chat every few seconds. Of course I don't want the chat to get spammed with photos so instead I want to go back and edit the first message that I send using edit_media.
Currently my callback function to the command looks like this:
def callback(update: Update, context: CallbackContext):
first = True
while True:
image = screenshot('Snake') # Gets a screenshot of the window
bytesio = io.BytesIO()
image.save(bytesio, format='PNG')
bytes_image = bytes(bytesio.getvalue()) # Convert to bytes object
if first: # First time send a full message
context.bot.send_photo(update.message.chat_id, bytes_image)
first = False
else:
update.message.edit_media(media=InputMediaPhoto(media=bytes_image)) # Edit the message with the next photo
time.sleep(2)
When this function runs, the first message is sent successfully, but trying to edit the message raises an error:
telegram.error.BadRequest: Message can't be edited
How do I solve this so that the image is successfully edited?
The full traceback of the error is:
Traceback (most recent call last):
File "C:\Users\...\telegram\ext\dispatcher.py", line 555, in process_update
handler.handle_update(update, self, check, context)
File "C:\Users\...\telegram\ext\handler.py", line 198, in handle_update
return self.callback(update, context)
File "C:\Users\...\TelegramGameHub\main.py", line 56, in snake
update.message.edit_media(media=InputMediaPhoto(media=bytes_image))
File "C:\Users\...\telegram\message.py", line 2016, in edit_media
return self.bot.edit_message_media(
File "C:\Users\...\telegram\bot.py", line 130, in decorator
result = func(*args, **kwargs)
File "C:\Users\...\telegram\bot.py", line 2723, in edit_message_media
return self._message(
File "C:\Users\...\telegram\ext\extbot.py", line 199, in _message
result = super()._message(
File "C:\Users\...\telegram\bot.py", line 332, in _message
result = self._post(endpoint, data, timeout=timeout, api_kwargs=api_kwargs)
File "C:\Users\...\telegram\bot.py", line 295, in _post
return self.request.post(
File "C:\Users\...\telegram\utils\request.py", line 354, in post
result = self._request_wrapper('POST', url, fields=data, **urlopen_kwargs)
File "C:\Users\...\telegram\utils\request.py", line 279, in _request_wrapper
raise BadRequest(message)
The problem is that your line
update.message.edit_media(media=InputMediaPhoto(media=bytes_image))
doesn't edit the photo message that you sent. It instead tries to edit the message that initially triggered the callback.
To get a hold of the message that you send, you'll need to do something like
message = context.bot.send_photo(update.message.chat_id, bytes_image)
Note that PTB provides shortcuts for these kind of things. So you could abbreviate this to
message = update.message.reply_photo(bytes_image)
which is (almost) equivalent.
Now you can call message.edit_media(...) to update the photo.
Another thing to note is that your loop
while True:
...
time.sleep(2)
is blocking. That means that after entering the function callback, your bot won't process any other updates. (Side note in case you know what run_async does: Even if you run this callback asynchronously, it will block one of the worker threads and as soon as enough updates have triggered this callback, all the worker threads are blocked and your bot won't process any more updates)
I highly recommend to use the built in JobQueue for such things. See also the wiki page and the example.
Disclaimer: I'm currently the maintainer of python-telegram-bot.
Fully loaded multi-tenant Django application with 1000's of WebSockets using Daphne/Channels, running fine for a few months and suddenly tenants all calling it the support line the application running slow or outright hanging. Narrowed it down to WebSockets as HTTP REST API hits came through fast and error free.
None of the application logs or OS logs indicate some issue, so only thing to go on is the exception noted below. It happened over and over again here and there throughout 2 days.
I don't expect any deep debugging help, just some off-the-cuff advice on possibilities.
AWS Linux 1
Python 3.6.4
Elasticache Redis 5.0
channels==2.4.0
channels-redis==2.4.2
daphne==2.5.0
Django==2.2.13
Split configuration HTTP served by uwsgi, daphne serves asgi, Nginx
May 10 08:08:16 prod-b-web1: [pid 15053] [version 119.5.10.5086] [tenant_id -] [domain_name -] [pathname /opt/releases/r119.5.10.5086/env/lib/python3.6/site-packages/daphne/server.py] [lineno 288] [priority ERROR] [funcname application_checker] [request_path -] [request_method -] [request_data -] [request_user -] [request_stack -] Exception inside application: Lock is not acquired.
Traceback (most recent call last):
File "/opt/releases/r119.5.10.5086/env/lib/python3.6/site-packages/channels_redis/core.py", line 435, in receive
real_channel
File "/opt/releases/r119.5.10.5086/env/lib/python3.6/site-packages/channels_redis/core.py", line 484, in receive_single
await self.receive_clean_locks.acquire(channel_key)
File "/opt/releases/r119.5.10.5086/env/lib/python3.6/site-packages/channels_redis/core.py", line 152, in acquire
return await self.locks[channel].acquire()
File "/opt/python3.6/lib/python3.6/asyncio/locks.py", line 176, in acquire
yield from fut
concurrent.futures._base.CancelledError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/releases/r119.5.10.5086/env/lib/python3.6/site-packages/channels/sessions.py", line 183, in __call__
return await self.inner(receive, self.send)
File "/opt/releases/r119.5.10.5086/env/lib/python3.6/site-packages/channels/middleware.py", line 41, in coroutine_call
await inner_instance(receive, send)
File "/opt/releases/r119.5.10.5086/env/lib/python3.6/site-packages/channels/consumer.py", line 59, in __call__
[receive, self.channel_receive], self.dispatch
File "/opt/releases/r119.5.10.5086/env/lib/python3.6/site-packages/channels/utils.py", line 58, in await_many_dispatch
await task
File "/opt/releases/r119.5.10.5086/env/lib/python3.6/site-packages/channels_redis/core.py", line 447, in receive
self.receive_lock.release()
File "/opt/python3.6/lib/python3.6/asyncio/locks.py", line 201, in release
raise RuntimeError('Lock is not acquired.')
RuntimeError: Lock is not acquired.
First, lets have a look at the source of the RuntimeError: Lock is not acquired. error. As given by the traceback, the release() method in the file /opt/python3.6/lib/python3.6/asyncio/locks.py is defined like so:
def release(self):
"""Release a lock.
When the lock is locked, reset it to unlocked, and return.
If any other coroutines are blocked waiting for the lock to become
unlocked, allow exactly one of them to proceed.
When invoked on an unlocked lock, a RuntimeError is raised.
There is no return value.
"""
if self._locked:
self._locked = False
self._wake_up_first()
else:
raise RuntimeError('Lock is not acquired.')
A primitive lock is a synchronization primitive that is not owned by a particular thread when locked.
When attempting to release an unlocked lock by calling the release() method, the RuntimeError will be raised, as the method should only be called in the locked state. The state changes to unlocked when called in the locked state.
Now for the previous error raised in the acquire() method in the same file, the acquire() method is defined like so:
async def acquire(self):
"""Acquire a lock.
This method blocks until the lock is unlocked, then sets it to
locked and returns True.
"""
if (not self._locked and (self._waiters is None or
all(w.cancelled() for w in self._waiters))):
self._locked = True
return True
if self._waiters is None:
self._waiters = collections.deque()
fut = self._loop.create_future()
self._waiters.append(fut)
# Finally block should be called before the CancelledError
# handling as we don't want CancelledError to call
# _wake_up_first() and attempt to wake up itself.
try:
try:
await fut
finally:
self._waiters.remove(fut)
except exceptions.CancelledError:
if not self._locked:
self._wake_up_first()
raise
self._locked = True
return True
So in order for the concurrent.futures._base.CancelledError error you're getting to be raised, the await fut must've caused the issue.
To fix it, you can have a look at Awaiting an asyncio.Future raises concurrent.futures._base.CancelledError instead of waiting for a value/exception to be set
Basically, you might have an awaitable in your code that you didn't await, and by not awaiting it, you never handed control back to the event loop or store the awaitable, causing it to be immediately cleaned up, completely cancelling it (and all of the awaitables it controlled).
Simply make sure you await the results of the awaitables in your code, finding any you missed.
I am trying to get ticks result from the kiteconnect API using KiteTicker. I have used the code given in the documentation
import logging
from kiteconnect import KiteTicker
logging.basicConfig(level=logging.DEBUG)
# Initialise
kws = KiteTicker("my_api", "my_access_token")
def on_ticks(ws, ticks):
# Callback to receive ticks.
print(ticks)
def on_connect(ws, response):
# Callback on successful connect.
# Subscribe to a list of instrument_tokens (RELIANCE and ACC here).
ws.subscribe([738561, 5633])
# Set RELIANCE to tick in `full` mode.
ws.set_mode(ws.MODE_FULL, [738561])
def on_close(ws, code, reason):
# On connection close stop the event loop.
# Reconnection will not happen after executing `ws.stop()`
ws.stop()
# Assign the callbacks.
kws.on_ticks = on_ticks
kws.on_connect = on_connect
kws.on_close = on_close
# Infinite loop on the main thread. Nothing after this will run.
# You have to use the pre-defined callbacks to manage subscriptions.
kws.connect()
It ran successfully for the first time but not it's showing below error:
2020-06-26 20:09:26+0530 [-] Starting factory <kiteconnect.ticker.KiteTickerClientFactory object at 0x000001DE88068A88>
DEBUG:kiteconnect.ticker:Start WebSocket connection.
Traceback (most recent call last):
File "<ipython-input-52-9b59ab5163b2>", line 31, in <module>
kws.connect()
File "C:\Users\yp229\anaconda3\lib\site-packages\kiteconnect\ticker.py", line 532, in connect
reactor.run(**opts)
File "C:\Users\yp229\anaconda3\lib\site-packages\twisted\internet\base.py", line 1282, in run
self.startRunning(installSignalHandlers=installSignalHandlers)
File "C:\Users\yp229\anaconda3\lib\site-packages\twisted\internet\base.py", line 1262, in startRunning
ReactorBase.startRunning(self)
File "C:\Users\yp229\anaconda3\lib\site-packages\twisted\internet\base.py", line 765, in startRunning
raise error.ReactorNotRestartable()
ReactorNotRestartable
I am not able to find the reasoning behind the same and also unable to get the log file :| . I was trying to achieve below output:
List of kite ticks (I wanted to store it in the data frame but the printing it self giving the error)
Applying certain condition on ticks result
Any help would be really useful.
I'm using the Firebase Realtime Database listener to listen to changes on a database path.
My program recently crashed because of the following 503 error that seems to be raised by the underlying requests library:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.7/site-packages/firebase_admin/db.py", line 123, in _start_listen
for sse_event in self._sse:
File "/usr/local/lib/python3.7/site-packages/firebase_admin/_sseclient.py", line 128, in __next__
self._connect()
File "/usr/local/lib/python3.7/site-packages/firebase_admin/_sseclient.py", line 112, in _connect
self.resp.raise_for_status()
File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 940, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 503 Server Error: Service Unavailable for url: https://database_url...
My listener initialization is wrapped in a try statement, so I'm unsure why this wasn't caught, swallowed and retried as I expected it to:
def init_listener():
try:
listener = firebase_admin.db.reference(db_path).listen(handle_change)
except Exception as e:
time.sleep(1) # Retry in one second.
init_listener()
I'd like to handle future 503 errors, but I'm not sure how to go about doing this.
Additionally, I'm using except Exception as e above for demo/debugging purposes, but I'm also not sure if requests.exceptions.HTTPError will be specific enough to catch only 500 errors (though I don't know what other errors can be raised).
From the firebase_admin reference docs:
This API is based on the event streaming support available in the
Firebase REST API. Each call to listen() starts a new HTTP connection
and a background thread. This is an experimental feature.
The key here is that this all runs in a background thread. Therefore, wrapping the call to listen() in a try/except will not catch exceptions thrown in the thread. There is no simple way to catch the exceptions happening in the background thread.
To solve your issue, you will probably need to know more about why the database is returning an HTTP 503 status. Or you will need to switch to some other firebase_admin API that will allow you to catch and ignore these exceptions.
At the moment i am writing a syslog client that will send messages to a remote syslog server. So far this is working pretty ok but i am running into the following problem.
When the syslog server goes down for some reason i need to catch this so the program will stop sending syslog messages and we can investigate the problem.
Unfortunately, the program continues running and doesn't see that the TCP socket is closed and raise an exception.
I only receive a traceback in my terminal:
--- Logging error --- Traceback (most recent call last):
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\logging\handlers.py", line 941, in emit
self.socket.sendall(msg) ConnectionAbortedError: [WinError 10053] Call stack:
File "c:\Users\Administrator\.vscode\extensions\ms-python.python-2018.12.1\pythonFiles\ptvsd_launcher.py", line 45, in <module>
main(ptvsdArgs)
File "c:\Users\Administrator\.vscode\extensions\ms-python.python-2018.12.1\pythonFiles\lib\python\ptvsd\__main__.py", line 265, in main
wait=args.wait)
File "c:\Users\Administrator\.vscode\extensions\ms-python.python-2018.12.1\pythonFiles\lib\python\ptvsd\__main__.py", line 258, in handle_args
debug_main(addr, name, kind, *extra, **kwargs)
File "c:\Users\Administrator\.vscode\extensions\ms-python.python-2018.12.1\pythonFiles\lib\python\ptvsd\_local.py", line 45, in debug_main
run_file(address, name, *extra, **kwargs)
File "c:\Users\Administrator\.vscode\extensions\ms-python.python-2018.12.1\pythonFiles\lib\python\ptvsd\_local.py", line 79, in run_file
run(argv, addr, **kwargs)
File "c:\Users\Administrator\.vscode\extensions\ms-python.python-2018.12.1\pythonFiles\lib\python\ptvsd\_local.py", line 140, in _run
_pydevd.main()
File "c:\Users\Administrator\.vscode\extensions\ms-python.python-2018.12.1\pythonFiles\lib\python\ptvsd\_vendored\pydevd\pydevd.py", line 1925, in main
debugger.connect(host, port)
File "c:\Users\Administrator\.vscode\extensions\ms-python.python-2018.12.1\pythonFiles\lib\python\ptvsd\_vendored\pydevd\pydevd.py", line 1283, in run
return self._exec(is_module, entry_point_fn, module_name, file, globals, locals)
File "c:\Users\Administrator\.vscode\extensions\ms-python.python-2018.12.1\pythonFiles\lib\python\ptvsd\_vendored\pydevd\pydevd.py", line 1290, in _exec
pydev_imports.execfile(file, globals, locals) # execute the script
File "c:\Users\Administrator\.vscode\extensions\ms-python.python-2018.12.1\pythonFiles\lib\python\ptvsd\_vendored\pydevd\_pydev_imps\_pydev_execfile.py", line 25, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "c:\Users\Administrator\OneDrive\Documents\Python Scripts\testlogger.py", line 71, in <module>
my_logger.info(i) Message: 'test4' Arguments: ()
Relevant code:
my_logger = logging.getLogger('MyLogger')
my_logger.setLevel(logging.ERROR)
my_logger.setLevel(logging.INFO)
my_logger.setLevel(logging.DEBUG)
try:
handler = logging.handlers.SysLogHandler(('IP ADDRESS HOST', 514), socktype=socket.SOCK_STREAM)
my_logger.addHandler(handler)
except Exception as e:
print (e)
list1 = ['test','test2','test3','test4','test5','test6','test7','test8']
for i in list1:
try:
my_logger.info(i) #here i expected that an exception would be raised when the TCP socket is not alive anymore
except Exception as e:
print (e)
How can i make sure that the program stops and i can do the appropriate exception handling?
Thanks!
The default behavior of the SysLogHandler class (and all the ones who are using TCP) is to retry a connection, this is explained in the docs of the createSocket() method:
Tries to create a socket; on failure, uses an exponential back-off
algorithm. On initial failure, the handler will drop the message it
was trying to send. When subsequent messages are handled by the same
instance, it will not try connecting until some time has passed. The
default parameters are such that the initial delay is one second, and
if after that delay the connection still can’t be made, the handler
will double the delay each time up to a maximum of 30 seconds.
This behaviour is controlled by the following handler attributes:
retryStart (initial delay, defaulting to 1.0 seconds).
retryFactor (multiplier, defaulting to 2.0).
retryMax (maximum delay, defaulting to 30.0 seconds).
As it doesn't seem to have an option for the behavior of "not retrying" which you seem to want, so if you really want that you can create your own handler by subclassing it and overriding the createSocket() method with something like:
class MySysLogHandler(logging.handlers.SysLogHandler):
def createSocket(self):
try:
self.sock = self.makeSocket()
except OSError:
# do your own error handling here ...
You can dig a bit more by looking at the source code of createSocket() in CPython Github repo (beware, this is from master branch and might not be the exact version of Python you're using)
I switched to another solution and stopped using the SyslogHandler class.
I now use the following class were i wrote my own syslog sender through a socket.
class Syslog:
def __init__(self,host="localhost",port=514,facility=Facility.DAEMON):
self.host = host
self.port = port
self.facility = facility
self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
def connect(self):
try:
self.socket.connect((self.host, self.port))
except Exception as e:
print("failed setting up connection")
def send(self, message,level):
data = "<%d>%s" % (level + self.facility*8, message + "\n")
try:
self.socket.sendall(data.encode('utf-8'))
except Exception as e:
print("send failed")
#if __name__ == '__main__':
syslog1 = Syslog(host='HOSTIPADDRESS-NAME')
syslog1.connect()
messages = ["test1","test2","test3","test4","test5","test6","test7"]
for message in messages:
syslog1.send(message,Level.WARNING)
This is working quite well and runs into an exception when the syslog server goes down unexpectedly. The only problem now i discovered while debugging is the following:
When i shut down the syslog server it throws not immediatly an exception when i try to send a message.
Please see example below:
1.) the syslog server is started, i send the first message "test1" from the for loop, successfull.
2.) i shutdown the syslog server, now i send the second message "test2" from the for loop. Nothing happens, no exception!
3.) i send the third message "test3", now an exception is thrown.
How is this possible?