I've just got started using pika(v 0.9.4) with Tornado (through the use of pika.adapters.tornado_connection.TornadoConnection) and I was wondering what's the appropriate way of catching errors when using, say: queue_delete for when the queue you're trying to delete doesn't exist. RabbitMQ raises AMQPError, but I am not sure how this can be handled in an async way.
Anyone has any insights on this ?
Disclaimer: I'm the author of stormed-amqp
I would suggest trying with stormed-amqp
import logging
logging.basicConfig()
from tornado.ioloop import IOLoop
from stormed import Connection
def on_connect():
ch = conn.channel()
ch.queue_declare(queue='hello', durable=False)
ch.queue_declare(queue='hello', durable=True)
def on_error(e):
print "Got Connection error", e.reply_text, e.reply_code
io_loop.stop()
conn = Connection(host='localhost')
conn.on_error = on_error
conn.connect(on_connect)
io_loop = IOLoop.instance()
io_loop.start()
Try to avoid the error. If you declare a connection to the queue, and it doesn't exist, it will be created. Then immediately delete it.
Or, if you will use that queue again in the next week or so, i.e. it is not single-use, then just leave it around and handle deletion as a system admin activity that cleans up long term idle queues.
Or just declare your queues with the auto-delete attribute and they will go away when you disconnect.
Related
I'm creating a script that is posting a message to both discord and twitter, depending on some input. I have to methods (in separate .py files), post_to_twitter and post_to_discord. What I want to achieve is that both of these try to execute even if the other fails (e.g. if there is some exception with login).
Here is the relevant code snippet for posting to discord:
def post_to_discord(message, channel_name):
client = discord.Client()
#client.event
async def on_ready():
channel = # getting the right channel
await channel.send(message)
await client.close()
client.run(discord_config.token)
and here is the snippet for posting to twitter part (stripped from the try-except blocks):
def post_to_twitter(message):
auth = tweepy.OAuthHandler(twitter_config.api_key, twitter_config.api_key_secret)
auth.set_access_token(twitter_config.access_token, twitter_config.access_token_secret)
api = tweepy.API(auth)
api.update_status(message)
Now, both of these work perfectly fine on their own and when being called synchronously from the same method:
def main(message):
post_discord.post_to_discord(message)
post_tweet.post_to_twitter(message)
However, I just cannot get them to work concurrently (i.e. to try to post to twitter even if discord fails or vice-versa). I've already tried a couple of different approaches with multi-threading and with asyncio.
Among others, I've tried the solution from this question. But got an error No module named 'IPython'. When I omitted the IPython line, changed the methods to async, I got this error: RuntimeError: Cannot enter into task <ClientEventTask state=pending event=on_ready coro=<function post_to_discord.<locals>.on_ready at 0x7f0ee33e9550>> while another task <Task pending name='Task-1' coro=<main() running at post_main.py:31>> is being executed..
To be honest, I'm not even sure if asyncio would be the right approach for my use case, so any insight is much appreciated.
Thank you.
In this case running the two things in completely separate threads (and completely separate event loops) is probably the easiest option at your level of expertise. For example, try this:
import post_to_discord, post_to_twitter
import concurrent.futures
def main(message):
with concurrent.futures.ThreadPoolExecutor() as pool:
fut1 = pool.submit(post_discord.post_to_discord, message)
fut2 = pool.submit(post_tweet.post_to_twitter, message)
# here closing the threadpool will wait for both futures to complete
# make exceptions visible
for fut in (fut1, fut2):
try:
fut.result()
except Exception as e:
print("error: ", e)
Hi all I have the following code but for some reason I keep getting the following error but it seems to work on a colleagues pc. We can't seem to figure out why this won't work on mine.
We have also double checked that we're importing the same socketio using dir()
I've tried specifying the namespace both on sio.connect and in the sio.emit but still no luck!
socketio.exceptions.BadNamespaceError: / is not a connected namespace.
bearerToken = 'REDACT'
core = 'REDACT'
output = 'REDACT'
import socketio
import json
def getListeners(token, coreUrl, outputId):
sio = socketio.Client(reconnection_attempts=5, request_timeout=5)
sio.connect(url=coreUrl, transports='websocket')
#sio.on('mwedge:batch:stats')
def batchStats(data):
if (outputId in data['outputStats']):
listeners = data['outputStats'][outputId][16]
print("Number of listeners ", len(listeners))
ips = []
for listener in listeners:
ips.append(listener[1])
print("Ips", ips)
def authCallback(data):
print(json.dumps(data))
sio.emit(event='auth',
data={
'token': token
},
callback=authCallback)
getListeners(bearerToken, core, output)
The Socket.IO connection involves a number of exchanges between the client and the server. The connect() function initiates this process, but this continues in the background. The connection ends when the handler for your connect event is invoked. At this point you can emit.
The problem with your code is that you are not waiting until the connection handshakes are completed, so your emit() call happens before there is a connection established. The solution is to add a connect event handler, and move your emit() call there.
As an additional note, I suggest you set up your event handlers before you call the connect() function.
I need to create a function that checks to make sure Mongo servers are running using the ping function. I set up the clients right there (the config file has dictionary with ports numbers)
clientList = []
for value in configuration["mongodbServer"]:
client = motor.motor_tornado.MotorClient('mongodb://localhost:{}'.format(value))
clientList.append(client)
and then i run this function:
class MongoChecker(Checker):
formatter = 'stashboard.formatters.MongoFormatter'
def check(self):
for x in clientList:
if x.ping:
return x.ping
and the error i get:
yielded unknown object MotorDatabase(Database(MongoClient([]), 'ping'))\n",
I think my issue is that i'm using the ping function wrong. I can't find any other documentation on that or any other kind of feature that would check to see if the servers are still running. If anyone knows of a better way to monitor the status using Motor, i'm open. Thanks!
First, there's no "ping" function. Hence MotorClient thinks you're trying to access the database named "ping". The database named "ping" is shown in the "unknown object" exception. For all MongoDB commands like "ping", just use MotorDatabase's command method.
Second, Motor is asynchronous. You must use Motor methods in a Tornado coroutine with the "yield" statement. For example:
#gen.coroutine
def check():
try:
result = yield client.admin.command({'ping': 1})
print(result)
except ConnectionFailure as exc:
print(exc)
If you want to test this out synchronously, you can run the IOLoop just long enough for the coroutine to complete:
from pymongo.errors import ConnectionFailure
from tornado import gen
from tornado.ioloop import IOLoop
import motor.motor_tornado
client = motor.motor_tornado.MotorClient()
IOLoop.current().run_sync(check)
For an introduction to Tornado coroutines, see Refactoring Tornado Coroutines and the Tornado documentation.
i write a Tcp Server with Tornado.
here is the code:
#! /usr/bin/env python
#coding=utf-8
from tornado.tcpserver import TCPServer
from tornado.ioloop import IOLoop
from tornado.gen import *
class TcpConnection(object):
def __init__(self,stream,address):
self._stream=stream
self._address=address
self._stream.set_close_callback(self.on_close)
self.send_messages()
def send_messages(self):
self.send_message(b'hello \n')
print("next")
self.read_message()
self.send_message(b'world \n')
self.read_message()
def read_message(self):
self._stream.read_until(b'\n',self.handle_message)
def handle_message(self,data):
print(data)
def send_message(self,data):
self._stream.write(data)
def on_close(self):
print("the monitored %d has left",self._address)
class MonitorServer(TCPServer):
def handle_stream(self,stream,address):
print("new connection",address,stream)
conn = TcpConnection(stream,address)
if __name__=='__main__':
print('server start .....')
server=MonitorServer()
server.listen(20000)
IOLoop.instance().start()
And i face some eorror assert self._read_callback is None, "Already reading",i guess the eorror is because multiple commands to read from socket at the same time.and then i change the function send_messages with tornado.gen.coroutine.here is code:
#gen.coroutine
def send_messages(self):
yield self.send_message(b'hello \n')
response1 = yield self.read_message()
print(response1)
yield self.send_message(b'world \n')
print((yield self.read_message()))
but there are some other errors. the code seem to stop after yield self.send_message(b'hello \n'),and the following code seem not to execute.
how should i do about it ? If you're aware of any Tornado tcpserver (not HTTP!) code with tornado.gen.coroutine,please tell me.I would appreciate any links!
send_messages() calls send_message() and read_message() with yield, but these methods are not coroutines, so this will raise an exception.
The reason you're not seeing the exception is that you called send_messages() without yielding it, so the exception has nowhere to go (the garbage collector should eventually notice and print the exception, but that can take a long time). Whenever you call a coroutine, you should either use yield to wait for it to finish, or IOLoop.current().spawn_callback() to run the coroutine in the "background" (this tells Tornado that you do not intend to yield the coroutine, so it will print the exception as soon as it occurs). Also, whenever you override a method you should read the documentation to see whether coroutines are allowed (when you override TCPServer.handle_stream() you can make it a coroutine, but __init__() may not be a coroutine).
Once the exception is getting logged, the next step is to fix it. You can either make send_message() and read_message() coroutines (getting rid of the handle_message() callback in the process), or you can use tornado.gen.Task() to call coroutine-style code from a coroutine. I generally recommend using coroutines everywhere.
Related to this question I have the following code which subscribes to a redis pubsub queue and uses the handler provided in __init__ to feed the messages to the class that processes them:
from threading import Thread
import msgpack
class Subscriber(Thread):
def __init__(self, redis_connection, channel_name, handler):
super(Subscriber, self).__init__(name="Receiver")
self.connection = redis_connection
self.pubsub = self.connection.pubsub()
self.channel_name = channel_name
self.handler = handler
self.should_die = False
def start(self):
self.pubsub.subscribe(self.channel_name)
super(Subscriber, self).start()
def run(self):
for msg in self.pubsub.listen():
if self.should_die:
return
try:
data = msg["data"]
unpacked = msgpack.unpackb(data)
except TypeError:
# stop non-msgpacked, invalid, messages breaking stuff
# other validation happens in handler
continue
self.handler(unpacked)
def die(self):
self.should_die = True
In the linked question above, it is noted that pubsub.listen() never returns if the connection is dropped. Therefore, my die() function, while it can be called, will never actually cause the thread to terminate because it is hanging on the call to listen() inside the thread's run().
The accepted answer on the linked question mentions hacking redis-py's connection pool. I really don't want to do this and have a forked version of redis-py (at least until the fix is hopefully accepted into master), but I've had a look at the redis-py code anyway and don't immediately see where this change would be made.
Does anyone have an idea how to cleanly solve the hanging redis-py listen() call?
What issues will I incur by directly using Thread._Thread__stop?
To close this out after so many years. It ended up being a bug in the redis library. I debugged it and submitted the PR. It shouldn't occur any more.