We Keep Coding

Confluent-Kafka Python - Describe consumer groups (to get the lag of each consumer group)

I want to get the details of the consumer group using confluent-kafka. The cli equivalent of that is
`
./kafka-consumer-groups.sh --bootstrap-server XXXXXXXXX:9092 --describe --group my-group
My end goal is to get the value of lag from the output. Is there any method in confluent-kafka python API to get these details. There is a method in the java API but I couldn't find it in python API.
I tried using the describe_configs method in the adminClient API but it ended up throwing kafkaException with following details
This most likely occurs because of a request being malformed by the client library or the message was sent to an incompatible broker. See the broker logs for more details.

For now I have come up with the following solution. It's a work around to get the combined lag of a consumer group
def get_lag(topic,numPartitions):
diff = list()
for i in range(numPartitions):
topic_partition = TopicPartition(topic, partition=i)
low, high = consumer.get_watermark_offsets(topic_partition)
currentList = consumer.committed([topic_partition])
current = currentList[0].offset
diff.append(high-current)
return sum(diff) # Combined Lag

How to add a timeout to method start_consuming() on pika library

I have a BlockingConnection, and I follow the examples of pika documentation. But in all of them, the example of code to start consuming messages are:
connection = pika.BlockingConnection()
channel = connection.channel()
channel.basic_consume('test', on_message)
try:
channel.start_consuming()
except KeyboardInterrupt:
channel.stop_consuming()
connection.close()
(with more or less details).
I have to code many scripts, and I want to run one after another (for test/research purposes). But the above code require that I added ^C in each one.
I try to add some timeouts explained in the documentation, but I haven't luck. For example, if I find a parameter for set if client don't consuming any message in the last X seconds, then script finish. Is this posible in pika lib? or I have to change the approach?

Don't use start_consuming if you don't want your code to block. Either use SelectConnection or this method that uses consume. You can add a timeout to the parameters passed to consume.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.

import pika
parameters = pika.ConnectionParameters(host="localhost")
connection = pika.BlockingConnection(parameters)
channel = connection.channel()
def ack_message(channel, method):
"""Note that `channel` must be the same pika channel instance via which
the message being ACKed was retrieved (AMQP protocol constraint).
"""
if channel.is_open:
channel.basic_ack(method.delivery_tag)
else:
# Channel is already closed, so we can't ACK this message;
# log and/or do something that makes sense for your app in this case.
pass
def callback(channel,method, properties, body):
ack_message(channel,method)
print("body",body, flush=True)
channel.basic_consume(
queue="hello", on_message_callback=callback)
channel.start_consuming()
connection.close()
I the original code is the answer of Luke Bakken.
But I have edited the code a lil bit.
:)

It's too late but perhaps someone gets benefited from that. You can use blocked_connection_timeout argument in pika.ConnectionParameters() as follows,
connection = pika.BlockingConnection(
pika.ConnectionParameters(
heartbeat=600,
blocked_connection_timeout=600,
host=self.queue_host,
port=constants.RABBTIMQ_PORT,
virtual_host=self.rabbitmq_virtual_host,
credentials=pika.PlainCredentials(
username=self.rabbitmq_username,
password=self.rabbitmq_password
)
)
)

Message type websocket gdax (coinbase)

I am interested in getting real-time data using the Gdax (Coinbase) WebSocket. I'm a total noob so I am inspecting the example Gdax posted in their documentation:
import gdax, time
class myWebsocketClient(gdax.WebsocketClient):
def on_open(self):
self.url = "wss://ws-feed.gdax.com/"
self.products = ["LTC-USD"]
self.message_count = 0
print("Lets count the messages!")
def on_message(self, msg):
self.message_count += 1
if 'price' in msg and 'type' in msg:
print ("Message type:", msg["type"],
"\t# {}.3f".format(float(msg["price"])))
def on_close(self):
print("-- Goodbye! --")
wsClient = myWebsocketClient()
wsClient.start()
print(wsClient.url, wsClient.products)
while (wsClient.message_count < 500):
print ("\nmessage_count =", "{} \n".format(wsClient.message_count))
time.sleep(1)
wsClient.close()
The output is:
...
Message type: received # 50.78.3f
Message type: open # 50.78.3f
Message type: done # 51.56.3f
Message type: received # 51.59.3f
Message type: open # 51.59.3f
Message type: done # 51.51.3f
Message type: done # 51.17.3f
Message type: done # 51.66.3f
Kernel died, restarting
I have a few question regarding this code and output:
What does the message type (received, open, done, match) mean, which type is used for doing calculations, and why are some types skipped?
Why does running the code always ends in 'Kernel died, restarting'?
The documentation states that this code is for illustration only. Does that mean that this isn't a proper way of getting real-time data in order to do stuff with it?
If you know some good articles or books that can teach a noob how to work with WebSockets, I would love to hear about them!

1) see the full documentation for each message here
2) Everything I find related to that issue stems from environment set up. Whether that is library dependencies not being properly installed or other environmental factors.
3) It's a proper way to set up a connection to the WebSocket but they don't provide any error handling or other logic. It's usually to cover themselves legally and reduce expectations for the code they provide (i.e. when someone receives error's similar to this they aren't liable to fix, update, help, etc)

Python interpreter (3.6.2 64bit win) crashed on close() for me too.
Here is a fix (from https://github.com/danpaquin/gdax-python/issues/152):
client.stop = True
client.thread.join()
client.ws.close()
I just added these in the on_close method and no more crashes (so far).
ps: the issue linked say that it should be fixed in the latest gdax version but the latest pip gdax (1.0.6) still crashed.

There are rather a few pitfalls to be aware of in building a full, real time order book (level 3) that I cannot document here but you may be interested to learn GDAX now offer a level 2 (almost real-time) update which will send you the updated prices for the order book. Probably much simpler to implement.

Redis publish - Wrong number of argument for 'set'

I'm trying to use websockets with Django for small parts of my application.
Trying the first example to broadcast a message with django-websocket-redis
from ws4redis.publisher import RedisPublisher
redis_publisher = RedisPublisher(facility='foobar', broadcast=True)
redis_publisher.publish_message('Hello World')
I'm actually receiving the message into subscribed clients but I'm getting this error:
wrong number of arguments for 'set' command
[...]
Exception location my_virtualenv/local/lib/python2.7/site-packages/redis/connection.py in read_response, line 344
(traced from the publish_message() call)
My versions:
Django==1.6.2
django-websocket-redis==0.4.0
redis==2.9.1
Can someone help me to debug that ?

Looks like it's a bug.
Fix:
in ws4redis.redis_store.RedisStore in publish_message, change
self._connection.set(channel, message, ex=expire)
to
self._connection.setex(channel, expire, message)
the redis SET command does not take a 3rd argument. What I believe was meant was to set a value that expires after a number of seconds, which is the redis SETEX command. The py-redis setex method is called like setex(name, time, value).
This resolves the "Wrong number of argument for 'set'" error.
ref: https://github.com/jrief/django-websocket-redis/pull/30

I finally set the expiration time to 0 as a workaround
WS4REDIS_EXPIRE = 0
This prevents ws4redis to store anything in redis.
Fixed since 0.4.1

pyxmpp: quick tutorial for creating a muc client?

I'm attempting to write a quick load-test script for our ejabberd cluster that simply logs into a chat room, posts a couple of random messages, then exits.
We had attempted this particular test with tsung, but according to the authors, the muc functionality did not make it into this release.
pyxmpp seems to have this functionality, but darned if I can figure out how to make it work. Here's hoping someone has a quick explanation of how to build the client and join/post to the muc.
Thanks!

Hey I stumbled over your question a few times, while trying the same thing.
Here is my answer:
Using http://pyxmpp.jajcus.net/svn/pyxmpp/trunk/examples/echobot.py as a quickstart, all you have to do is import the MUC-Stuff
from pyxmpp.jabber.muc import MucRoomState, MucRoomManager
And once your Client is connected, you can connect to your room:
def session_started(self):
"""Handle session started event. May be overriden in derived classes.
This one requests the user's roster and sends the initial presence."""
print u'SESSION STARTED'
self.request_roster()
p=Presence()
self.stream.send(p)
print u'ConnectToParty'
self.connectToMUC()
def connectToMUC(self):
self.roomManager = MucRoomManager(self.stream);
self.roomHandler = MucRoomHandler()
self.roomState = self.roomManager.join(
room=JID('room#conference.server.domain'),
nick='PartyBot',
handler=self.roomHandler,
history_maxchars=0,
password = None)
self.roomManager.set_handlers()
To send a message, all you have to do is call self.roomState.send_message("Sending this Message")
To do stuff, inherit from MucRoomHandler and react on events. Notice the "set_handlers()" to roomManager though, it's is important, otherwise callbacks will not be called..