We Keep Coding

Kafka consumer receives message if set group_id to None, but it doesn't receive any message if not None?

I have the following Kafka consumer, it works well if assigning the group_id to None - it received all historical messages and my newly tested message.
consumer = KafkaConsumer(
topic,
bootstrap_servers=bootstrap_servers,
auto_offset_reset=auto_offset_reset,
enable_auto_commit=enable_auto_commit,
group_id=group_id,
value_deserializer=lambda x: json.loads(x.decode('utf-8'))
)
for m in consumer:
However, it doesn't receive anything if I set the group_id to some value. I tried to run the test producer to send new messages and nothing is received.
The consumer console does show the following message:
2020-11-07 00:56:01 INFO ThreadPoolExecutor-0_0 base.py (Re-)joining group my_group
2020-11-07 00:56:07 INFO ThreadPoolExecutor-0_0 base.py Successfully joined group my_group with generation 497
2020-11-07 00:56:07 INFO ThreadPoolExecutor-0_0 subscription_state.py Updated partition assignment: []
2020-11-07 00:56:07 INFO ThreadPoolExecutor-0_0 consumer.py Setting newly assigned partitions set() for group my_group

One partition of a topic can only be consumed by one consumer within the same ConsumerGroup.
If you do not set the group.id, the KafkaConsumer will generate a new, random group.id for you. As this group.id is unique you will see data is being consumed.
If you have multiple consumers running with the identical group.id, only one consumer will read the data whereas the other one stays idle not consuming anything.

I know, this is not the solution to the author's problem. Still, if you landed here you might be having this problem for another reason. Same as I had.
So, at least for kafka-python v2.0.2 and Aiven Kafka broker setup, the problem was solved by adding dry call of consumer.poll().
This is especially weird since this is not required when no group_id is asssigned.
Output from:
def get():
for message in consumer:
print(message.value)
consumer.commit()
Is nothing in this case
While below works as expected. It reads out only new messages from last commit():
Output from:
def get():
consumer.poll()
for message in consumer:
print(message.value)
consumer.commit()
It outputs all messages in this topic since last commit, as expected
JFYI, class constructor looks like this:
consumer = KafkaConsumer(
topics,
bootstrap_servers=self._service_uri,
auto_offset_reset='earliest',
enable_auto_commit=False,
client_id='my_consumer_name',
group_id=self.GROUP_ID,
security_protocol="SSL",
ssl_cafile=self._ca_path,
ssl_certfile=self._cert_path,
ssl_keyfile=self._key_path,
)
¯\_(ツ)_/¯

Heartbeat session expired, marking coordinator dead

When I am using python-kafka(1.4.4) library to consume the message(Kafka version 1.1.0,Python 3.7).It throw this error again and again.And I don't know where going wrong,here is my Python code,the consumer initial:
consumer = KafkaConsumer('dolphin-spider-google-book-bookinfo',
bootstrap_servers=['mq-server:9092'],
group_id = "google-book",
client_id = "dolphin-pipline-google-bookinfo-consumer-foolman",
# Manage kafka offsets manual
enable_auto_commit = False,
consumer_timeout_ms=50000,
# consume from beginning
auto_offset_reset = "earliest",
max_poll_interval_ms =350000,
session_timeout_ms = 60000,
request_timeout_ms = 700000
)
Here is the consume logic:
def consume_bookinfo(self):
while True:
try:
for books in self.consumer:
logger.info("Get books info offset: %s" ,books.offset)
self.sub_process_handle(books.value,books.offset)
except Exception as e:
logger.error(e)
def sub_process_handle(self,bookinfo,offset):
number_of_threadings = len(threading.enumerate())
if(number_of_threadings < 13):
t = threading.Thread(target=self.background_process,name="offset-" + str(offset), args=(bookinfo,), kwargs={})
t.start()
else:
# If all threading running
# Using main thread to handle
# Slow down kafka consume speed
logger.info("Reach max handle thread,sleep 20s to wait thread release...")
time.sleep(20)
self.sub_process_handle(bookinfo,offset)
def background_process(self,bookinfo):
self.parse_bookinfo(bookinfo)
self.consumer.commit_async(callback=self.offset_commit_result)
I start multi thread to handle the consume logic.But run some time,throw this error:
2019-01-30 02:46:52,948 - /home/dolphin/source/dolphin-pipline/dolphin/biz/spider_bookinfo_consumer.py[line:37] - INFO: Get books info offset: 9304
2019-01-30 02:46:52,948 - /home/dolphin/source/dolphin-pipline/dolphin/biz/spider_bookinfo_consumer.py[line:51] - INFO: Reach max handle thread,sleep 20s to wait thread release...
2019-01-30 02:47:12,968 - /home/dolphin/source/dolphin-pipline/dolphin/biz/spider_bookinfo_consumer.py[line:61] - INFO: commit offset success,offsets: {TopicPartition(topic='dolphin-spider-google-book-bookinfo', partition=0): OffsetAndMetadata(offset=9305, metadata='')}
2019-01-30 04:27:47,322 - /usr/local/lib/python3.5/site-packages/kafka/coordinator/base.py[line:964] - WARNING: Heartbeat session expired, marking coordinator dead
2019-01-30 04:27:47,323 - /usr/local/lib/python3.5/site-packages/kafka/coordinator/base.py[line:698] - WARNING: Marking the coordinator dead (node 0) for group google-book: Heartbeat session expired.
2019-01-30 04:27:47,433 - /usr/local/lib/python3.5/site-packages/kafka/cluster.py[line:353] - INFO: Group coordinator for google-book is BrokerMetadata(nodeId=0, host='35.229.69.193', port=9092, rack=None)
2019-01-30 04:27:47,433 - /usr/local/lib/python3.5/site-packages/kafka/coordinator/base.py[line:676] - INFO: Discovered coordinator 0 for group google-book
2019-01-30 04:27:47,433 - /usr/local/lib/python3.5/site-packages/kafka/coordinator/consumer.py[line:341] - INFO: Revoking previously assigned partitions {TopicPartition(topic='dolphin-spider-google-book-bookinfo', partition=0)} for group google-book
2019-01-30 04:27:47,433 - /usr/local/lib/python3.5/site-packages/kafka/coordinator/base.py[line:434] - INFO: (Re-)joining group google-book
2019-01-30 04:27:47,437 - /usr/local/lib/python3.5/site-packages/kafka/coordinator/base.py[line:504] - INFO: Elected group leader -- performing partition assignments using range
2019-01-30 04:27:47,439 - /usr/local/lib/python3.5/site-packages/kafka/coordinator/base.py[line:333] - INFO: Successfully joined group google-book with generation 470
2019-01-30 04:27:47,439 - /usr/local/lib/python3.5/site-packages/kafka/consumer/subscription_state.py[line:257] - INFO: Updated partition assignment: [TopicPartition(topic='dolphin-spider-google-book-bookinfo', partition=0)]
2019-01-30 04:27:47,439 - /usr/local/lib/python3.5/site-packages/kafka/coordinator/consumer.py[line:238] - INFO: Setting newly assigned partitions {TopicPartition(topic='dolphin-spider-google-book-bookinfo', partition=0)} for group google-book
2019-01-30 04:27:47,694 - /home/dolphin/source/dolphin-pipline/dolphin/biz/spider_bookinfo_consumer.py[line:63] - ERROR: commit offset failed,detail: CommitFailedError: Commit cannot be completed since the group has already
rebalanced and assigned the partitions to another member.
This means that the time between subsequent calls to poll()
was longer than the configured max_poll_interval_ms, which
typically implies that the poll loop is spending too much
time message processing. You can address this either by
increasing the rebalance timeout with max_poll_interval_ms,
or by reducing the maximum size of batches returned in poll()
with max_poll_records.
2019-01-30 04:27:47,694 - /home/dolphin/source/dolphin-pipline/dolphin/biz/spider_bookinfo_consumer.py[line:63] - ERROR: commit offset failed,detail: CommitFailedError: Commit cannot be completed since the group has already
rebalanced and assigned the partitions to another member.
This means that the time between subsequent calls to poll()
was longer than the configured max_poll_interval_ms, which
typically implies that the poll loop is spending too much
time message processing. You can address this either by
increasing the rebalance timeout with max_poll_interval_ms,
or by reducing the maximum size of batches returned in poll()
with max_poll_records.
How to avoid this problem?What should I do?

Lets first look at what might be the cause of this error. As it is discussed in the official kafka consumer documentation(here) kafka detects connected consumers when they call poll()
After subscribing to a set of topics, the consumer will automatically join the group when poll(Duration) is invoked. The poll API is designed to ensure consumer liveness. As long as you continue to call poll, the consumer will stay in the group and continue to receive messages from the partitions it was assigned. Underneath the covers, the consumer sends periodic heartbeats to the server. If the consumer crashes or is unable to send heartbeats for a duration of session.timeout.ms, then the consumer will be considered dead and its partitions will be reassigned.
So to stay in the group, you must continue to call poll. The max.poll.interval.ms setting declares how long a consumer can stay in a group without calling poll().
Each call to poll() returns many records(default is 500) which is then iterated through in the for message in consumer. Next call to poll() is made when you finished processing all of returned records.
If it takes too long for your program to process the records max.poll.interval.ms is going to pass and you will get kicked out of the group.
Here is what you can do:
Increase max.poll.interval.ms
Decrease max.poll.records
In #Dolphin answer,he is actually is decreasing max.poll.records to 1. I prefer doing it this way:
self.consumer = kafka.KafkaConsumer(topic, bootstrap_servers='servers:ports',
group_id='group_id',max_poll_records=1,max_poll_interval_ms=300000)
the important part is max_poll_records=1. Of course, you might want to set it to a lager value than 1.

Tweak to consumer poll function:
def consume_bookinfo(self):
while True:
try:
msg_pack = self.consumer.poll(timeout_ms=5000,max_records=1)
for messages in msg_pack.items():
for message in messages:
var_type = type(message)
if(isinstance(message,TopicPartition)):
logger.info("TopicPartition: %s", TopicPartition)
if(var_type == list):
for consumer_record in message:
#for books in self.consumer.poll(max_records = 5):
logger.info("Get books info offset: %s" ,consumer_record.offset)
self.sub_process_handle(consumer_record.value,consumer_record.offset)
except Exception as e:
logger.error(e)
This works fine for me!!!!!!!!!!!!!!!!

how to use callback in beanstalkd ？

There is 3 machines and one publisher, two consumers.
I am using golang to publish order to some machine.
And the machine is used python to be consumer.
I want to know how can I get the result that the order is finished or failed in publisher.
And if the order is not belong to machine one, what should i do?
release or bury it?
python:
consumer:
import beanstalkc
def get_beanstalk_data(conf):
beanstalk = beanstalkc.Connection(host='127.0.0.1',port=11300)
beanstalk.use('cloud')
beanstalk.watch('cloud')
beanstalk.ignore('default')
job = beanstalk.reserve()
if job.body == "one": #job.body == "two"
#TODO
job.delete()
return job.body
else:
#TODO what should I do in here, because there is two consumer and get different orders
while True:
data = get_beanstalk_data(conf)
print data
golang:
publish:
package main
import (
"fmt"
"github.com/kr/beanstalk"
"time"
)
func main() {
c, err := beanstalk.Dial("tcp", "127.0.0.1:11300")
id, err := c.Put([]byte("hello"), 1, 0, 120*time.Second)
if err != nil {
fmt.Println(err)
}
fmt.Println(id)
}

The right way to let publisher know the status of a job is using callbacks
With job let the publisher put a callback url (a queue or http) and on success or failure of the job th consumer may send a status message to the status call back
so the job structure may loom like
//JobRequest has the struct storing request name and body
type JobRequest struct {
ID string
RequestBody []byte
CallbackURL *string
}
The json strng of the above struct will be the job body.The consumer would get the CallbackURL and will senbd status to that url
Trying to explain as reqiuested details
Let's call producer and consumer(s) master and workers(s) .
When ever a job is available master would create a job object which has
job id (A unique value to identify teh job)
RequestBody (The details of the job)
StatusCallbackURL (URL the worker would hit with the job status)
One of the workers listening to the queue reserve the job,there buy telling i will try to do this job
Decode the json and get the job details .Now does the job
On success delete the job from queue and send status to the CallbackURL
On failure if non temporary failure send status as fail to the CallbackURL and delete the job
If temporary failure do nothing as after reserve timeout over it wil be re-enqueued
Now this object is converted to json and put in queue
PS: Do not delete the job before succesfully completing .On Completion or Permanant Failure only delte the job

How can I get my logger in a Python Flask app to use unique IDs per page view for logging?

I have a Flask app with a logging system that I wrote based on the standard Python logger.
I want to have a unique ID for each page view, so I can track the entire process and check for logged errors and premature ending.
The first thing I tried was to put unique ID creator into the __init__ for the logger object.The result of that was that all requests had the same view. I moved where the unique ID was created to a method, and this improved the situation - multiple IDs appeared in the logs, and everything appeared to be working.
However, it seems sometimes two requests use the same logger object. It seems when one request is running, another initiates and runs the ID generation method. Then the first request starts using the new ID also...
22:04:31 - MPvzGNAelE : in content
22:04:31 - MPvzGNAelE : in parse options
22:04:31 - MPvzGNAelE : finished parse options
22:04:31 - MPvzGNAelE : about to query API for user info. user id : 7401173, resource id: 59690
#the following is where the 2nd requests starts
22:04:31 - SbEOjmbFSa : in frame initial
22:04:31 - SbEOjmbFSa : in frame 2 - cleaned IDs are 1114 127059932
22:04:31 - SbEOjmbFSa : in parse options
22:04:31 - SbEOjmbFSa : finished parse options
22:04:31 - SbEOjmbFSa : in frame finishing - for 1114 127059932
#the following is the first request continuing with the new ID
22:04:31 - SbEOjmbFSa : api user info status is success
22:04:31 - SbEOjmbFSa : user_id is 5549565, user name is joe_spryly
22:04:31 - SbEOjmbFSa : config[data_source] is 0
22:04:31 - SbEOjmbFSa : api seems to be working, so going to retrieve items for 59690 7401173
22:04:31 - SbEOjmbFSa : getting items from API for 59690 7401173
This is my log object code...
class AS_Log(object):
def __init__(self):
self.log=logging.getLogger('aman_log')
logging.basicConfig(filename='amanset_log',level=logging.DEBUG)
def generate_id(self):
from random import choice
import string
chars=string.letters+string.digits
self.log_id=''.join([choice(chars) for i in range(10)])
def format_date(self,timestamp):
return datetime.fromtimestamp(timestamp).strftime('%m-%d-%y %H:%M:%S')
def log_msg(self,message):
self.log.debug('{0} - {1} : {2}'.format(self.format_date(time()),self.log_id,message))
I initiate the log in the flask app like
as=AS_Log()
Then call generate_id per request like
#app.route('/<resource_id>/<user_id>/')
def aman_frame(resource_id,user_id):
as.generate_id()
return amanset_frame(resource_id,user_id)
then use the log in the amanset_frame function.
I have some notion that this is related application context in Flask but I don't understand how to use that to fix this. Do I use app_context(), and if so, how?

Ultimately this is caused by mutating the shared as.log_id field across multiple execution contexts (most likely threads, but it could be greenlets or even sub-processes if as is in the shared parent interpreter as with certain uwsgi / mod_wsgi setups).
request-1 --> mutates log_id --> does ... work ... --> response
request-2 --> mutates log_id --> does ... work ... --> response
The fix is to either use a proper thread-local for the logger, so a new logger is created per thread of execution. Werkzeug has a LocalProxy that makes this a bit easier to manage too.

RabbitMQ / Pika - guaranteeing that messages are received in the order created?

As a simple example, I'm adding 5 items to a new RabbitMQ(v 2.6.1) queue:
import pika
connection = pika.BlockingConnection(pika.ConnectionParameters(host='not.my.real.server.net'))
channel = connection.channel()
channel.queue_declare(queue='dw.neil',durable=True)
# add 5 messages to the queue, the numbers 1-5
for x in range(5):
message = x+1
channel.basic_publish(exchange='',routing_key='dw.neil', body=str(message))
print " [x] Sent '%s'" % message
connection.close()
I purge my queue and then run the above code to add the 5 items:
nkodner#hadoop4 sports_load_v2$ python send_5.py
[x] Sent '1'
[x] Sent '2'
[x] Sent '3'
[x] Sent '4'
[x] Sent '5'
Now, I'm trying to simulate failed processing. Given the following code to consume from the queue. Notice that I have the call to basic_ack commented out:
#!/usr/bin/env python
import pika
connection = pika.BlockingConnection(pika.ConnectionParameters(host='not.my.real.server.net'))
channel = connection.channel()
channel.queue_declare(queue='dw.neil',durable=True)
method_frame, header_frame, body=channel.basic_get(queue='dw.neil')
print method_frame, header_frame
print "body: %s" % body
#channel.basic_ack(delivery_tag=method_frame.delivery_tag)
connection.close()
I run the receiving code to grab an item off the queue. As I'd expect, I get item #1:
nkodner#hadoop4 sports_load_v2$ python r5.py
<Basic.GetOk(['message_count=9', 'redelivered=False', 'routing_key=dw.neil', 'delivery_tag=1', 'exchange='])>
<BasicProperties([])>
body: 1
Since the call to channel.basic_ack() is commented out, I would expect the unacknowledged message to be placed on the queue so that the next consumer will get it. I would hope that message #1 is the first message (again) out of the queue, with the Redelivered property set to True. Instead, message #2 is received:
nkodner#hadoop4 sports_load_v2$ python r5.py
<Basic.GetOk(['message_count=9', 'redelivered=False', 'routing_key=dw.neil', 'delivery_tag=1', 'exchange='])>
<BasicProperties([])>
body: 2
And all of the other messages in the queue are received before #1 comes back with Redelivered flag set to True:
...
nkodner#hadoop4 sports_load_v2$ python r5.py
<Basic.GetOk(['message_count=9', 'redelivered=False', 'routing_key=dw.neil', 'delivery_tag=1', 'exchange='])>
<BasicProperties([])>
body: 5
nkodner#hadoop4 sports_load_v2$ python r5.py
<Basic.GetOk(['message_count=9', 'redelivered=True', 'routing_key=dw.neil', 'delivery_tag=1', 'exchange='])>
<BasicProperties([])>
body: 1
Are there any properties or options I could be setting so that I keep getting #1 delivered until its acknowledged?
My use-case is loading a data warehouse with sequentially-generated files. We're using message-based processing to let my program know some new files are ready and are to be loaded into the DW. We have to process the files in the order that they're generated.

This has been addressed in RabbitMQ 2.7.0 - we were running 2.6.1.
From the release notes:
New features in this release include:
order preserved of messages re-queued for a consumer

Try using channel.basic_reject - this should push the unacknowledged message back to RabbitMQ which will treat the message as a new message. Also -- if you have a failed message stuck, you can use channel.basic_recover to tell RabbitMQ to redeliver all non-acknowledged messages.
http://pika.github.com/communicating.html#channel.Channel.basic_reject
http://pika.github.com/communicating.html#channel.Channel.basic_recover
http://www.rabbitmq.com/extensions.html#negative-acknowledgements provides distinguishing information on Basic.Reject vs Basic.Nack.
Message ordering semantics are explained at http://www.rabbitmq.com/semantics.html