I am attempting to send data to a Kafka topic via python as a remote producer. My script reports no exceptions, but nothing shows up in Kafka. I'm able to get a response from the brokers (all on one separate host) using the consumer method. Looking through the forums I saw to make sure and flush the write cache, but no luck there. Script is below:
from kafka import KafkaProducer, KafkaConsumer
from json import dumps
producer = KafkaProducer(bootstrap_servers='192.168.1.100:9093', value_serializer=lambda x: dumps(x).encode('utf-8'))
producer.send('home-sensehat-temperature',{"timestamp": "2020-08-12 23:31:19.102347", "temperature": 127.6969})
producer.flush()
consumer=KafkaConsumer(bootstrap_servers='192.168.1.100:9093')
print(consumer.topics())
The response I get from consumer.topics() is:
{'home-sensehat-temperature', 'home-camera-path', 'home-sensehat-humidity', 'home-sensehat-pressure'}
So this implies I can make a good connection to the brokers.
I tried digging through the kafka broker logs but couldn't find anything. Any help would be greatly appreciated!
Got it! Everything was fine on the remote producer, but I had to add a config parameter on the brokers. Specifically the advertised.listeners parameter in the config/server.properties file that each broker uses. Details are in the below post:
Kafka : How to connect kafka-console-consumer to fetch remote broker topic content?
Thanks everyone!
Related
I have a quick question regarding sending and receiving STOMP messages over an AMQ broker.
I have a python script that is sending data as STOMP messages to an AMQ instance, and another script which listens to that messages topic and grabs it. Everything is working as expected so far, but I'm curious about the security of the system. Would someone on the network be able to use a packet sniffer or similar tool to read the messages that are being sent/received by the broker? Or are they unable to see the data without the AMQ server login? My gut tells me it's the latter, but I wanted to confirm.
For context, my sender sends out the data using stomp.py
conn = stomp.Connection(host_and_ports=[(ip, port)])
conn.connect(wait=True)
conn.send(body=clean_msg, destination=f"/topic/{topic}")
Is that conn.send call encrypting or protecting my data in any way? If it isn't, how do I go about doing so? All my research into ActiveMQ and STOMP encryption leads me to encrypting the login or using SSL to login to the AMQ server, which leads me to believe that as long as the login is secure, I should be fine.
Thanks in advance!
STOMP is a text oriented protocol so unless you're using SSL/TLS then anybody who has access to the network would be able to look at the packets and fairly easily read the message data that's being sent from your producer(s) to the broker and from the broker to the consumer(s).
From what I can tell your Python STOMP client is not using SSL/TLS so your transmissions would not be protected.
Furthermore, once the data is stored on the broker then anybody with file-system access would be able to read the data as it is not encrypted in the storage. You can, of course, mitigate this risk by enforcing standard user access.
I have tried this tutorial and have successfully consumed kafka topics that are published on a server at my work place. I am not the producer, just purely a consumer. However, the code in that tutorial is to stream in a terminal. Now I want to try it with Python and record the messages into text file (or something of that sort).
This is the code I use, after reading a few more threads and tutorials (such as here):
from kafka import KafkaConsumer
bootstrap_servers = ['xxxxxx:xxxx']
topicName: = 'my_topic_name'
consumer = KafkaConsumer(topicName, group_id='group1', bootstrap_servers=bootstrap_servers, consumer_timeout_ms=1000)
for msg in consumer:
print(msg.value)
Here I want to first print out the message. But I get this error after 1000ms timeout:
kafka.errors.NoBrokersAvailable: NoBrokersAvailable
which sounds logical to me, since a broker is needed and the code above does not seem to do anything with a broker.
If I don't set the consumer_timeout_ms=1000ms, the Python console get just stuck without displaying anything.
How do I resolve this?
More details:
I am doing the following in parallel:
1 - Run zookeeper in one terminal
2 - Run kafka cluster in another terminal
3 - Streaming the topics (nicely) in another terminal with the command kafka-consumer-console
4 - In another terminal, run the Python code in this question.
All of these terminals are Ubuntu in WLS2 (Windows).
If you're able to use the WSL terminal with kafka-console-consumer, then running Python code there should work the same.
If you're connecting to a remote Kafka server, chances are the WSL2 network settings are simply not able to reach that address. (multiple issues elsewhere talk about WSL2 and not having external internet access) . Therefore, you should really consider running Python code on the Windows host itself. Otherwise, sounds like you'll need to adjust your network configurations.
The for loop will wait for new messages to a topic, not read existing data until you add another parameter to the consumer to tell it to
FWIW, you can use kafka-console-consumer ... >> file.txt to write out to a file
This question already has answers here:
Connect to Kafka running in Docker
(5 answers)
Closed 2 years ago.
I have Kafka running on a container on my desktop.
I can connect to it just fine using a tool called "Kafka tool" where I can see my topics for example.
I'm having issues reading and writing to/from a Kafka topic.
what's annoying me is that it won't give me an error message, it's behaving like if the topic doesn't have any messages on it, but it does, I can see using the tool, even added two messages manually.
the topic exists and has two messages on it (which I added manually using this UI)
problem:
the code that sends messages to the topic runs fine, but the messages don't make it to Kafka
the code that reads messages from the topic doesn't read anything. It sits there like if there are no messages to be read.
Also, I can use the same consume to list the topics (which indicates the connection was successful)
The kafka version is 2.4.
Any idea what the problem may be?
I have tried "bootstrap_servers=['localhost:9092', 'kafka-server:9092']" but it also didnt work
Thanks
KafkaProducer: You need to execute flush after send
producer.send('testTopic', b'Hello, World!')
producer.flush()
KafkaConsumer: Specify bootstrap_servers and auto_offset_reset
consumer = KafkaConsumer('testTopic',
bootstrap_servers=['localhost:9092'],
auto_offset_reset='earliest')
for message in consumer:
print(message)
In a web application with MQTT in python (using paho-mqtt lib) I would like to know if there is a way to get the broker status in real time, because the only way that i find is to store the variable "rc" into the method on_connect but it's more like a client/connection state.
EDIT 1 : after reading mosquitto broker documentation, i found that you can subscribe to '$SYS/broker/connection/#' which is supposed to give you back 1 if the connection is up and 0 if it goes down. However when i do :
subscribe.callback(self.message_callback, '$SYS/broker/connection/#', port = port, hostname=broker, auth=authentication, protocol=client.MQTTv31, tls=TLS)
impossible to get payload and topic this message although i'm doing exactly the same command to get messages from my sensors except that the topic is '#' and it's working perfectly.
Does anyone knows why ?
There is no way to poll the state of the connection to the broker from the client.
The on_disconnect callback should be called when the connection to the broker is dropped.
This should be kicked off when the keep alive times out, but also as the result of a failure to publish (if you try to publish data before the timeout expires).
Also the rc from a call to the publish command will indicate if the connection has dropped.
I'm trying to design a system that will process large amounts of data and send updates to the client about its progress. I'd like to use nginx (which, thankfully, just started supporting websockets) and uwsgi for the web server, and I'm passing messages through the system with zeromq. Ideally the solution could be written in Python, but I'm also open to a Nodejs or even a Go solution.
Here is the flow that I'd like to achieve:
Client visits a website and requests that a large amount of data be processed.
The server farms out the processing to another process/server [the worker] via zeromq, and replies to the client request explaining that processing has begun, including information about how to set up a websocket with the server.
The client sets up the websocket connection and waits for updates.
When the processing is done, the worker sends a "processing done!" message to the websocket process via zeromq, and the websocket process pushes the message down to the client.
Is what I describe possible? I guess I was thinking that I could run uwsgi in emperor mode so that it can handle one process (port) for the webserver and another for the websocket process. I'm just not sure if I can find a way to both receive zeromq message and manage websocket connections all from the same process. Maybe I have to initiate the final websocket push from the worker?
Any help/correct-direction-pointing/potential-solutions would be much appreciated. Any sample or snippet of an nginx config file with websockets properly routed would be appreciated as well.
Thanks!
Sure, that should be possible. You might want to look at zerogw.