This question already has answers here:
Connect to Kafka running in Docker
(5 answers)
Closed 2 years ago.
I have Kafka running on a container on my desktop.
I can connect to it just fine using a tool called "Kafka tool" where I can see my topics for example.
I'm having issues reading and writing to/from a Kafka topic.
what's annoying me is that it won't give me an error message, it's behaving like if the topic doesn't have any messages on it, but it does, I can see using the tool, even added two messages manually.
the topic exists and has two messages on it (which I added manually using this UI)
problem:
the code that sends messages to the topic runs fine, but the messages don't make it to Kafka
the code that reads messages from the topic doesn't read anything. It sits there like if there are no messages to be read.
Also, I can use the same consume to list the topics (which indicates the connection was successful)
The kafka version is 2.4.
Any idea what the problem may be?
I have tried "bootstrap_servers=['localhost:9092', 'kafka-server:9092']" but it also didnt work
Thanks
KafkaProducer: You need to execute flush after send
producer.send('testTopic', b'Hello, World!')
producer.flush()
KafkaConsumer: Specify bootstrap_servers and auto_offset_reset
consumer = KafkaConsumer('testTopic',
bootstrap_servers=['localhost:9092'],
auto_offset_reset='earliest')
for message in consumer:
print(message)
Related
I have tried this tutorial and have successfully consumed kafka topics that are published on a server at my work place. I am not the producer, just purely a consumer. However, the code in that tutorial is to stream in a terminal. Now I want to try it with Python and record the messages into text file (or something of that sort).
This is the code I use, after reading a few more threads and tutorials (such as here):
from kafka import KafkaConsumer
bootstrap_servers = ['xxxxxx:xxxx']
topicName: = 'my_topic_name'
consumer = KafkaConsumer(topicName, group_id='group1', bootstrap_servers=bootstrap_servers, consumer_timeout_ms=1000)
for msg in consumer:
print(msg.value)
Here I want to first print out the message. But I get this error after 1000ms timeout:
kafka.errors.NoBrokersAvailable: NoBrokersAvailable
which sounds logical to me, since a broker is needed and the code above does not seem to do anything with a broker.
If I don't set the consumer_timeout_ms=1000ms, the Python console get just stuck without displaying anything.
How do I resolve this?
More details:
I am doing the following in parallel:
1 - Run zookeeper in one terminal
2 - Run kafka cluster in another terminal
3 - Streaming the topics (nicely) in another terminal with the command kafka-consumer-console
4 - In another terminal, run the Python code in this question.
All of these terminals are Ubuntu in WLS2 (Windows).
If you're able to use the WSL terminal with kafka-console-consumer, then running Python code there should work the same.
If you're connecting to a remote Kafka server, chances are the WSL2 network settings are simply not able to reach that address. (multiple issues elsewhere talk about WSL2 and not having external internet access) . Therefore, you should really consider running Python code on the Windows host itself. Otherwise, sounds like you'll need to adjust your network configurations.
The for loop will wait for new messages to a topic, not read existing data until you add another parameter to the consumer to tell it to
FWIW, you can use kafka-console-consumer ... >> file.txt to write out to a file
I am attempting to send data to a Kafka topic via python as a remote producer. My script reports no exceptions, but nothing shows up in Kafka. I'm able to get a response from the brokers (all on one separate host) using the consumer method. Looking through the forums I saw to make sure and flush the write cache, but no luck there. Script is below:
from kafka import KafkaProducer, KafkaConsumer
from json import dumps
producer = KafkaProducer(bootstrap_servers='192.168.1.100:9093', value_serializer=lambda x: dumps(x).encode('utf-8'))
producer.send('home-sensehat-temperature',{"timestamp": "2020-08-12 23:31:19.102347", "temperature": 127.6969})
producer.flush()
consumer=KafkaConsumer(bootstrap_servers='192.168.1.100:9093')
print(consumer.topics())
The response I get from consumer.topics() is:
{'home-sensehat-temperature', 'home-camera-path', 'home-sensehat-humidity', 'home-sensehat-pressure'}
So this implies I can make a good connection to the brokers.
I tried digging through the kafka broker logs but couldn't find anything. Any help would be greatly appreciated!
Got it! Everything was fine on the remote producer, but I had to add a config parameter on the brokers. Specifically the advertised.listeners parameter in the config/server.properties file that each broker uses. Details are in the below post:
Kafka : How to connect kafka-console-consumer to fetch remote broker topic content?
Thanks everyone!
This question already has answers here:
Display the contents of a log file as it is updated
(3 answers)
Closed 6 years ago.
I am trying to create a web application with Flask.
The problem is that it has been 2 weeks since I am stuck on a problem.
I would like to run a python command that launches a server, retrieve the standard output, and display in real time on the web-site.
I do not know at all how to do because if I use "render_template" I do not see how to update the web site-the values sent in the console.
I use python 2.7, thank you very much
It's gonna take a lot of work to get this done and probably more then you think but I'll still try to help you out.
To get any real time updates to the browser you're going to need something like a socket connection, something which allows you to send multiple messages at any time. Not just when the browser requests it.
So imagine this with a regular http connection you can only receive a message once and once you receive that message you cannot receive a message again. You can only call return once and not again.
Once you call return, you cannot call return again to send another message.
So basically with a regular http request you can only receive the log messages once and once any changes have been made to the log you cannot send those changes to the client again since the connection is end.
The connection is end the moment you call return.
There is a way to fix this by using a socket connection. A socket connection would allow you to open a connection with the user and server and they both can send messages at any time as long as the connection is open. The connection is only not open when you manually close it.
Check this answer for ways you could have real time updates with flask. If you want to do it with sockets (which is what I suggest you to use) then use the websocket interface instead.
There's options like socketio for python which allow you to write websocket applications in python.
Overall this is gonna be split into 5 parts:
Start a websocket server when the Flask application start
Create a javsacript file (one that the browser loads) that connects with the websocket server
Find the function that gets triggered whenever Flask logging occurs
Send a socket message with the log inside of it
Make the browser display the log whenever it receives a websocket message
Here's a sample application written in Flask and socketio which should give you a idea on how to use socketio.
There's a lot to it and there's part you might be new to like websockets but don't let that stop you from doing what you want to do.
I hope this help, if any part confuses you then feel free to respond.
The simple part : server side, you could redirect the stdout and stderr of the server to a file,
import sys
print("output will be redirected")
# stdout is saved
save_stdout = sys.stdout
fh = open("output.txt","w")
sys.stdout = fh
the server itself would then read that file within a subprocess.
f = subprocess.Popen(['tail','-F',"output.txt", '-n1'],\
stdout=subprocess.PIPE,stderr=subprocess.PIPE)
p = select.poll()
p.register(f.stdout)
and the following threaded :
while True :
if p.poll(1):
output+=f.stdout.readline()
You can also use the tailhead or tailer libraries instead of the system tail
Now, the problem is that the standard output is a kind of active pipe and output is going to grow forever, so you'll need to keep only a frame of that output buffer.
If you have only one user that can connect to that window, the problem would be different, as you could flush the output as soon as is it send to that only client. See the difference between a terminal window and multiplexed, remote terminal window ?
I don't know flask, but client side, you only need some javascript to poll the server every second with an ajax request asking for the complete log (or --case of unique client-- the buffer to be appended to the DOM). You could also use websockets, but it's not an absolute necessity.
A compromise between the two is possible (infinite log with real time append / multiplexed at different rate) and it requires to keep a separate output buffer for each client.
I'm trying to write a simple IRC bot in Python. My bot would connect to a server, join a channel and then print to the console the messages received by the bot.
I did it, but the server always disconnects me after a while because of ping timeout. I tried to use some libraries (python-irc and oyoyo) and they both ALLEGEDLY handle automatically the ping-pong messages with the server. But I tried with irc.abjects.net and irc.criten.net and with both of them these libraries failed.
Then I made a script based on this one: http://hawkee.com/snippet/9725/. And it failed too. So... I suppose the problem is very stupid and simple, but I jsut can't figure out a solution.
Note: with the library irc3 the ping-pong works fine, but it uses asyncio, which conflicts with some other parts of my code (not related to irc). So I can't use it.
This question already has an answer here:
Closed 10 years ago.
Possible Duplicate:
How to write a twisted server that is also a client?
How can I create a tcp client server app with twisted, where also the server can send requests, not just answer them? Sort of like a p2p app but where clients always initiate the connection. Since I don't know when the requests from the server will occur, I don't see how I can do this once the reactor is started.
The question you have to ask yourself is: why is the server sending a request?
Presumably something has happened in the world that would prompt the server to send a request; it wouldn't just do it at random. Even if it did it at random, the thing that has happened in the world would be "some random amount of time has passed". In other words, callLater(random(...), doSomething).
When you are writing a program with Twisted, you start off by setting up ways to react to events. Then you run the reactor - i.e. the "thing that reacts to events" - forever. At any time you can set up new ways to react to incoming network events (reactor.connectTCP, reactor.listenTCP, reactor.callLater) or tear down existing waiting things (protocol.loseConnection, port.stopListening, delayedCall.cancel). You don't need to re-start the reactor; in fact, really, the only thing you should do before the reactor runs is do reactor.callWhenRunning(someFunctionThatListensOrConnects), and write someFunctionThatListensOrConnects to do all your initial set-up. That set-up then happens once the reactor is already running, which demonstrates that you don't need to do anything in advance; the reactor is perfectly capable of changing its configuration as it runs.
If the event that causes the server to send an event to client B the fact that client A sent it a message, then your question is answered by the FAQ, "how do I make input on one connection result in output on another?"