We Keep Coding

Streaming Kafka with Python :kafka.errors.NoBrokersAvailable: NoBrokersAvailable

I have tried this tutorial and have successfully consumed kafka topics that are published on a server at my work place. I am not the producer, just purely a consumer. However, the code in that tutorial is to stream in a terminal. Now I want to try it with Python and record the messages into text file (or something of that sort).
This is the code I use, after reading a few more threads and tutorials (such as here):
from kafka import KafkaConsumer
bootstrap_servers = ['xxxxxx:xxxx']
topicName: = 'my_topic_name'
consumer = KafkaConsumer(topicName, group_id='group1', bootstrap_servers=bootstrap_servers, consumer_timeout_ms=1000)
for msg in consumer:
print(msg.value)
Here I want to first print out the message. But I get this error after 1000ms timeout:
kafka.errors.NoBrokersAvailable: NoBrokersAvailable
which sounds logical to me, since a broker is needed and the code above does not seem to do anything with a broker.
If I don't set the consumer_timeout_ms=1000ms, the Python console get just stuck without displaying anything.
How do I resolve this?
More details:
I am doing the following in parallel:
1 - Run zookeeper in one terminal
2 - Run kafka cluster in another terminal
3 - Streaming the topics (nicely) in another terminal with the command kafka-consumer-console
4 - In another terminal, run the Python code in this question.
All of these terminals are Ubuntu in WLS2 (Windows).

If you're able to use the WSL terminal with kafka-console-consumer, then running Python code there should work the same.
If you're connecting to a remote Kafka server, chances are the WSL2 network settings are simply not able to reach that address. (multiple issues elsewhere talk about WSL2 and not having external internet access) . Therefore, you should really consider running Python code on the Windows host itself. Otherwise, sounds like you'll need to adjust your network configurations.
The for loop will wait for new messages to a topic, not read existing data until you add another parameter to the consumer to tell it to
FWIW, you can use kafka-console-consumer ... >> file.txt to write out to a file

Recommend Way to Interact with an SSH session in Python

I'm working on a little python program to speed up managing various Raspberry Pi servers over SSH. It's all done but for one thing. Interacting with an SSH session isn't wokring the way I want it to.
I can interact with a command but some commands (specifically apt full-upgrade) which ask or have potential to ask questions whilst they're running aren't working. So when it reaches the point where it asks do you want to continue [Y/n] it falls over. I believe it's because apt can't read from the stdin so aborts.
I know I could run the apt command with the -y flag and bypass the question but ideally I'd like to be able to capture requests and ask the user for input. I've been using Paramiko to manage my SSH sessions and what I'm doing is capturing the stdout and passing it to the find function to look for things like [Y/n] and if it finds that then redirect the user to an input prompt which works but because theres no stdin when apt asks the question it aborts and when I send my user input back to the SSH session I get a socket closed error.
I've been looking for alternatives or ways to get round the issue but apart from seeing fabric mentioned as an alternative to paramiko I can't see a lot of other options out there. Does anyone know of any alternatives I can try to paramiko. I don't think fabric will work for me given its based off paramiko so I assume I'd hit the same error there. I'd appreciate any recoomendations or pointers if there's other parts of Paramiko I can try (I've stuck to using exec_command). I have tried channels which work to a point but I don't think keeping the SSH session open is the issue I think I need someway to keep stdin open/accessible to the apt command on the remote machine so it doesn't abort the command.
At the minute the best idea I've got to get round it is to run the command let it potentially abort look in stdout for the relevant phrases then run the command again after giving the user chance to set their inputs and pass the whole lot to stdin?
EDIT:
My program in steps:
login to the remote host
issue a command
use .find on the command to check for the use of 'sudo'
if sudo is present additionally send the user password to stdin along with the command
read the stdout to check for keywords/phrases like '[Y/n]' which are present when the user is being asked for input when running a command
if a keyword is found then ask the user for their input which can then be sent back to stdin to continue the command.
Steps 5 and 6 are where it fails and returns with a socket closed error. Looking online I don't think the issue is with paramiko as such but with the command running on the remote host. In my case sudo apt full-upgrade.
When I run that command it runs up to the 'Would you like to continue' point the automatically aborts, I believe the issue there is because there is nothing present in the stdin at that point (thats what I'm asking the user for) Apt automatically aborts
This is the part of my code where I'm running the commands:
admin = issue_cmd.find('sudo')
connect.connect(ip_addr, port, uname, passwd)
stdin, stdout, stderr = connect.exec_command(issue_cmd, get_pty=True)
if admin != -1:
print('Sudo detected. Attempting to elevate privileges...')
stdin.write(passwd + '\n')
stdin.flush()
else:
continue
output = stdout.read()
search = str(output).find('[Y/n]')
if search != -1:
resp = input(': ')
print(resp)
stdin.write(resp + '\n')
stdin.flush()
else:
pass
print(stdout.read().decode('utf-8').strip("\n"))
print(stderr.read().decode('utf-8').strip("\n"))
connect.close()
and here's the error message I'm seeing:
OSError: Socket is closed

accessing router via telnet lib python

I'm working on a code that uses telnetlib of python to connect to a router and execute commands and stores the output in a file.
I'm using read_until('#') function and expecting a Router prompt, then execute the next command but my code freezes when I receive a '--More--' data from the remote telnet side. I tried using a pattern match to find '--More--' but then sometime the --More-- keyword doesn't come at once.
Any suggestion ?
Do I have to send some IAC command to the remote telnet side ?

sometime the --More-- keyword doesn't come at once
Try passing in a timeout.
Example: set timeout to 5 seconds for read_until():
read_until('--More--', 5)
Alternatively, you could use the expect() function to look for either '#' or '--More--' with a timeout:
expect(['#', '--More--'], 5)

Paramiko get stdout from connection object (not exec_command)

I'm writing a script that uses paramiko to ssh onto several remote hosts and run a few checks. Some hosts are setup as fail-overs for others and I can't determine which is in use until I try to connect. Upon connecting to one of these 'inactive' hosts the host will inform me that you need to connect to another 'active' IP and then close the connection after n seconds. This appears to be written to the stdout of the SSH connection/session (i.e. it is not an SSH banner).
I've used paramiko quite a bit, but I'm at a loss as to how to get this output from the connection, exec_command will obviously give me stdout and stderr, but the host is outputting this immediately upon connection, and it doesn't accept any other incoming requests/messages. It just closes after n seconds.
I don't want to have to wait until the timeout to move onto the next host and I'd also like to verify that that's the reason for not being able to connect and run the checks, otherwise my script works as intended.
Any suggestions as to how I can capture this output, with or without paramiko, is greatly appreciated.

I figured out a way to get the data, it was pretty straight forward to be honest, albeit a little hackish. This might not work in other cases, especially if there is latency, but I could also be misunderstanding what's happening:
When the connection opens, the server spits out two messages, one saying it can't chdir to a particular directory, then a few milliseconds later it spits out another message stating that you need to connect to the other IP. If I send a command immediately after connecting (doesn't matter what command), exec_command will interpret this second message as the response. So for now I have a solution to my problem as I can check this string for a known message and change the flow of execution.
However, if what I describe is accurate, then this may not work in situations where there is too much latency and the 'test' command isn't sent before the server response has been received.
As far as I can tell (and I may be very wrong), there is currently no proper way to get the stdout stream immediately after opening the connection with paramiko. If someone knows a way, please let me know.

Retrieving netstat from local server python script

How do i retrieve the netstat -a data from my python script from a local server. I have tried subprocess.Popen(['ssh','server','pass','netstat','-a'],stdout=file1) but it does not work. Any advice?

You must have key based authorisation from the desired server where netstat is suppose to run.
In the absence on key based authorisation, the ssh command will return a prompt asking for password.
Below is the link for key based authorisation:
http://wp.uberdose.com/2006/10/16/ssh-automatic-login/
or
http://linuxproblem.org/art_9.html
Once the keys are exchanged, the command mentioned in your question does not require any password and no prompt will occur when you run it.
subprocess.Popen(['ssh','server','netstat','-a'],stdout=file1)