I have installed Kafka with all default settings on my Ubuntu partition
On my windows partition on the same machine, using python, I have followed the steps of this guide.
Producer1.py creates the topic 'First_Topic' in my Kafka
But Consumer1.py does not show any information and times out after some time.
When I access Kafka from my ubuntu terminal, am able to see the newly created Kafka but no messages are consumed/available.
So initially my doubt was that am not able to connect from Windows partition to Ubuntu, but as I can see that the Kafka topic has got created then it shouldn't be that case.
Only change in code is the localhost which I've replaced as bootstrap_servers = ['DESKTOP-MyUbuntu:9092'] from the original bootstrap_servers = ['localhost:9092']
What changes do I need to do to read the kafka messages in my consumer1.py?
As mentioned in the comments, you'll need to modify advertised.listeners to expose your external desktop hostname or IP (run ipconfig and use the address given by the router) from the WSL2 environment. By default Kafka only accepts local connections (local, here, meaning within the same WSL hypervisor)
There's also some firewall or port forwards on WSL2 that needs adjusted - there's some answers here on it (use port 9092 for Kafka)
Related
I am attempting to create a simple producer and consumer with two Python scripts, using Kafka deployed on Microk8s. However, when running the producer.py script, I get the following error on repeat:
...|FAIL|rdkafka#producer-1| [thrd:...:9092/bootstrap]: ...:9092/bootstrap: Connect to ipv4#localhost:9092 failed: Connection refused (after 0ms in state CONNECT, ... identical error(s) suppressed
I am fairly confident that this issue is a result of the listeners not being configured correctly, but I have so far been unable to figure out what I need to do to fix them, due to what I assume is my complete lack of any knowledge in this area. I have reviewed these resources, in addition to several others from this site, but have been unable to find a solution, or at least a solution I can understand enough to act upon.
Steps to Reproduce:
The Python scripts to generate the producer and consumer can be found here.
For Microk8s installation, I followed these instructions. I also installed Helm, since my project requirements dictate that I use Helm charts.
I then installed Kafka using:
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install kafka-release bitnami/kafka
The Python code in the linked post uses 'localhost:9092', as the error also shows - Connect to ipv4#localhost:9092 failed
If you are trying to run that code in a k8s pod, then you need to give the external broker DNS addresses, not the local pod address.
If you run the Python code from outside the k8s cluster, you need to expose a ClusterIP / NodePort external service or Ingress (as the linked Strimzi post shows; plus, you can can still use Strimzi Operator with Helm, so you don't really need the Bitnami Charts).
At a high level, the advertisted.listeners tells clients how to connect to a specific broker. If you advertise localhost, the pod will try to connect to itself, even if the bootstrap connection worked (setup by just listeners). If you advertise kafka.svc.cluster.local, then it will try to connect to the kafka service in the default namespace... But you still need to actually set boostrap.servers = kafka.svc.cluster.local:9092, for example.
I am trying to install CKAN on a raspberry pi using the instructions found here. The RPi is running the latest Raspberry Pi OS (w/desktop / January 11th 2021 / Kernel 5.4) All is successful and CKAN seems to function normally when viewed on the local machine, but any attempt to access CKAN from other machines on the network get Connection Refused. Is CKAN blocking requests from external machines by default, and if so how do I change this? Both jetty and solr are accessible from external machines, and I don't have a firewall in between these machines.
Many thanks for your help.
This was solved by passing a host and port variable when running the ckan server. Adding -H 0.0.0.0 to the run command enabled listening on all interfaces.
I am trying to connect my python application to kafka running on AWS EC2. I am able to connect with ec2 via terminal i check with telnet <ec2 ip> 9092. I am able to connect via this but not able to connect with python application.
Even if my python application starts with without any error with ec2 ip address, i am not able to receive any data from my kafka topic from ec2 to local machine.
When i add my pulic ip address to:
advertised.listeners=PLAINTEXT://<local ip addrss>:9092
Debezium connector with kafka-connect won't start , but without enabling advertised.listeners it works.
How do i configure kafka and kafka-connect so that i can consume kafka topic from ec2 instance on my local machine?
You need to set advertised.listeners to be the EC2 Public DNS/IP, restart the broker, then open the VPC / firewall connection on the listening port.
Debezium's rest.advertised.listener property is different from Kafka broker's, and you woudn't need it set on your local machine.
Python and Kafka Connect should share the same bootstrap.server protocol
You can test your listeners better using kafkacat -L -b <bootstrap>:9092
I have a compute engine instance running on Google cloud platform.
I would like to use the Python interpreter of the compute engine as a remote interpreter with Pycharm. This means that I would be using Pycharm on my local machine and running computations remotely.
Any clue on how to achieve this?
The following requires, as James Hirschhorn pointed out, the Professional verison of PyCharm.
Assign a public IP to the remote machine on GCP.
Run gcloud compute config-ssh to automatically add the VMs of your project to your ~/.ssh/config or manually add the public IP of your VM to it. If you skipped step 1. then you have to run gcloud compute config-ssh every time you re-start the remote VM, because it always gets a new IP assigned. The ~/.ssh/config gets populated with many entries in the following format:
Host the-vm-host-name # use this in PyCharm's Host field
HostName 123.456.789.00 # the VM's IP address
Use the Host name of the remote you want to connect in your Deployment configuration in PyCharm
Add a remote interpreter: select the remote server from the drop-down (the one previously created) and point PyCharm to the executable python of your Python installation.
Done
My understanding is that you need the Pycharm Ultimate Edition to support remote servers. If you have Ultimate, then you can follow these instructions.
It's fairly easy to accomplish.
You need:
PyCharm Pro
Create and format SSH keys
Config your Compute Engine instance with the SSH keys
Configure PyCharm
You can follow this tutorial that I wrote.
***Sorry for using the Term Host & Guest I see that has confused things slightly. By Host I mean Client which listens on your desired port for incoming connections and by Guest I mean Server which attempts to retrieve the IP of your pre-defined Dyndns.org address. Then tries a connection. You may say that's 'backwards' but for my applications scenario that's the fitting method. Since I'm in the United Kingdom my public IP I mentioned starts with 91.
Here's two diagrams to explain if the Server connections to the Client running on the same OS.
Ubuntu - Linux
Connection over 127.0.0.1 = SUCCESS!
Connection over public IP = SUCCESS!
Windows - Laptop TinyXP - Laptop Vista - VM TInyXP
Connection over 127.0.0.1 = SUCCESS!
Connection over public IP = FAILED!
So the problem is Windows. I've tried the Twisted examples and again they work over local host but not via public IP on Windows so it can't be my script. It can't be the firewall as the two TinyXP installations don't have the default firewall, whilst the Vista laptop did have the default shipped one however I unblocked the process after it popped up prompting me. and thus did not yield any better results.
To forward the port I went to my Orange Livebox's configuration page (192.168.1.1) and forwarded the TCP port 4005. Just as I did for when I tested it on Ubuntu (Instead using the 4004 port) so it can't be that either. I attempted to use wireshark (Found it hard to understand). I started capturing packets on all ports, then fired up the Client followed by the Server. Resulting entries are here http://pastebin.com/ddytpydz
Alas I have no more ideas on what I can do. The twisted version I'm using is one of the latest whilst the Python version is 2.6 Any one have some ideas?
So maybe there is some windows limitations for listenings pors. As far as I remeber some Windows can have max 5 ports open for listening. But I don't remeber exacly.
Sounds like a firewall problem. I would add either the port or python.exe to the approved list.
Are any of the Windows machines joined to a domain?