I am attempting to create a simple producer and consumer with two Python scripts, using Kafka deployed on Microk8s. However, when running the producer.py script, I get the following error on repeat:
...|FAIL|rdkafka#producer-1| [thrd:...:9092/bootstrap]: ...:9092/bootstrap: Connect to ipv4#localhost:9092 failed: Connection refused (after 0ms in state CONNECT, ... identical error(s) suppressed
I am fairly confident that this issue is a result of the listeners not being configured correctly, but I have so far been unable to figure out what I need to do to fix them, due to what I assume is my complete lack of any knowledge in this area. I have reviewed these resources, in addition to several others from this site, but have been unable to find a solution, or at least a solution I can understand enough to act upon.
Steps to Reproduce:
The Python scripts to generate the producer and consumer can be found here.
For Microk8s installation, I followed these instructions. I also installed Helm, since my project requirements dictate that I use Helm charts.
I then installed Kafka using:
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install kafka-release bitnami/kafka
The Python code in the linked post uses 'localhost:9092', as the error also shows - Connect to ipv4#localhost:9092 failed
If you are trying to run that code in a k8s pod, then you need to give the external broker DNS addresses, not the local pod address.
If you run the Python code from outside the k8s cluster, you need to expose a ClusterIP / NodePort external service or Ingress (as the linked Strimzi post shows; plus, you can can still use Strimzi Operator with Helm, so you don't really need the Bitnami Charts).
At a high level, the advertisted.listeners tells clients how to connect to a specific broker. If you advertise localhost, the pod will try to connect to itself, even if the bootstrap connection worked (setup by just listeners). If you advertise kafka.svc.cluster.local, then it will try to connect to the kafka service in the default namespace... But you still need to actually set boostrap.servers = kafka.svc.cluster.local:9092, for example.
Related
I receive an error similar to this when trying to start ray using a docker image from a local repository
The proposed solutions involve making some changes to the nodes' docker config files, and then restarting the docker daemon.
However, I believe that (please correct if I'm wrong) the setup commands in ray config files are run within the docker containers, rather than directly on the node machines. So I'm unsure of how to apply them when using Ray.
How can I avoid the error?
I am new to linux/aws in general and I am trying to deploy a dash webapp onto an ec2 instance. The webapp is written in python and uses an aws database. I created an EC2 instance, set the security group to allow all traffic, uses the default VPC and internet gateway. I successfully installed the all the app dependencies but anytime I run the app.py file. The public dns doesnt load the webpage. I have tried pinging the public IP and that works. I really have a limited knowledge base hear and have tried different options but cant seem to get it working. Please help :)
Public IP-https://ec2-3-8-100-74.eu-west-2.compute.amazonaws.com/
security group
webapp
I've been smacking my head on this for a couple days and finally got it. I know it's been a while but hopefully this helps someone else. Had a hard time finding answers elsewhere. Very similar to you, I had the ec2 instance set up, the security groups and vpc set up (those steps aren't too difficult and are well-documented). I had some successful pings, but was getting a "connection refused" error through the browser.
The "app.run_server()" parameters were the missing piece for me:
if __name__ == '__main__':
app.run_server(host= '0.0.0.0',port=80)
At that point calling the .py app gave me a 'permission denied,' which I was able to get around by running as sudo ("sudo python3 my_app.py") -- and by sudo pip install-ing necessary packages. (All through ssh, fwiw).
After finally running successfully I was given an IP from the dash app corresponding to my private IPv4 on EC2, and at that point could set my browser to the PUBLIC IPv4 and get to the app. Huzzah.
Playing around with it a little, it looks like as long as you have:
host= '0.0.0.0'
you'll run it online. Without that, it runs only locally (you'll see IP as 127.0.0.1). Then it's a matter of making sure whatever port you're using (:80, :443, :8050) is open according to firewalls and security groups. Dash for me defaults to :8050, and that port might be fine as long as it's allowed through security groups.
QUICK UPDATE:
I tried leaving it on port :8050, and also opened :8050 to all ipv4 in my security group. That let me run everything successfully without using "sudo python3".
if __name__ == '__main__':
app.run_server(host= '0.0.0.0',port=80)
With "python3 my_app.py" in ssh
I have installed Kafka with all default settings on my Ubuntu partition
On my windows partition on the same machine, using python, I have followed the steps of this guide.
Producer1.py creates the topic 'First_Topic' in my Kafka
But Consumer1.py does not show any information and times out after some time.
When I access Kafka from my ubuntu terminal, am able to see the newly created Kafka but no messages are consumed/available.
So initially my doubt was that am not able to connect from Windows partition to Ubuntu, but as I can see that the Kafka topic has got created then it shouldn't be that case.
Only change in code is the localhost which I've replaced as bootstrap_servers = ['DESKTOP-MyUbuntu:9092'] from the original bootstrap_servers = ['localhost:9092']
What changes do I need to do to read the kafka messages in my consumer1.py?
As mentioned in the comments, you'll need to modify advertised.listeners to expose your external desktop hostname or IP (run ipconfig and use the address given by the router) from the WSL2 environment. By default Kafka only accepts local connections (local, here, meaning within the same WSL hypervisor)
There's also some firewall or port forwards on WSL2 that needs adjusted - there's some answers here on it (use port 9092 for Kafka)
I am attempting to connect to a remote MySQL DB from a virtual machine (Ubuntu, running on Azure).
When I access the DB from my computer via the command line, I enter:
mysql -u username -h www.foobar.nyc -p
Which prompts me for the password. When I enter the password, it successfully logs me in to the remote db.
Now when I perform the same actions as above, but instead from a remote vm that I have ssh into, I get the following error returned after entering my pw.
ERROR 1129 (HY000): Host 'xxx.xx.xxx.xxx' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'
Googling this error brings me to: http://dev.mysql.com/doc/refman/5.7/en/blocked-host.html
I'd like to understand why I am getting so many connection errors - is that normal? Is there a setting perhaps with Azure that I need to look into? I know Azure has the endpoints manager. When using the Python mysql library, it reads that errors occurring File "/site-packages/mysql/connector/connection.py", line 418, in _open_connection on self._do_handshake()
What I hope to gain from this question:
Understand why "many" errors are occurring - what is causing this and is receiving such high numbers of errors normal (as some comments in the MySQL documentation I linked to appear to suggest).
Understand the differences that enable the same actions to work from local in command line but not from command line when ssh into remote Azure based vm.
Thanks.
1.Understand why "many" errors are occurring - what is causing this and is receiving such high numbers of errors normal (as some comments in the MySQL documentation I linked to appear to suggest).
Per my experience, you can check the mysql error logs, details on enabling error logs are # How to see log files in MySQL?.
2.Understand the differences that enable the same actions to work from local in command line but not from command line when ssh into remote Azure based vm.
Based on the latest comment you provided, the new created VM was not experiencing the same issue, it may not be an azure platform specific related issue.
I am looking to host a basic Websocket server.
The code I want to see running is : FastFlicker
Do you know how and where I can host this application online?
PythonAnywhere dev here. Unfortunate we can't host websocket-based apps on our site right now. The toggle you spotted enables/disables websockets for our in-browser consoles, it's not related to running your own websocket server.
I've added an upvote for websockets to our own issue tracker, but for now you'll have to use a different service :-(
Your solution is OpenShift, even with the free plan you can host FastFlicker.
Click Add Application, choose the good cartridges (Python 2.7).
Then use your gitHub repo url to get the source.
Once the application is running, you need to SSH it to change the address and the port (see this Post).
To be able to ssh you need first to generate a ssh key and to add it in setting on the website
Ok, now kill all processes that uses your port. (lsof -i :8080)
Start your application and now it's working!
(It is in app-deployments/current/repo/ for me, then python FastFlicker.py &)
It is currently hosted here : ws://main-fastflicker.rhcloud.com:8000/
And to test it, you know you can use this generic client..