I have tried this tutorial and have successfully consumed kafka topics that are published on a server at my work place. I am not the producer, just purely a consumer. However, the code in that tutorial is to stream in a terminal. Now I want to try it with Python and record the messages into text file (or something of that sort).
This is the code I use, after reading a few more threads and tutorials (such as here):
from kafka import KafkaConsumer
bootstrap_servers = ['xxxxxx:xxxx']
topicName: = 'my_topic_name'
consumer = KafkaConsumer(topicName, group_id='group1', bootstrap_servers=bootstrap_servers, consumer_timeout_ms=1000)
for msg in consumer:
print(msg.value)
Here I want to first print out the message. But I get this error after 1000ms timeout:
kafka.errors.NoBrokersAvailable: NoBrokersAvailable
which sounds logical to me, since a broker is needed and the code above does not seem to do anything with a broker.
If I don't set the consumer_timeout_ms=1000ms, the Python console get just stuck without displaying anything.
How do I resolve this?
More details:
I am doing the following in parallel:
1 - Run zookeeper in one terminal
2 - Run kafka cluster in another terminal
3 - Streaming the topics (nicely) in another terminal with the command kafka-consumer-console
4 - In another terminal, run the Python code in this question.
All of these terminals are Ubuntu in WLS2 (Windows).
If you're able to use the WSL terminal with kafka-console-consumer, then running Python code there should work the same.
If you're connecting to a remote Kafka server, chances are the WSL2 network settings are simply not able to reach that address. (multiple issues elsewhere talk about WSL2 and not having external internet access) . Therefore, you should really consider running Python code on the Windows host itself. Otherwise, sounds like you'll need to adjust your network configurations.
The for loop will wait for new messages to a topic, not read existing data until you add another parameter to the consumer to tell it to
FWIW, you can use kafka-console-consumer ... >> file.txt to write out to a file
Background
I have created an Airflow webserver using a Composer Environment within Google Cloud Platform. i.e. 3 nodes, composer-1.10.0-airflow-1.10.6 image version, machine type n1-standard-1.
I have not yet configured any networks for this environment.
The Airflow works fine for simple test DAGs, i.e.:
The problem
I wrote a ping_ip DAG for determining whether a physical machine (i.e. my laptop) is connected to the internet. (Code: https://pastebin.com/FSBPNnkP)
I tested the python used to ping the machine locally (via response = os.system("ping -c 1 " + ip_address)) and it returned 0, aka Active Network.
When I moved this code into an Airflow DAG, the code ran fine, but this time returned 256 for the same IP address.
Here's the DAG code in a pastebin:
https://pastebin.com/FSBPNnkP
Here are the Airflow Logs for the triggered DAG pasted above:
[2020-04-28 07:59:35,671] {base_task_runner.py:115} INFO - Job 2514: Subtask ping_ip 1 packets transmitted, 0 received, 100% packet loss, time 0ms
[2020-04-28 07:59:35,673] {base_task_runner.py:115} INFO - Job 2514: Subtask ping_ip [2020-04-28 07:59:35,672] {logging_mixin.py:112} INFO - Network Error.
[2020-04-28 07:59:35,674] {base_task_runner.py:115} INFO - Job 2514: Subtask ping_ip [2020-04-28 07:59:35,672] {python_operator.py:114} INFO - Done. Returned value was: ('Network Error.', 256)
I guess I have Networking issues for external IPs in my server.
Does anybody know how to ping an external IP from within an Airflow Service managed by GCP?
The end goal is to create a DAG that prompts a physical machine to run a python script. I thought this process should start with a simple sub-DAG that checks to see if the machine is connected to the internet. If I'm going about this the wrong way, please lemme know.
What worked for me is removing the response part. Here's the code:
import os
def ping_ip():
ip_address = "8.8.8.8" # My laptop IP
response = os.system("ping -c 1 " + ip_address)
if response == 0:
pingstatus = "Network Active."
else:
pingstatus = "Network Error."
print("\n *** Network status for IP Address=%s is : ***" % ip_address)
print(pingstatus)
return pingstatus
print(ping_ip())
Let me give my opinion.
Composer by default uses the default network that contains a firewall rule that allow ICMP protocol (ping). So, I think any public IP should be reached out, for example, when PYPI packages are installed you usually don't configure anything special, the PYPI repositories are accessible.
A machine that has Internet access not necessarily means that it has a public IP, e.g. it can be under NAT or any other network configuration (network is not my expertise). To ensure you are specifying the public address of your Internet connection, you can use tools like https://www.myip.com/, where you will see the Public IP (e.g 189.226.116.31) and your Host IP (e.g. 10.0.0.30), if you get something similar, you will need to use the public one.
If you are using the Host IP, it is possible that it is working locally as that IP is reachable from the same private network you are in, the traffic is not going outside the network. But in the case of Composer where your DAG was uploaded, the nodes are completely outside of your local network.
I didn't find what the ping code 256 could mean, but if you are using the correct public IP, you can try increasing the timeout of the response with -W, it is probably only taking more time to reach out the IP.
The VMs created by Composer are unlikely to have "ping" installed. These are standard images. I think you are basically invoking the Linux "ping" command and it fails because it is not installed in the node/vm. So you need to change your implementation to "ping" the server another way.
You can SSH to the Composer node VMs and install "ping" and then rerun the DAG. But even if it works I would not consider it a clean solution and it will not scale. But it is okay to do this for a pilot.
Lastly, if your goal is to execute a Python script have you thought of using a Python Operator from within a DAG. If you want to somehow decouple the execution of Python script from the DAG itself, an alternative is to use a PubSub + CloudFunction combination.
Other probable causes for being unable to reach External IPs is misconfigured firewall rules. To fix this you must:
Define an egress firewall rule to enable ping to your destination IP and attach the firewall rule to a "tag".
Make sure you attach the same "tag" to the VMs/nodes created for Composer.
I've been trying to use various Python libraries for working with Connman and the dbus, particularly this sample code:
https://github.com/liamw9534/pyconnman/blob/master/demo/demo.py
The problem I have is that when connecting to a WPA2 access point for the very first time, I will always receive a timeout message. For example:
CONN> list-services
CONN> agent-start /test/agent ssid=myNetwork passphrase=myPassphrase
CONN> service-connect /net/connman/service/wifi_xxxxx__managed_psk
Eventually this is the message I receive back from the interface:
Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken
I can confirm at this point that Connman has no connected to a wifi network or obtained an IP address. The only way I can manage to get this to work is by using the Connman application itself from a Linux terminal:
connmanctl
connmanctl> agent on
connmanctl> connect wifi_xxxxx__managed_psk
Agent RequestInput wifi_xxxxx__managed_psk
Passphrase = [ Type=psk, Requirement=mandatory ]
Passphrase? myPassword
connmanctl> Connected wifi_xxxxx__managed_psk
This creates a settings folder under /var/lib/connman for the wifi network. I can now use the demo.py script mentioned above to either disconnect or reconnect.
Connman is still a bit of a mystery to me in many ways, and I'm not sure why I have to use the interactive shell to connect to a network for the first time. Any ideas?
In case you're still looking for the answer :
Connman needs an agent to answer the security questions (in the WPA2: it's the password). You can either run an agent and reply to Connman questions or you can create a file in /var/lib/connman with the right keys. See here. Once a file is created or deleted Connman will auto magically act accordingly (try to connect or disconnect.
A basic file would look like:
[service_mywificonfig]
Type = wifi
Security = wpa2
Name = myssid
Passphrase = yourpass
I tried to do game(actually at least to run it on my computer) that I saw in one tutorial. Unfortunately, their forum is not frequently visited, so maybe here I can found the solution.
But when I try to run it on different computers, which are connected to one Wi-fi, client on the second computer(local server is on the first) doesn't find server:
I do this way:
Start server and set to localhost:8000
Start the first client and type localhost:8000, after what it connects to server
In cmd type ipconfig, find IPv4,
Start the second client, and type [IPv4]:8000, but it isn't connecting to server
What have I to do, to make it work?
Here are my code samples:
Client
Server
If it is needed, I can also share PodSixNet files too.
After investigationg with you in the comments, it's clear now that your server is running on the local loopback (i.e. 127.0.0.1), therefore you can't access it from another interface (e.g. [IPv4]).
When you run server, you should set it to 0.0.0.0:8000 (which means all interfaces, port 8000) instead of localhost:8000.
I'm using:
Python 3.4.2
PyMongo 3.0.2
mongolab running mongod 2.6.9
uWSGI 2.0.10
CherryPy 3.7.0
nginx 1.6.2
uWSGI start params:
--socket 127.0.0.1:8081 --daemonize --enable-threads --threads 2 --processes 2
I setup my MongoClient ONE time:
self.mongo_client = MongoClient('mongodb://user:pw#host.mongolab.com:port/mydb')
self.db = self.mongo_client['mydb']
I try and save a JSON dict to MongoDB:
result = self.db.jobs.insert_one(job_dict)
It works via a unit test that executes the same code path to mongodb. However when I execute via CherryPy and uWSGI using an HTTP POST, I get this:
pymongo.errors.ServerSelectionTimeoutError: No servers found yet
Why am I seeing this behavior when run via CherryPy and uWSGI? Is this perhaps the new thread model in PyMongo 3?
Update:
If I run without uWSGI and nginx by using the CherryPy built-in server, the insert_one() works.
Update 1/25 4:53pm EST:
After adding some debug in PyMongo, it appears that topology._update_servers() knows that the server_type = 2 for server 'myserver-a.mongolab.com'. However server_description.known_servers() has the server_type = 0 for server 'myserver.mongolab.com'
This leads to the following stack trace:
result = self.db.jobs.insert_one(job_dict)
File "/usr/local/lib/python3.4/site-packages/pymongo/collection.py", line 466, in insert_one
with self._socket_for_writes() as sock_info:
File "/usr/local/lib/python3.4/contextlib.py", line 59, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.4/site-packages/pymongo/mongo_client.py", line 663, in _get_socket
server = self._get_topology().select_server(selector)
File "/usr/local/lib/python3.4/site-packages/pymongo/topology.py", line 121, in select_server
address))
File "/usr/local/lib/python3.4/site-packages/pymongo/topology.py", line 97, in select_servers
self._error_message(selector))
pymongo.errors.ServerSelectionTimeoutError: No servers found yet
We're investigating this problem, tracked in PYTHON-961. You may be able to work around the issue by passing connect=False when creating instances of MongoClient. That defers background connection until the first database operation is attempted, avoiding what I suspect is a race condition between spin up of MongoClient's monitor thread and multiprocess forking.
As mentioned here: https://stackoverflow.com/a/54314615/8953378
I added ?ssl=true&ssl_cert_reqs=CERT_NONE to my connection string, and it fixed the issue.
so instead of:
connection_string = "mongodb+srv://<USER>:<PASSWORD>#<CLUSTER>/<COLLECTION>"
I wrote:
connection_string = "mongodb+srv://<USER>:<PASSWORD>#<CLUSTER>/<COLLECTION>?ssl=true&ssl_cert_reqs=CERT_NONE"
(Note that if you have other parameters in your connection string, you need to change the ? to & )
I am not sure if you are using the MongoDB paired with AWS Cloud service. But if you are, I found that you have to specify which IP Address you want MongoDB to have access to.
So what you need to do is add the IP Address of your host server to allow entry.
In MongoAtlas, this can be done at this page
I know there was already a solution to the same issue, but I didn't find a solution that helped my situation, so wanted to post this, so others could benefit if they ever face the same problem that I do.
I fixed it for myself by downgrading from pymongo 3.0 to 2.8. No idea what's going on.
flask/bin/pip uninstall pymongo
flask/bin/pip install pymongo==2.8
I had the same problem with Pymongo 3.5
Turns out replacing localhost with 127.0.0.1 or corresponding ip address of your mongodb instance solves the problem.
I solved this by installing dnspython (pip install dnspython). The issue is that: "The "dnspython" module must be installed to use mongodb+srv:// URIs"
Go to your Atlas Console > Network Access, then add your client IP address,
Ex. 0.0.0.0/00
(Note: All client ips can access your database)
In my case
I was using Mongo Atlas
I got another IP adress after a router reboot
hence I had to add that IP to the whitelist on Mongo Atlas settings via
MongoAtlas website -> Network Access -> IP Whitelist -> Add IP Address -> Add Current IP Address
then wait for IP Address's status to change to Active and then try to run the app again
if you are using a repl.it server to host, just add the host ip you used to configure your server, for me it was 0.0.0.0, which is the most common
I was facing the same exception today. In my case, the proxy settings were probably blocking the connection since I could establish a successful connection to the mongodb by changing my wifi. Even if this question is marked as solved already, it can hopefully narrow down the problem for some others.
I've come accross the same problem and finally I found that the client IP is blocked by the firewall of the mongo server.
I encountered this too.
This could be due to pymongo3 isn't fork safe.
I fix this by adding --lazy-apps param to uwsgi, this can avoid the "fork safe" problem.
seeing uwsgi doc preforking-vs-lazy-apps-vs-lazy.
Notice, no sure for this two having positive connection.
I simply added my current IP address in the network access tab, as it got changed automatically. Deleted the earlier one, there was a slight change in IP address.
pymongo 3 will not tell you your connection failed when you instantiate your client. You may not be connected.
https://api.mongodb.com/python/3.5.1/api/pymongo/mongo_client.html
"it no longer raises ConnectionFailure if they are unavailable ..
You can check if the server is available like this:"
from pymongo.errors import ConnectionFailure
client = MongoClient()
try:
# The ismaster command is cheap and does not require auth.
client.admin.command('ismaster')
except ConnectionFailure:
print("Server not available")
maybe you can try to add your server ip address into the mongod.conf file.
if you use linux(ubuntu) os,you can try my solution:
modify mongod.conf file:
vi /etc/mongod.conf
and you can add mongodb server ip address behind 127.0.0.1,and save:
net:
port:27017
bindIp:127.0.0.1,mongodb server ip
in the teminal:
sudo service mongod restart
Now,you can try to connect mongodb by using pymongo MongoClient.
That error has occurred because there is no MongoDB server running in the background. To run the MongoDB server open cmd or anaconda prompt and type this:-
"C:\Program Files\MongoDB\Server\3.6\bin\mongod.exe"
then run
import pymongo
myclient = pymongo.MongoClient()
mydb = myclient["mydatabase"]
myclient.list_database_names()
I'm using pymongo 3.2 and I run into the same error, however it was a missconfiguration in my case. After enabling authorization, I forgot to update the port in the url which ended up in a connection timout. Probably it is worth to mention that ?authSource might be required as it is typically different than the database storing the application data.
I commented out bindIP variable in mongod.conf instead of allowing all connections (for which you have to enter 0.0.0.0). Of course, beware of the consequence.
The developers are investigating this problem, tracked in PYTHON-961. You may be able to work around the issue by running mongod.exe manually and monitoring it. This issue arises when the console freezes and you can hit the enter if the mongod console is got stuck. This is the simplest solution for now until the developers fix this bug.
I ran into the same issue during development. It was due to the fact that mongodb wasn't running on my local machine (sudo systemctl restart mongod to get mongodb running on Ubuntu).
I faced the same error on windows and I just started the MongoDB service
open services ctrl+R then type services.msc then Enter
For my case I only set my ip allow list 0.0.0.0 allow anywhere but you can set your ip using "what is my ip" and copy paste it to network access > add ip
I have been struggling with same problem. Read and either insert did not work at all failed with ServerSelectionTimeoutError.
I have been using pymongo==3.11.4 on Ubuntu 18.04 LTS.
Tried use connect=False, pass extra ?ssl=true&ssl_cert_reqs=CERT_NONE options to my connection string and other suggestions listed above. In my case they didn't work.
Finally simple tried to upgrade to pymongo==3.12.1 and connection started to work without passing connect=false, and other extra arguments suggested.
login = '<USERNAME>'
password = '<PASSWORD>'
host = '*.mongodb.net'
db = '<DB>'
uri = f'mongodb+srv://{login}:{password}#{host}/{db}?retryWrites=true&w=majority'
client = MongoClient(uri, authsource='admin')#, connect=False)
collection = client.db.get_collection('collection_name')
# t = collection.find_one({'hello': '1'})
t = collection.insert_one({'hello': '2'})
print(t)
Make sure you entered the user password, not the MongoDB account password. I encountered similar issue. In my case, I mistakenly entered the MongoDB account password instead of the user password.
I had this issue today. I managed to deal with it by:
installing dnspython library > going to MongoDB webpage > signing in > security > network access > add IP address > adding the IP address from where my request comes from.
Hope this could help someone.
I had the same issue..the code that was working perfectly fine 2 minutes before gave this error. I was looking for solutions over google for about 30 minutes and it automatically got fixed. The problem could be my home internet connection. Just a guess but if you haven't made any changes to the code or any other config file best to wait for sometime and retry.
I was also facing the same issue. Then, I added
import certifi
Client = MongoClient("mongodb+srv://<username>:<password>#cluster0.ax9ugoz.mongodb.net/?retryWrites=true&w=majority", tlsCAFile=certifi.where())
and it solved my issue.
Certifi provides a collection of Root Certificates for validating the trustworthiness of SSL certificates while verifying the identity of TLS hosts.
This has been fixed in PyMongo with this pull_request.
This problem solved when I just toggled the MongoDB in the services to running which was stopped previously.
First set up the MongoDB environment.
Run this on CMD - "C:\Program Files\MongoDB\Server\3.6\bin\mongod.exe"
Open another CMD and run this - "C:\Program Files\MongoDB\Server\3.6\bin\mongo.exe"
And then you can use pymongo [anaconda prompt]
import pymongo
from pymongo import MongoClient
client = MongoClient()
db = client.test_db
collection = db['test_coll']
Refer - https://docs.mongodb.com/tutorials/install-mongodb-on-windows/