We Keep Coding

Kafka Producer Function is not producing messages to Kafka via Google Cloud Functions

I am using GCP with its Cloud Functions to execute web scrapers on a frequent basis. Also locally, my script is working without any problems.
I have a setup.py file in which I am initializing the connection to a Kafka Producer. This looks like this:
p = Producer(
{
"bootstrap.servers": os.environ.get("BOOTSTRAP.SERVERS"),
"security.protocol": os.environ.get("SECURITY.PROTOCOL"),
"sasl.mechanisms": os.environ.get("SASL.MECHANISMS"),
"sasl.username": os.environ.get("SASL.USERNAME"),
"sasl.password": os.environ.get("SASL.PASSWORD"),
"session.timeout.ms": os.environ.get("SESSION.TIMEOUT.MS")
}
)
def delivery_report(err, msg):
"""Called once for each message produced to indicate delivery result.
Triggered by poll() or flush()."""
print("Got here!")
if err is not None:
print("Message delivery failed: {}".format(err))
else:
print("Message delivered to {} [{}]".format(msg.topic(), msg.partition()))
return "DONE."
I am importing this setup in main.py in which my scraping functions are defined. This looks similar to this:
from setup import p, delivery_report
def scraper():
try:
# I won't insert my whole scraper here since it's working fine ...
print(scraped_data_as_dict)
p.produce(topic, json.dumps(scraped_data_as_dict), callback=delivery_report)
p.poll(0)
except Exception as e:
# Do sth else
The point here is: I am printing my scraped data in the console. But it doesn't do anything with the producer. It's not even logging an failed producer message (deliver_report) on the console. It's like my script is ignoring the producer command. Also, there are no Error reports in the LOG of the Cloud Function. What am I doing wrong since the function is doing something, except the important stuff? What do I have to be aware of when connection Kafka with Cloud Functions?

Request timed out: timeout('timed out') in Python's HTTPServer

I am trying to create a simple HTTP server that uses the Python HTTPServer which inherits BaseHTTPServer. [https://github.com/python/cpython/blob/main/Lib/http/server.py][1]
There are numerous examples of this approach online and I don't believe I am doing anything unusual.
I am simply importing the class via:
"from http.server import HTTPServer, BaseHTTPRequestHandler"
in my code.
My code overrides the do_GET() method to parse the path variable to determine what page to show.
However, if I start this server and connect to it locally (ex: http://127.0.0.1:50000) the first page loads fine. If I navigate to another page (via my first page links) that too works fine, however, on occasion (and this is somewhat sporadic), there is a delay and the server log shows a Request timed out: timeout('timed out') error. I have tracked this down to the handle_one_request method in the BaseHTTPServer class:
def handle_one_request(self):
"""Handle a single HTTP request.
You normally don't need to override this method; see the class
__doc__ string for information on how to handle specific HTTP
commands such as GET and POST.
"""
try:
self.raw_requestline = self.rfile.readline(65537)
if len(self.raw_requestline) > 65536:
self.requestline = ''
self.request_version = ''
self.command = ''
self.send_error(HTTPStatus.REQUEST_URI_TOO_LONG)
return
if not self.raw_requestline:
self.close_connection = True
return
if not self.parse_request():
# An error code has been sent, just exit
return
mname = 'do_' + self.command ## the name of the method is created
if not hasattr(self, mname): ## checking that we have that method defined
self.send_error(
HTTPStatus.NOT_IMPLEMENTED,
"Unsupported method (%r)" % self.command)
return
method = getattr(self, mname) ## getting that method
method() ## finally calling it
self.wfile.flush() #actually send the response if not already done.
except socket.timeout as e:
# a read or a write timed out. Discard this connection
self.log_error("Request timed out: %r", e)
self.close_connection = True
return
You can see where the exception is thrown in the "except socket.timeout as e:" clause.
I have tried overriding this method by including it in my code but it is not clear what is causing the error so I run into dead ends. I've tried creating very basic HTML pages to see if there was something in the page itself, but even "blank" pages cause the same sporadic issue.
What's odd is that sometimes a page loads instantly, and almost randomly, it will then timeout. Sometimes the same page, sometimes a different page.
I've played with the http.timeout setting, but it makes no difference. I suspect it's some underlying socket issue, but am unable to diagnose it further.
This is on a Mac running Big Sur 11.3.1, with Python version 3.9.4.
Any ideas on what might be causing this timeout, and in particular any suggestions on a resolution. Any pointers would be appreciated.

After further investigation, this particular appears to be an issue with Safari. Running the exact same code and using Firefox does not show the same issue.

Whats difference between websocket and flask-streaming from scenario aspect

I'm developing a BS kafka monitor tool. The program will listen to a kafka topic, and constantly output the new message from that topic. So which is the best approach to send those message constantly to browser side?
The program uses flask, so currently I'm using the stream_with_context to send new message to browser side. This works for now, but I wonder if this is the correct scenario to use stream_with_context since most usage case is for downloading and video streaming? or maybe I should use websocket?
#read_controller.route('/v1/listenkafka/<string:kafkaId>', methods=['GET'])
def start_stream(kafkaId):
try:
mykafka_json = eval(my_storage.get(kafkaId))
mykafka = kafkaserver(ip=mykafka_json['ip'], id=kafkaId, port=mykafka_json['port'])
return Response(stream_with_context(mykafka.consume_topic(mykafka_json['topic'])))
except Exception as e:
print(f"{e}")
return jsonify(f"{e}"), 400
#The generator listen to kafka and feed to stream
def consume_topic(self, topic, groupid='test-consumer-group'):
consumer = KafkaConsumer(topic,
group_id=groupid,
bootstrap_servers=[f"{self.ip}:{self.port}"])
print(f"Topic: {topic}#{self.ip}:{self.port} starts steaming at {datetime.now()}")
try:
for messages in consumer:
mykafka_json = eval(my_storage.get(self.id))
print(mykafka_json)
if mykafka_json['flag']:
my_storage.delete(self.id)
return
else:
message = {'topic':messages.topic,
'partition':messages.partition,
'offset':messages.offset,
'key':messages.key,
'value':messages.value}
print (message['value'])
yield message['value']
except StopIteration as e:
#TODO:: handle return
print(e)
finally:
print(f"Topic-{topic} finish at {datetime.now()}")
So, should I use stream_with_context in this scenario or should I switch to use websockt?
Thanks

Ok now I undertand。
The stream_with_context actually will return ALL contents from beginning at each time the front request.
So it is a tool for downloading, not for constantly pushing new data from server to client
Eventually, I chosed flask-socketIO, it is a better choice than websocket, but you need to study the sample to understand how it works...The doc miss some details...

Finding the real error in a Webtest test failure

I'm using Python and Webtest to test a WSGI application. I found that exceptions raised in the handler code tend to be swallowed by Webtest, which then raises a generic:
AppError: Bad response: 500 Internal Server Error
How do I tell it to raise or print the original error that caused this?

While clj's answer certainly works, you may still want to access the response in your test case. To do this, you can use expect_errors=True (from the webtest documentation) when you make your request to the TestApp, and that way no AppError will be raised. Here is an example where I am expecting a 403 error:
# attempt to access secure page without logging in
response = testapp.get('/secure_page_url', expect_errors=True)
# now you can assert an expected http code,
# and print the response if the code doesn't match
self.assertEqual(403, response.status_int, msg=str(response))

Your WSGI framework and server contains handlers which catch exceptions and performs some action (render a stacktrace in the body, log the backtrace to a logfile, etc). Webtest, by default, does not show the actual response, which might be useful if your framework renders a stacktrace in the body. I use the following extension to Webtest when I need to look at the body of the response:
class BetterTestApp(webtest.TestApp):
"""A testapp that prints the body when status does not match."""
def _check_status(self, status, res):
if status is not None and status != res.status_int:
raise webtest.AppError(
"Bad response: %s (not %s)\n%s", res.status, status, res)
super(BetterTestApp, self)._check_status(status, res)
Getting more control over what happens to the exception depends on what framework and server you are using. For the built in wsgiref module you might be able to override error_output to achieve what you want.

Python - How to check if Redis server is available

I'm developing a Python Service(Class) for accessing Redis Server. I want to know how to check if Redis Server is running or not. And also if somehow I'm not able to connect to it.
Here is a part of my code
import redis
rs = redis.Redis("localhost")
print rs
It prints the following
<redis.client.Redis object at 0x120ba50>
even if my Redis Server is not running.
As I found that my Python Code connects to the Server only when I do a set() or get() with my redis instance.
So I dont want other services using my class to get an Exception saying
redis.exceptions.ConnectionError: Error 111 connecting localhost:6379. Connection refused.
I want to return proper message/Error code. How can I do that??

If you want to test redis connection once at startup, use the ping() command.
from redis import Redis
redis_host = '127.0.0.1'
r = Redis(redis_host, socket_connect_timeout=1) # short timeout for the test
r.ping()
print('connected to redis "{}"'.format(redis_host))
The command ping() checks the connection and if invalid will raise an exception.
Note - the connection may still fail after you perform the test so this is not going to cover up later timeout exceptions.

The official way to check if redis server availability is ping ( http://redis.io/topics/quickstart ).
One solution is to subclass redis and do 2 things:
check for a connection at instantiation
write an exception handler in the case of no connectivity when making requests

As you said, the connection to the Redis Server is only established when you try to execute a command on the server. If you do not want to go head forward without checking that the server is available, you can just send a random query to the server and check the response. Something like :
try:
response = rs.client_list()
except redis.ConnectionError:
#your error handlig code here

There are already good solutions here, but here's my quick and dirty for django_redis which doesn't seem to include a ping function (though I'm using an older version of django and can't use the newest django_redis).
# assuming rs is your redis connection
def is_redis_available():
# ... get redis connection here, or pass it in. up to you.
try:
rs.get(None) # getting None returns None or throws an exception
except (redis.exceptions.ConnectionError,
redis.exceptions.BusyLoadingError):
return False
return True
This seems to work just fine. Note that if redis is restarting and still loading the .rdb file that holds the cache entries on disk, then it will throw the BusyLoadingError, though it's base class is ConnectionError so it's fine to just catch that.
You can also simply except on redis.exceptions.RedisError which is the base class of all redis exceptions.
Another option, depending on your needs, is to create get and set functions that catch the ConnectionError exceptions when setting/getting values. Then you can continue or wait or whatever you need to do (raise a new exception or just throw out a more useful error message).
This might not work well if you absolutely depend on setting/getting the cache values (for my purposes, if cache is offline for whatever we generally have to "keep going") in which case it might make sense to have the exceptions and let the program/script die and get the redis server/service back to a reachable state.

I have also come across a ConnectionRefusedError from the sockets library, when redis was not running, therefore I had to add that to the availability check.
r = redis.Redis(host='localhost',port=6379,db=0)
def is_redis_available(r):
try:
r.ping()
print("Successfully connected to redis")
except (redis.exceptions.ConnectionError, ConnectionRefusedError):
print("Redis connection error!")
return False
return True
if is_redis_available(r):
print("Yay!")

Redis server connection can be checked by executing ping command to the server.
>>> import redis
>>> r = redis.Redis(host="127.0.0.1", port="6379")
>>> r.ping()
True
using the ping method, we can handle reconnection etc. For knowing the reason for error in connecting, exception handling can be used as suggested in other answers.
try:
is_connected = r.ping()
except redis.ConnectionError:
# handle error

Use ping()
from redis import Redis
conn_pool = Redis(redis_host)
# Connection=Redis<ConnectionPool<Connection<host=localhost,port=6379,db=0>>>
try:
conn_pool.ping()
print('Successfully connected to redis')
except redis.exceptions.ConnectionError as r_con_error:
print('Redis connection error')
# handle exception