We Keep Coding

Distributed python: Celery send_task gets COMMAND_INVALID

Context
I developed a Flask API that sends tasks to my computing environment.
To use this, you should make a post request to the API.
Then, the API received your request, process it and send necessary data, through the RABBITMQ broker, a message to be held by the computing environment.
At the end, it should send the result back to the API
Some code
Here is an example of my API and my Celery application:
#main.py
# Package
import time
from flask import Flask
from flask import request, jsonify, make_response
# Own module
from celery_app import celery_app
# Environment
app = Flask()
# Endpoint
#app.route("/test", methods=["POST"])
def test():
"""
Test route
Returns
-------
Json formatted output
"""
# Do some preprocessing in here
result = celery_app.send_task(f"tasks.Client", args=[1, 2])
while result.state == "PENDING":
time.sleep(0.01)
result = result.get()
if result["sucess"]:
result_code = 200
else:
result_code = 500
output = str(result)
return make_response(
jsonify(
text=output,
code_status=result_code, ),
result_code,
)
# Main thread
if __name__ == "__main__":
app.run()
In a different file, I have setup my celery application connected to RABBITMQ Queue
#celery_app.py
from celery import Celery, Task
celery_app = Celery("my_celery",
broker=f"amqp://{USER}:{PASSWORD}#{HOSTNAME}:{PORT}/{COLLECTION}",
backend="rpc://"
)
celery_app.conf.task_serializer = "pickle"
celery_app.conf.result_serializer = "pickle"
celery_app.conf.accept_content = ["pickle"]
celery_app.conf.broker_connection_max_retries = 5
celery_app.conf.broker_pool_limit = 1
class MyTask(Task):
def run(self, a, b):
return a + b
celery_app.register_task(MyTask())
To run it, you should launch:
python3 main.py
Do not forget to run the celery worker (after registering tasks in it)
Then you can make a post request on it:
curl -X POST http://localhost:8000/test
The problem to resolve
When this simple API is running, I am sending request on my endpoint.
Unfortunatly, it fails 1 time on 4.
I have 2 messages:
The first message is:
amqp.exceptions.PreconditionFailed: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more
Then, because of the time out, my server has lost the message so:
File "main.py", line x, in test
result = celery_app.send_task("tasks.Client", args=[1, 2])
amqp.exceptions.InvalidCommand: Channel.close_ok: (503) COMMAND_INVALID - unimplemented method
Resolve this error
There are 2 solutions to get around this problem
retry to send a tasks until it fails 5 times in a row (try / except amqp.exceptions.InvalidCommand)
change the timeout value.
Unfortunatly, it doesn't seems to be the best ways to solve it.
Can you help me ?
Regards
PS:
my_packages:
Flask==2.0.2
python==3.6
celery==4.4.5
rabbitmq==latest

1. PreconditionFailed
I change my RabbitMQ version from latest to 3.8.14.
Then, I set up a celery task timeout using time_limit and soft_time_limit.
And it works :)
2. InvalidCommand
To resolve this problem, I use this retryfunctionnaluity.
I setup:
# max_retries=3
# autoretry_for=(InvalidCommand,)

Send response to EventSource after webook (Flask + uWSGI + nginx)

I have Flask application with route (webhook) receiving POST requests (webhooks) from external phone application (incomming call = POST request). This route sets threading.Event.set() and based on this event, another route (eventsource) sends an event stream to opened EventSource connection on a webpage created by yet another route (eventstream).
telfa_called = Event()
telfa_called.clear()
call = ""
#telfa.route('/webhook', methods=['GET', 'POST'])
def webhook():
global call
print('THE CALL IS HERE')
x = request.data
y = ET.fromstring(x.decode())
caller_number = y.find('caller_number').text
telfa_called.set() # setting threading.Event for another route
return Response(status=200)
#telfa.route('/eventstream', methods = ['GET','POST'])
#login_required
def eventstream():
jsid = str(uuid.uuid4())
return render_template('telfa/stream.html', jsid=jsid)
def eventsource_gen():
while 1:
if telfa_called.wait(10):
telfa_called.clear()
print('JE TO TADY')
yield "data: {}\n\n".format(json.dumps(call))
#telfa.route('/eventsource', methods=['GET', 'POST'])
def eventsource():
return Response(eventsource_gen(), mimetype='text/event-stream')`
Everything works great when testing in pure Python application. The problem starts, when I move this to production server, where I use uWSGI with nginx. (Other parts of this Python application work without any troubles.)
When the eventSource connection is opened and incomming webhook should be processed, whole flask server stucks (for all other users, too), page stops to load and I cannot find, where the error is.
I only know, the POST request from external application is received, but the response to EventSource is not made.
I suspect it has something to do with processes - the EventSource connection from JavaScript is one process, the webhook route another - and they do not communicate. So or so, I suppose this has to have very trivial solution, but I didn't find it in past 3 days and nights. Any hints, please? Thanks in advance.
To be complete, this my uwsgi config file:
[uwsgi]
module = wsgi:app
enable-threads = true
master = true
processes = 5
threads = 2
uid = www-data
gid= www-data
socket = /tmp/myproject.sock
chmod-socket = 666
vacuum = true
die-on-term = true
limit-as=512
buffer-size = 512000
workers = 5
max-requests = 100
req-logger = file:/tmp/uwsg-req.log
logger = file:/tmp/uwsgi.log`

Flask App with Slow Queries, Multiple Client Users, and Hosted on Kubernetes

I've got a Flask app in which I hope to accomplish the following things:
Have an endpoint that will run a series of queries
This endpoint needs to respond to the HTTP request within a limited number of seconds.
The queries can take up to several minutes to finish so I need them to run in a separate thread, with multiple clients polling the server every so often to see if they have fresh data to be returned to them
Hopefully hosted on Kubernetes with multiple instances of the pod running.
My below implementation has several issues:
The poll endpoint seems unnecesarily large, most of this is just dealing with the Queue of queries and making sure that each client gets their own results back, and not someone elses.
Not sure what is going on, but when I try to host more than one instance of this pod on Kubernetes, its like some poll requests from some users are being sent to instances in which their uuid does not exist.
I'm hoping for some understanding of what I'm doing wrong with threading and Queues because this seems like a hacky way of doing this. And also, how can I make the results of these queries available to all instances of Kubernetes running?
Thanks!
from flask import Flask, render_template, request, jsonify, g
from Queue import Queue
from threading import Thread
from time import sleep
app = Flask(__name__, template_folder='Templates')
#app.route('/')
def index():
return render_template('index.html')
#app.before_first_request
def before_first_request():
g.output = Queue()
g.data_results = {}
return ""
#app.route('/data')
def data():
"""
Endpoint hit to fire of a request for data from a given user (uuid)
"""
params = request.args.to_dict()
uuid = params['uuid']
# Create a list for this user, to store their results
g.data_results[uuid] = []
list_of_queries = ["SELECT * FROM tbl1;",
"SELECT * FROM tbl2;",
"SELECT * FROM tbl3;"]
for query in list_of_queries:
t = Thread(target=worker, args=(query, uuid, g.output))
t.daemon = True
t.start()
return jsonify({'msg':'Queries started'})
def worker(*args):
query, uuid, output = args
# Will actually be something like `result = run_query(query)`
result = {'uuid':uuid}
sleep(10)
output.put(result)
#app.route('/poll')
def poll():
"""
Endpoint hit ever x seconds from frontend
to see if the data is ready
"""
params = request.args.to_dict()
uuid_from_client = params['uuid']
# If client polls for result, but server has no record of this uuid
# This can happen in kubernetes with multiple instances running
if g.data_results.get(uuid_from_client) is None:
return jsonify({'msg':'pong', 'data':None, 'freshdata':None})
try:
output = g.output
# This line throws an error if there is nothing to get
results = output.get(False)
output.task_done()
# What is the uuid associated with the most recently returned data
# More than 1 chunk of data can be in here
uuid_from_data = results['uuid']
g.data_results[uuid_from_data].append(results)
except:
uuid_from_data = None
results = None
results_for_client_uuid = g.data_results[uuid_from_client]
if len(results_for_client_uuid) > 0:
res = results_for_client_uuid.pop(0)
else:
res = None
return jsonify({'msg':'pong', 'data':res})
if __name__ == "__main__":
with app.app_context():
app.run(host='0.0.0.0')

Setup your app architecture to use queuing softwares so that there is separation of concerns in terms of what job it does.
Here is a great article that can help you give some insight http://blog.gorgias.io/deploying-flask-celery-with-docker-and-kubernetes/
and one more https://endocode.com/blog/2015/03/24/using-googles-kubernetes-to-build-a-distributed-task-management-cluster/

Run function on Flask server every x seconds to update Redis cache without clients making separate calls

I currently have a flask app that makes a call to S3 as well as an external API with the following structure before rendering the data in javascript:
from flask import Flask, render_template,make_response
from flask import request
import requests
import requests_cache
import redis
from boto3.session import Session
import json
app = Flask(__name__)
#app.route('/test')
def test1():
bucket_root = 'testbucket'
session = Session(
aws_access_key_id='s3_key',
aws_secret_access_key='s3_secret_key')
s3 = session.resource('s3')
bucket = s3.Bucket(bucket_root)
testvalues = json.dumps(s3.Object(bucket_root,'all1.json').get()['Body'].read())
r = requests.get(api_link)
return render_template('test_html.html',json_s3_test_response=r.content,
limit=limit, testvalues=testvalues)
#app.route('/test2')
def test2():
bucket_root = 'testbucket'
session = Session(
aws_access_key_id='s3_key',
aws_secret_access_key='s3_secret_key')
s3 = session.resource('s3')
bucket = s3.Bucket(bucket_root)
testvalues = json.dumps(s3.Object(bucket_root,'all2.json').get()['Body'].read())
r = requests.get(api_link)
return render_template('test_html.html',json_s3_test_response=r.content,
limit=limit, testvalues=testvalues)
#app.errorhandler(500)
def internal_error(error):
return "500 error"
#app.errorhandler(404)
def not_found(error):
return "404 error",404
#app.errorhandler(400)
def custom400(error):
return "400 error",400
//catch all?
#app.errorhandler(Exception)
def all_exception_handler(error):
return 'error', 500
Obviously I have a lot of inefficiencies here, but my main question is:
To me it seems like I'm calling S3 and the external API for each client, every time they refresh the page. This increases the chance for the app to crash due to timeouts (and my poor error handling) and diminishes performance. I would like to resolve this by periodically caching the S3 results (say every 10 mins) into a local redis server (already set up and running) as well as just pinging the external API just once from the server every few seconds before passing it onto ALL clients.
I have code that can store the data into redis every 10 mins in a regular python script, however, I'm not sure where to place this within the flask server? Do I put it as it's own function or keep the call to redis in the #app.route()?
Thank you everyone for your time and effort. Any help would be appreciated! I'm new to flask so some of this has been confusing.

Socketio client switching to xhr-polling running with flask app

I'm running a socketio server with a flask app using gevent. My namespace code is here:
class ConversationNamespace(BaseNamespace):
def __init__(self, *args, **kwargs):
request = kwargs.get('request', None)
if request:
self.current_app = request['current_app']
self.current_user = request['current_user']
super(ConversationNamespace, self).__init__(*args, **kwargs)
def listener(self):
r = StrictRedis(host=self.current_app.config['REDIS_HOST'])
p = r.pubsub()
p.subscribe(self.current_app.config['REDIS_CHANNEL_CONVERSATION_KEY'] + self.current_user.user_id)
conversation_keys = r.lrange(self.current_app.config['REDIS_CONVERSATION_LIST_KEY'] +
self.current_user.user_id, 0, -1)
# Reverse conversations so the newest is up top.
conversation_keys.reverse()
# Emit conversation history.
pipe = r.pipeline()
for key in conversation_keys:
pipe.hgetall(self.current_app.config['REDIS_CONVERSATION_KEY'] + key)
self.emit(self.current_app.config['SOCKETIO_CHANNEL_CONVERSATION'] + self.current_user.user_id, pipe.execute())
# Listen for new conversations..
for m in p.listen():
conversation = r.hgetall(self.current_app.config['REDIS_CONVERSATION_KEY'] + str(m['data']))
self.emit(self.current_app.config['SOCKETIO_CHANNEL_CONVERSATION'] +
self.current_user.user_id, conversation)
def on_subscribe(self):
self.spawn(self.listener)
What I'm noticing in my app is that when I first start the SocketIO server (code below), the clients are able to connect via a websocket in firefox and chrome
#!vendor/venv/bin/python
from gevent import monkey
monkey.patch_all()
from yellowtomato import app_instance
import werkzeug.serving
from socketio.server import SocketIOServer
app = app_instance('sockets')
#werkzeug.serving.run_with_reloader
def runServer():
SocketIOServer(('0.0.0.0', app.config['SOCKET_PORT']), app, resource='socket.io').serve_forever()
runServer()
After sometime (maybe an hour or so), when I try to connect to that namespace via the browser client, it no longer communicates with a websocket but rather xhr-polling. Moreover, it takes about 20 seconds before the first response comes from the server. It gives the end user the perception that things have become very slow (but its only when rendering the page on the first subscibe, the xhr polling happens frequently and events get pushed to clients in a timely fashion).
What is triggering this latency and how can I assure that clients connect quickly using websockets.

Figured it out - I was running via the command line in an ssh session. Ending the sessions killed the parent process which was causing gevent to not work properly.
Forking the SocketIOServer process in a screen session fixed the problem