I'm trying to use celery to schedule and run tasks on a fleet of servers. Each task is somewhat long running (few hours), and involves using subprocess to call a certain program with the given inputs. This program produces a lot of output both in stdout and stderr.
Is there some way to show the output produced by the program to the client in near real time? Stream the output, so that the client can watch the output spewed by the task running on the server without logging into the server?
You did not specify many requirements and constraints. I'm going to assume you already have a redis instance somewhere.
What you can do is read the output from the other process line by line and publish it through redis:
Here's an example where you can echo data into a file /tmp/foo for testing:
import redis
redis_instance = redis.Redis()
p = subprocess.Popen(shlex.split("tail -f /tmp/foo"), stdout=subprocess.PIPE)
while True:
line = p.stdout.readline()
if line:
redis_instance.publish('process log', line)
else:
break
In a separate process:
import redis
redis_instance = redis.Redis()
pubsub = redis_instance.pubsub()
pubsub.subscribe('process log')
while True:
for message in pubsub.listen():
print message # or use websockets to comunicate with a browser
If you want the process to end, you can e.g. send a "quit" after the celery task is done.
You can use different channels (the string in subscribe) to separate the output from different processes.
You can also store your log output in redis, if you want to,
redis_instance.rpush('process log', message)
and later retrieve it in full.
The one way I see how to do it is to write custom Logger which will be used for stderr and stdout (see the docs:
from celery.app.log import Logger
Logger.redirect_stdouts_to_logger(MyLogger())
Your logger can save the data into the database, Memcached, Redis or whatever shared storage you'll use to get the data.
I'm not sure about the structure of the logger, but I guess something like this will work:
from logging import Logger
class MyLogger(Logger):
def log(lvl, msg):
# Do something with the message
This is an old question but it's still pretty much the only result about this specific topic.
Here's how I went about it,
I created a simple file-like object that publishes to a specific channel over Redis
class RedisFileObject(object):
def __init__(self, _key):
self.connection = redis.Redis()
self.key = _key
self.connection.publish('debug', 'Created channel %s' % self.key)
def write(self, data):
self.connection.publish(self.key, data)
def close(self):
pass
I have a BaseTask from which all of my tasks inherits various functions incl. this one that replaces stdout and stderr with the Redis file-like object.
def capture_output(self):
sys.stdout = RedisFileObject(self.request.id)
sys.stderr = RedisFileObject(self.request.id)
From there on anything written to stdout/stderr will be forwarded to a Redis channel named after the task id.
Related
from kafka import KafkaProducer, errors, admin, KafkaConsumer
SERVERS = ['localhost:9092']
TEST_TOPIC = 'test-topic'
DATA = [{'A':'A'}, {'A':'A'}, {'A':'A'}]
class TestKafkaConsumer(unittest.TestCase):
#classmethod
def setUpClass(self):
self._producer = KafkaProducer(bootstrap_servers=SERVERS, value_serializer=lambda x:dumps(x).encode('utf-8'))
def _send_data(self):
for data in DATA:
print(self._producer.send(TEST_TOPIC, value=data))
def test_basic_processing(self):
self._send_data()
received = []
consumer = KafkaConsumer(TEST_TOPIC, bootstrap_servers=SERVERS)
for msg in consumer:
message = json.loads(msg.value.decode('utf-8'))
received.append(message)
if (len(received) >= len(DATA)):
self.assertEqual(received, DATA)
This should succeed pretty quickly, as it just sends the data to the the Kafka broker in a pretty straightforward manner. However, it times out; the consumer never reads a single message. If I move the consumer portion to a different file and run it in a different terminal window, the messages are "consumed" pretty instantly. Why is the unittest not working for a consumer in this unittest?
You're producing records with your producer and then you're reading, this might be your problem.
When your consumer is started, you already had produced records, so, from the consumer point of view, there are no new messages.
You should run your consumer in a different thread, before your producer start producing.
Yannick
I'm using Python 2.7 (sigh), celery==3.1.19, librabbitmq==1.6.1, rabbitmq-server-3.5.6-1.noarch, and redis 2.8.24 (from redis-cli info).
I'm attempting to send a message from a celery producer to a celery consumer, and obtain the result back in the producer. There is 1 producer and 1 consumer, but 2 rabbitmq's (as brokers) and 1 redis (for results) in between.
The problem I'm facing is:
In the consumer, I get back get an AsyncResult via async_result =
ZipUp.delay(unique_directory), but async_result.ready() never
returns True (at least for 9 seconds it doesn't) - even for a
consumer task that does essentially nothing but return a string.
I can see, in the rabbitmq management web interface, my message
being received by the rabbitmq exchange, but it doesn't show up in
the corresponding rabbitmq queue. Also, a log message sent by the
very beginning of the ZipUp task doesn't appear to be getting
logged.
Things work if I don't try to get a result back from the AsyncResult! But I'm kinda hoping to get the result of the call - it's useful :).
Below are configuration specifics.
We're setting up Celery as follows for returns:
CELERY_RESULT_BACKEND = 'redis://%s' % _SHARED_WRITE_CACHE_HOST_INTERNAL
CELERY_RESULT = Celery('TEST', broker=CELERY_BROKER)
CELERY_RESULT.conf.update(
BROKER_HEARTBEAT=60,
CELERY_RESULT_BACKEND=CELERY_RESULT_BACKEND,
CELERY_TASK_RESULT_EXPIRES=100,
CELERY_IGNORE_RESULT=False,
CELERY_RESULT_PERSISTENT=False,
CELERY_ACCEPT_CONTENT=['json'],
CELERY_TASK_SERIALIZER='json',
CELERY_RESULT_SERIALIZER='json',
)
We have another Celery configuration that doesn't expect a return value, and that works - in the same program. It looks like:
CELERY = Celery('TEST', broker=CELERY_BROKER)
CELERY.conf.update(
BROKER_HEARTBEAT=60,
CELERY_RESULT_BACKEND=CELERY_BROKER,
CELERY_TASK_RESULT_EXPIRES=100,
CELERY_STORE_ERRORS_EVEN_IF_IGNORED=False,
CELERY_IGNORE_RESULT=True,
CELERY_ACCEPT_CONTENT=['json'],
CELERY_TASK_SERIALIZER='json',
CELERY_RESULT_SERIALIZER='json',
)
The celery producer's stub looks like:
#CELERY_RESULT.task(name='ZipUp', exchange='cognition.workflow.ZipUp_%s' % INTERNAL_VERSION)
def ZipUp(directory): # pylint: disable=invalid-name
""" Task stub """
_unused_directory = directory
raise NotImplementedError
It's been mentioned that using queue= instead of exchange= in this stub would be simpler. Can anyone confirm that (I googled but found exactly nothing on the topic)? Apparently you can just use queue= unless you want to use fanout or something fancy like that, since not all celery backends have the concept of an exchange.
Anyway, the celery consumer starts out with:
#task(queue='cognition.workflow.ZipUp_%s' % INTERNAL_VERSION, name='ZipUp')
#StatsInstrument('workflow.ZipUp')
def ZipUp(directory): # pylint: disable=invalid-name
'''
Zip all files in directory, password protected, and return the pathname of the new zip archive.
:param directory Directory to zip
'''
try:
LOGGER.info('zipping up {}'.format(directory))
But "zipping up" doesn't get logged anywhere. I searched every (disk-backed) file on the celery server for that string, and got two hits: /usr/bin/zip, and my celery task's code - and no log messages.
Any suggestions?
Thanks for reading!
It appears that using the following task stub in the producer solved the problem:
#CELERY_RESULT.task(name='ZipUp', queue='cognition.workflow.ZipUp_%s' % INTERNAL_VERSION)
def ZipUp(directory): # pylint: disable=invalid-name
""" Task stub """
_unused_directory = directory
raise NotImplementedError
In short, it's using queue= instead of exchange= .
I have a Tornado web application, this app can receive GET and POST request from the client.
The POSTs request put an information received in a Tornado Queue, then I pop this information from the queue and with it I do an operation on the database, this operation can be very slow, it can take several seconds to complete!
In the meantime that this database operation goes on I want to be able to receive other POSTs (that put other information in the queue) and GET. The GET are instead very fast and must return to the client their result immediatly.
The problem is that when I pop from the queue and the slow operation begin the server doesn't accept other requests from the client. How can I resolve this?
This is the semplified code I have written so far (import are omitted for avoid wall of text):
# URLs are defined in a config file
application = tornado.web.Application([
(BASE_URL, Variazioni),
(ARTICLE_URL, Variazioni),
(PROMO_URL, Variazioni),
(GET_FEEDBACK_URL, Feedback)
])
class Server:
def __init__(self):
http_server = tornado.httpserver.HTTPServer(application, decompress_request=True)
http_server.bind(8889)
http_server.start(0)
transactions = TransactionsQueue() #contains the queue and the function with interact with it
IOLoop.instance().add_callback(transactions.process)
def start(self):
try:
IOLoop.instance().start()
except KeyboardInterrupt:
IOLoop.instance().stop()
if __name__ == "__main__":
server = Server()
server.start()
class Variazioni(tornado.web.RequestHandler):
''' Handle the POST request. Put an the data received in the queue '''
#gen.coroutine
def post(self):
TransactionsQueue.put(self.request.body)
self.set_header("Location", FEEDBACK_URL)
class TransactionsQueue:
''' Handle the queue that contains the data
When a new request arrive, the generated uuid is putted in the queue
When the data is popped out, it begin the operation on the database
'''
queue = Queue(maxsize=3)
#staticmethod
def put(request_uuid):
''' Insert in the queue the uuid in postgres format '''
TransactionsQueue.queue.put(request_uuid)
#gen.coroutine
def process(self):
''' Loop over the queue and load the data in the database '''
while True:
# request_uuid is in postgres format
transaction = yield TransactionsQueue.queue.get()
try:
# this is the slow operation on the database
yield self._load_json_in_db(transaction )
finally:
TransactionsQueue.queue.task_done()
Moreover I don't understand why if I do 5 POST in a row, it put all five data in the queue though the maximun size is 3.
I'm going to guess that you use a synchronous database driver, so _load_json_in_db, although it is a coroutine, is not actually async. Therefore it blocks the entire event loop until the long operation completes. That's why the server doesn't accept more requests until the operation is finished.
Since _load_json_in_db blocks the event loop, Tornado can't accept more requests while it's running, so your queue never grows to its max size.
You need two fixes.
First, use an async database driver written specifically for Tornado, or run database operations on threads using Tornado's ThreadPoolExecutor.
Once that's done your application will be able to fill the queue, so second, TransactionsQueue.put must do:
TransactionsQueue.queue.put_nowait(request_uuid)
This throws an exception if there are already 3 items in the queue, which I think is what you intend.
I'm writing a program that acts as a script manager. It consists of 3 parts:
A client - Receives a script name to run from the server.
Manager - Manages running scripts. Receives them from the client using a json transferred in a Pipe.
The scrips - .py scripts that are found under the addons directory in the project.
The important thing to notice is that all 3 components run simultaneously as processes (because I could have an alarm script running while accepting and executing a play music script.
Because it consists of 3 separate parts that interact with each other I don't know how to write proper unit tests for it.
So my questions are:
How can I write good unit tests for this?
Is this a design problem? if so, what am I doing wrong and what should I do to fix it?
Here is most of the code for the above components:
The Client
class MessageReceiver:
def __init__(self):
'''
Connect to the AMQP broker and starts listening for messages.
Creates the a Popen object to pass command info to the addon_manager script (which
is in charge of managing scripts)
'''
addon_manager_path = configuration.addon_manager_path()
addon_manager_path = os.path.join(addon_manager_path,'addon_manager.py')
execute = "python " + addon_manager_path
self.addon_manager = subprocess.Popen(execute, stdin=subprocess.PIPE, shell=True)
self.component_name= configuration.get_attribute("name")
if len(sys.argv)>1:
host_ip = sys.argv[1]
else:
host_ip = 'localhost'
#Start a connection to the AMQP server
self.connection = pika.BlockingConnection(pika.ConnectionParameters(host=host_ip))
#Create a channel to the server
self.channel = self.connection.channel()
self.channel.queue_declare(queue="example")
#callback method to be called when data is received
#It sends the data that is received by the client to the addon_manager
def data_received(ch, method, properties, body):
##TODO: Might want to add some checks. Is body a JSON? etc.
print("GOT IT")
self.addon_manager.communicate(body)
self.channel.basic_consume(data_received,queue='example',no_ack=True)
self.channel.start_consuming()
The Manager
class AddonManager:
def __init__(self):
self.command_analyzer = analyzer.Analyzer(configuration.get_attribute("commands"))
self.active_addons = {}
def activate_addon(self,command,json_data):
child_popen = self.command_analyzer.execute_command(command,json_data)
self.active_addons[analyzer.intent(json_data)] = child_popen
def communicate_with_addon(self,command,json_data,json_string):
child_popen = self.active_addons[analyzer.intent(json_data)]
#Child process hasn't finished running
if child_popen.returncode is None:
#Send data to child to process if he wants to
child_popen.stdin.write(json_string)
else:
#Process finished running. Can't send it anything. delete it. (deleting the Popen instance also kills zombie process)
del self.active_addons[analyzer.intent(json_data)]
self.activate_addon(command,json_data)
def get_input(self):
"""
Reads command from stdin, returns its JSON form
"""
json_string = sys.stdin.read()
json_data =json.loads(json_string)
print(json_data)
return json_data
def accept_commands(self):
while True:
json_data = self.get_input()
command = self.command_analyzer.is_command(json_data) # Check wether the command exists. Return it if it does
#The command exists
if command is not None:
#The addon is not currently active
if analyzer.intent(json_data) not in self.active_addons:
self.activate_addon(command,json_data)
#The addon is active and so we need to send the data to the subprocess
else:
self.communicate_with_addon(command,json_data,json_string)
manager = AddonManager()
manager.accept_commands()
This is a probably basic question, but I have not been able to find the answer.
I have a long-running process that produces data every few minutes that I would like the client to receive as soon as it is ready. Currently I have the long-running process in a Task Queue, and it adds channel messages to another Task Queue from within a for loop. The client successfully receives the channel messages and downloads the data using a get request; however, the messages are being sent from the task queue after the long-running process finishes (after about 10 minutes) instead of when the messages are added to the task queue.
How can I have the messages in the task queue sent immediately? Do I need to have the for loop broken into several tasks? The for loop creates a number of dictionaries I think I would need to post to the data store and then retrieve for the next iteration (does not seem like an ideal solution), unless there is an easier way to return data from a task.
When I do not add the messages to a Task Queue and send the messages directly in the for loop, the server does not seem to respond to the client's get request for the data (possibly due to the for loop of the long-running process blocking the response?)
Here is a simplified version of my server code:
from google.appengine.ext import db
from google.appengine.api import channel
from google.appengine.api import taskqueue
from google.appengine.api import rdbms
class MainPage(webapp2.RequestHandler):
def get(self):
## This opens the GWT app
class Service_handler(webapp2.RequestHandler):
def get(self, parameters):
## This is called by the GWT app and generates the data to be
## sent to the client.
#This adds the long-process to a task queue
taskqueue.Task(url='/longprocess/', params = {'json_request': json_request}).add(queue_name='longprocess-queue')
class longprocess_handler(webapp2.RequestHandler):
def post(self):
#This has a for loop that recursively uses data in dictionaries to
#produce kml files every few minutes
for j in range(0, Time):
# Process data
# Send message to client using a task queue to send the message.
taskqueue.Task(url='/send/', params).add(queue_name=send_queue_name)
class send_handler(webapp2.RequestHandler):
def post(self):
# This sends the message to the client
# This is currently not happening until the long-process finishes,
# but I would like it to occur immediately.
class kml_handler(webapp2.RequestHandler):
def get(self, client_id):
## When the client receives the message, it picks up the data here.
app = webapp2.WSGIApplication([
webapp2.Route(r'/', handler=MainPage),
webapp2.Route(r'/Service/', handler=Service_handler),
webapp2.Route(r'/_ah/channel/<connected>/', handler = connection_handler),
webapp2.Route(r'/longprocess/', handler = longprocess_handler),
webapp2.Route(r'/kml/<client_id>', handler = kml_handler),
webapp2.Route(r'/send/', handler = send_handler)
],
debug=True)
Do I need to break up the long-process into tasks that send and retrieve results from the data store in order to have the send_handler execute immediately, or am I missing something? Thanks
The App Engine development server only processes one request at a time. In production, these things will occur simultaneously. Try in production, and check that things behave as expected there.
There's also not much reason to use a separate task to send the channel messages in production - just send them directly from the main task.