I have a Celery task registered in my tasks.py file. When someone POST to /run/pk I run the task with the given parameters. This task also executes other tasks (normal Python functions), and I'd like to update my page (the HttpResponse returned at /run/pk) whenever a subtask finishes its work.
Here is my task:
from celery.decorators import task
#task
def run(project, branch=None):
if branch is None:
branch = project.branch
print 'Creating the virtualenv'
create_virtualenv(project, branch)
print 'Virtualenv created' ##### Here I want to send a signal or something to update my page
runner = runner(project, branch)
print 'Using {0}'.format(runner)
try:
result, output = runner.run()
except Exception as e:
print 'Error: {0}'.format(e)
return False
print 'Finished'
run = Run(project=project, branch=branch,
output=output, **result._asdict())
run.save()
return True
Sending push notifications to the client's browser using Django isn't easy, unfortunately. The simplest implementation is to have the client continuously poll the server for updates, but that increases the amount of work your server has to do by a lot. Here's a better explanation of your different options:
Django Push HTTP Response to users
If you weren't using Django, you'd use websockets for these notifications. However Django isn't built for using websockets. Here is a good explanation of why this is, and some suggestions for how to go about using websockets:
Making moves w/ websockets and python / django ( / twisted? )
With many years past since this question was asked, Channels is a way you could now achieve this using Django.
Then Channels website describes itself as a "project to make Django able to handle more than just plain HTTP requests, including WebSockets and HTTP2, as well as the ability to run code after a response has been sent for things like thumbnailing or background calculation."
There is a service called Pusher that will take care of all the messy parts of Push Notifications in HTML5. They supply a client-side and server-library to handle all the messaging and notifications, while taking care of all the HTML5 Websocket nuances.
Related
I have a python flask application where a person is gonna request data by sending a query from A form. This data goes through a certain python script that does some api-requests and covert the data into a geographic standard.
The thing is because this data can take some time based on how many datapoints there are, this will happen in the background (we are researching this for Azure). Also there is another problem, and that is cueing up. Because if one request is running, another one cannot be started up. And the last command cannot be saved:
#app.route('/handle_data', methods=['POST'])
def handle_data():
sper_year = int(request.form["Speryear"])
email = request.form["inputEmail"]
url = request.form["api-url-input"]
random_string=get_random_string(5)
# app.route('/request-completed')
Requested_data = Program_Converter.main(url, sper_year,random_string)
Requested_data = Program_Converter.main(url, sper_year,random_string) is the function that needs to be qued.
How do I do this?
I believe that the most recommended way is to run this task asynchronously. Take a look at Celery and pick up a backend (I recommend Redis), with this setup you can provide Celery a task that will run your GIPOD_Converter process in the background and store the result somewhere you chose, then it can be sent back to the user.
Note that Celery will provide you a task id and is up to your client (web interface or mobile app, I'm not sure what you're working with) to poll and endpoint and wait for the celery task to come to an end.
There are a couple of examples on who to implement this all over the web, but take a look at the Flask official documentation and check out this article from the mighty Miguel Grinberg, I believe those are the best starting points for you.
I hope I am posting this in the right place.
I am researching RabbitMQ for potential use in our Plone sites. We currently us Async on a dedicated worker client in the Plone server, but we are thinking about building a dedicated RabbitMQ server that will handle all Plone messaging and other activity.
My specific question is, what are the advantages of using Celery to work with RabbitMQ in Plone versus just using RabbitMQ? I found this plone add-on for Celery integration, but not sure if that is best route to go. I noticed Celery has the Flower tool for monitoring the queues, which would be a huge plus.
As a side question, if you feel so inclined, does anyone have any tips or references for integration RabbitMQ with Plone to handle all of these requests? I have been doing research, and get the general gist of RabbitMQ, but I can't seem to make the connection with Plone activities, such as Content Rules and PloneFormGen submissions for example. So far I see this add-on that I am going to install and see if I can figure out, but I am just trying to get a little guidance if I can.
Thanks for your time!
At first, ask yourself, if you need the features of RabbitMQ or just want to do some asynchronous tasks in Python with Plone.
If you don't really need RabbitMQ, you could look into David Glick's gists for how to integrate Celery with Plone (and still use RabbitMQ with Celery):
https://gist.github.com/davisagli/5824662
https://gist.github.com/davisagli/5824709
You could also look into collective.taskqueue (simple queues without Celery nor RabbitMQ), but it does not provide any monitoring solution yet.
If you really need RabbitMQ, skip Celery, and try out collective.zamqp. Celery tries to be broker by itself and would prevent you from using most of AMQP's and RabbitMQ's built-in features.
RabbitMQ ships with great web admin plugin for monitoring and there are also plugins for 3rd party monitoring systems (like Zenoss).
I'm sorry that collective.zamqp is still missing narrative documentation, but you can look into collective.zamqpdemo for various examples of its configuration and usage.
In short, c.zamqp allows you to define configure broker usage in terms of producers and consumers:
from five import grok
from zope.interface import Interface
from collective.zamqp.producer import Producer
from collective.zamqp.consumer import Consumer
class CreateItemProducer(Producer):
"""Produces item creation requests"""
grok.name("amqpdemo.create") # is also used as default routing key
connection_id = "superuser"
serializer = "msgpack"
queue = "amqpdemo.create"
durable = False
class ICreateItemMessage(Interface):
"""Marker interface for item creation message"""
class CreateItemConsumer(Consumer):
"""Consumes item creation messages"""
grok.name("amqpdemo.create") # is also used as the queue name
connection_id = "superuser"
marker = ICreateItemMessage
durable = False
Publish messages through transaction bound producer (to publish messages only after a successful transaction):
import uuid
from zope.component import getUtility
from collective.zamqp.interfaces import IProducer
producer = getUtility(IProducer, name="amqpdemo.create")
producer._register() # register to bound to successful transaction
message = {"title": u"My title"}
producer.publish(message)
And consume the messages in a familiar content event handler environment:
from zope.component.hooks import getSite
from collective.zamqp.interfaces import IMessageArrivedEvent
from plone.dexterity.utils import createContentInContainer
#grok.subscribe(ICreateItemMessage, IMessageArrivedEvent)
def createItem(message, event):
"""Consume item creation message"""
portal = getSite()
obj = createContentInContainer(
portal, "Document", checkConstraints=True, **message.body)
message.ack()
Finally, it decouples broker connection configuration from code and the actual connection parameters can be defined in buildout.cfg (allowing required amount of consuming instances):
[instance]
recipe = plone.recipe.zope2instance
...
zope-conf-additional =
%import collective.zamqp
<amqp-broker-connection>
connection_id superuser
heartbeat 120
# These are defaults, but can be defined when required:
# hostname localhost
# virtual_host /
# username guest
# password guest
</amqp-broker-connection>
<amqp-consuming-server>
connection_id superuser
site_id Plone
user_id admin
vhm_method_prefix /VirtualHostBase/https/example.com:443/Plone/VirtualHostRoot
</amqp-consuming-server>
c.zamqp cannot be directly called from RestrictedPython, so integrating it to PloneFormGen would need either a custom action adapter or a custom External method to be called from PFG's Python script adapter.
I want to schedule an email to be sent to a user upon a specific action.
However, if the user takes another action I want to cancel that email and have it not send.
How would I do that in django or python?
Beanstalkd
If you can install beanstalkd and run python script from command line I would use that to schedule emails. With beanstalkc client you can easily accomplish this. On ubuntu you might first need to install:
sudo apt-get install python-yaml python-setuptools
consumer.py:
import beanstalkc
def main():
beanstalk = beanstalkc.Connection(host='localhost', port=11300)
while True:
job = beanstalk.reserve()
print job.body
job.delete()
if __name__ == '__main__':
main()
Will print job 5 seconds after it get's inserted by producer.py. Offcourse this should be set longer to when you want to schedule your emails, but for demonstration purposes it will do. You don't want to wait half an hour to schedule message when testing ;).
producer.py:
import beanstalkc
def main():
beanstalk = beanstalkc.Connection(host='localhost', port=11300)
jid = beanstalk.put('foo', delay=5)
if __name__ == '__main__':
main()
GAE Task Queue
You could also use Google App engine task queue to accomplish this. You can specify an eta for your Task. Google App engine has a generous free quota. In the task queue webhook make Asynchronous Requests to fetch URL on your server which does the sending of emails.
I would set up a cron job which could handle everything you want to do...
If you didn't have access to cron, you could easily do this:
Write a model that stores the email, the time to send, and a BooleanField indicating if the email has been sent.
Write a view which selects all emails that haven't been sent yet but should have by now, and sends them.
Use something like OpenACS Uptime, Pingdom or any other service capable of sending HTTP GET requests periodically to call that view, and trigger the email sending. (Both are free, the former should request once every 15 minutes, and the latter can be configured to request up to every minute, and will do so from several locations.)
Sure, it's inelegant, but it's a method that works on basically any web host. I used to do something like this when I was writing PHP apps to run on a host that killed all processes after something like 15 seconds.
Are your using celery ? If true, see http://ask.github.com/celery/userguide/executing.html#eta-and-countdown
You said that you want to do it through Python or Django, but it seems as though something else will need to be involved. Considering you are on a shared host, there is a chance installing other packages could also be a problem.
Another possible solution could be something like this:
Use a javascript framework which can setup timed events, start/cancel them etc. I have done timed events using a framework called ExtJS. Although ExtJS is rather large im sure other frameworks such as jQuery or even raw javascript you could do a similar thing.
Set up a task on a user action, that will execute in 5 minutes. The action could be an ajax call to a python script which sends the email... If a user does something where the task needs to be stopped, just cancel the event.
It kind of seems complicated and convoluted, but it really isn't. If this seems like a path you would like to try out, let me know and I'll edit with some code
I have a Django web application. I also have a spell server written using twisted running on the same machine having django (running on localhost:8090). The idea being when user does some action, request comes to Django which in turn connects to this twisted server & server sends data back to Django. Finally Django puts this data in some html template & serves it back to the user.
Here's where I am having a problem. In my Django app, when the request comes in I create a simple twisted client to connect to the locally run twisted server.
...
factory = Spell_Factory(query)
reactor.connectTCP(AS_SERVER_HOST, AS_SERVER_PORT, factory)
reactor.run(installSignalHandlers=0)
print factory.results
...
The reactor.run() is causing a problem. Since it's an event loop. The next time this same code is executed by Django, I am unable to connect to the server. How does one handle this?
The above two answers are correct. However, considering that you've already implemented a spelling server then run it as one. You can start by running it on the same machine as a separate process - at localhost:PORT. Right now it seems you have a very simple binary protocol interface already - you can implement an equally simple Python client using the standard lib's socket interface in blocking mode.
However, I suggest playing around with twisted.web and expose a simple web interface. You can use JSON to serialize and deserialize data - which is well supported by Django. Here's a very quick example:
import json
from twisted.web import server, resource
from twisted.python import log
class Root(resource.Resource):
def getChild(self, path, request):
# represents / on your web interface
return self
class WebInterface(resource.Resource):
isLeaf = True
def render_GET(self, request):
log.msg('GOT a GET request.')
# read request.args if you need to process query args
# ... call some internal service and get output ...
return json.dumps(output)
class SpellingSite(server.Site):
def __init__(self, *args, **kwargs):
self.root = Root()
server.Site.__init__(self, self.root, **kwargs)
self.root.putChild('spell', WebInterface())
And to run it you can use the following skeleton .tac file:
from twisted.application import service, internet
site = SpellingSite()
application = service.Application('WebSpell')
# attach the service to its parent application
service_collection = service.IServiceCollection(application)
internet.TCPServer(PORT, site).setServiceParent(service_collection)
Running your service as another first class service allows you to run it on another machine one day if you find the need - exposing a web interface makes it easy to horizontally scale it behind a reverse proxying load balancer too.
reactor.run() should be called only once in your whole program. Don't think of it as "start this one request I have", think of it as "start all of Twisted".
Running the reactor in a background thread is one way to get around this; then your django application can use blockingCallFromThread in your Django application and use a Twisted API as you would any blocking API. You will need a little bit of cooperation from your WSGI container, though, because you will need to make sure that this background Twisted thread is started and stopped at appropriate times (when your interpreter is initialized and torn down, respectively).
You could also use Twisted as your WSGI container, and then you don't need to start or stop anything special; blockingCallFromThread will just work immediately. See the command-line help for twistd web --wsgi.
You should stop reactor after you got results from Twisted server or some error/timeout happening. So on each Django request that requires query your Twisted server you should run reactor and then stop it. But, it's not supported by Twisted library — reactor is not restartable. Possible solutions:
Use separate thread for Twisted reactor, but you will need to deploy your django app with server, which has support for long running threads (I don't now any of these, but you can write your own easily :-)).
Don't use Twisted for implementing client protocol, just use plain stdlib's socket module.
I have a normal Django site running. In addition, there is another twisted process, which listens for Jabber presence notifications and updates the Django DB using Django's ORM.
So far it works as I just call the corresponding Django models (after having set up the settings environment correctly). This, however, blocks the Twisted app, which is not what I want.
As I'm new to twisted I don't know, what the best way would be to access the Django DB (via its ORM) in a non-blocking way using deferreds.
deferredGenerator ?
twisted.enterprise.adbapi ? (circumvent the ORM?)
???
If the presence message is parsed I want to save in the Django DB that the user with jid_str is online/offline (using the Django model UserProfile). I do it with that function:
def django_useravailable(jid_str, user_available):
try:
userhost = jid.JID(jid_str).userhost()
user = UserProfile.objects.get(im_jabber_name=userhost)
user.im_jabber_online = user_available
user.save()
return jid_str, user_available
except Exception, e:
print e
raise jid_str, user_available,e
Currently, I invoke it with:
d = threads.deferToThread(django_useravailable, from_attr, user_available)
d.addCallback(self.success)
d.addErrback(self.failure)
"I have a normal Django site running."
Presumably under Apache using mod_wsgi or similar.
If you're using mod_wsgi embedded in Apache, note that Apache is multi-threaded and your Python threads are mashed into Apache's threading. Analysis of what's blocking could get icky.
If you're using mod_wsgi in daemon mode (which you should be) then your Django is a separate process.
Why not continue this design pattern and make your "jabber listener" a separate process.
If you'd like this process to be run any any of a number of servers, then have it be started from init.rc or cron.
Because it's a separate process it will not compete for attention. Your Django process runs quickly and your Jabber listener runs independently.
I have been successful using the method you described as your current method. You'll find by reading the docs that the twisted DB api uses threads under the hood because most SQL libraries have a blocking API.
I have a twisted server that saves data from power monitors in the field, and it does it by starting up a subthread every now and again and calling my Django save code. You can read more about my live data collection pipeline (that's a blog link).
Are you saying that you are starting up a sub thread and that is still blocking?
I have a running Twisted app where I use Django ORM. I'm not deferring it. I know it's wrong, but hadd no problems yet.