I'm using celery with django and am trying to get a Task, like the one below:
class task1 (Task)
def run (self):
launch_some_other_task.delay()
But it doesn't seem to be working, I can go into more detail as far as my code but figured I would just ask first if this sort of thing will work as doesn't seem to be working for me. I am finding this necessary as I am using selenium, a web testing framework, where sometimes it will just hang where I can't get any output from this so I want to be able to kill if off if a certain condition isn't met (updating a memcache variable with a certain value within a specified number of seconds).
Thanks for any advice on this
Make sure you've added the following to your urls.py:
import djcelery
djcelery.setup_loader()
Related
I created a python script that work as a bot for instagram (using selenium).
Currently I have 5 profile running, for each of them I have all the files stored in folders (called with the name of the ig profile) and for each profile I have a screen where I can see the "log" of each program.
But now, 5 profile are difficult to manage and sometimes also a little messy.
Is there a way to see the log of all 5 scripts in a unique window?
I'm open also to another way to run the scripts in the background, maybe not "screen" but something else.
Thankyou
If you really want to go the clean way and if you think this will get bigger, you might want to have a look towards Django and Celery.
You can create a web interface, so that you can monitor any way you like.
And you can have cron jobs with Celery so that your bot is always on, or has recurring tasks, etc...
More info on their respective docs, as usual. http://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html
I am new to web app programming in general, and flask in particular. I am trying to create a "graphical" interface to some Python code I wrote a while ago. It was suggested to me that instead of writing a graphical interface I should make it a web app with flask. The code I want it to run may take a while and I also want the user to be able to run multiple instances concurrently. I was told I should use celery for the concurrency.
Another major requirement is that feedback about the progress should be given to the user. Not knowing much about web programming from the start I did a bit of research to figure out how this can be done on the web and discovered server-sent events which seemed perfect to me. Trying to figure out how to do this in Flask, I found http://flask.pocoo.org/snippets/116/ which I got to work as described. After learning about celery and getting the example at http://flask.pocoo.org/docs/patterns/celery/ to work, I tried creating a simple example of a task that would have some sort of state that would update over time and feed that back to the user. The problem I ran into was that my celery tasks didn't seem to start.
I then found an example where server-sent events were used in a chat program that used redis, so I thought I would try that, but I had the same result. Below is what I wrote to implement the subscribe functionality:
#flask_app.route("/subscribe", methods=['GET'])
def subscribe():
targetname = request.args['target']
print 'The subscribe function was called with target: ' + targetname
return Response(buildSubscription(targetname), mimetype="text/event-stream")
I then have a celery task:
#celery.task()
def testPrint(printfunc,initvalue):
print "In testPrint"
x=hashlib.md5(initvalue)
i=0
while True:
time.sleep(5)
printfunc(x.hexdigest())
x.update(str(i))
i+=1
where celery is set by the make_celery function. I then have a function:
def ssePrint(mystring,sub):
print "In ssePrint"
red.pubsub(sub,mystring)
and a wrapper for that function:
def appPrint(subscription):
sub = subscription
def myprint(myinput):
ssePrint(myinput, sub)
return myprint
which is then called in
#flask_app.route('/start', methods=['POST'])
def start():
myobject = request.get_json()
testPrint.delay(appPrint(myobject['target']), myobject['target'])
When I then look at the shell that invoked celery, I do not see any tasks running. This happens when I also use gevent.Queue as described in the SSE Flask example.
My suspicion is that there might be an issue with communicating from a Celery task to something running outside of the task. Is there a standard way of extracting the kind of information I want from a Celery task? Should I consider another way of doing concurrency?
Thanks.
I am trying to write a Heroku app in python which will read and store data from a xively feed in real time. I want the app to run independently as a sort of 'backend process' to simply store the data in a database. (It does not need to 'serve up' anything for users (for site visitors).)
Right now I am working on the 'continuous reading' part. I have included my code below. It simply reads the datastream once, each time I hit my app's Heroku URL. How do I get it to operate continuously so that it keeps on reading the data from xively?
import os
from flask import Flask
import xively
app = Flask(__name__)
#app.route('/')
def run_xively_script():
key = 'FEED_KEY'
feedid = 'FEED_ID'
client = xively.XivelyAPIClient(key)
feed = client.feeds.get(feedid)
datastream = feed.datastreams.get("level")
level = datastream.current_value
return "level is %s" %(level)
I am new to web development, heroku, and python... I would really appreciate any help(pointers)
{
PS:
I have read about Heroku Scheduler and from what I understand, it can be used to schedule a task at specific time intervals and when it does so, it starts a one-off dyno for the task. But as I mentioned, my app is really meant to perform just one function->continuously reading and storing data from xively. Is it necessary to schedule a separate task for that? And the one-off dyno that the scheduler will start will also consume dyno hours, which I think will exceed the free 750 dyno-hours limit (as my app's web dyno is already consuming 720 dyno-hours per month)...
}
Using the scheduler, as you and #Calumb have suggested, is one method to go about this.
Another method would be for you to setup a trigger on Xively. https://xively.com/dev/docs/api/metadata/triggers/
Have the trigger occur when your feed is updated. The trigger should POST to your Flask app, and the Flask app can then take the new data, manipulate it and store it as you wish. This would be the most near realtime, I'd think, because Xively is pushing the update to your system.
This question is more about high level architecture decisions and what you are trying to accomplish than a specific thing you should do.
Ultimately, Flask is probably not the best choice for an app to do what you are trying to do. You would be better off with just pure python or pure ruby. With that being said, using Heroku scheduler (which you alluded to) makes it possible to do something like what you are trying to do.
The simplest way to accomplish your goal (assuming that you want to change minimal amount of code and that constantly reading data is really what you want to do. Both of which you should consider) is to write a loop that runs when you call that task and grabs data for a few seconds. Just use a for loop and increment a counter for however many times you want to get the data.
Something like:
for i in range(0,5):
key = 'FEED_KEY'
feedid = 'FEED_ID'
client = xively.XivelyAPIClient(key)
feed = client.feeds.get(feedid)
datastream = feed.datastreams.get("level")
level = datastream.current_value
time.sleep(1)
However, Heroku has limits on how long something can run before it returns a value. Otherwise the router will return a 503 or 500. But you could use the scheduler to then schedule this to run every certain amount of time.
Again, I think that Flask and Heroku is not the best solution for what it sounds like you are trying to do. I would review your use case and go back to the drawing board on what the best method to accomplish it our.
This is probably a truly basic thing that I'm simply having an odd time figuring out in a Python 2.5 app.
I have a process that will take roughly an hour to complete, so I made a backend. To that end, I have a backend.yaml that has something like the following:
-name: mybackend
options: dynamic
start: /path/to/script.py
(The script is just raw computation. There's no notion of an active web session anywhere.)
On toy data, this works just fine.
This used to be public, so I would navigate to the page, the script would start, and time out after about a minute (HTTP + 30s shutdown grace period I assume, ). I figured this was a browser issue. So I repeat the same thing with a cron job. No dice. Switch to a using a push queue and adding a targeted task, since on paper it looks like it would wait for 10 minutes. Same thing.
All 3 time out after that minute, which means I'm not decoupling the request from the backend like I believe I am.
I'm assuming that I need to write a proper Handler for the backend to do work, but I don't exactly know how to write the Handler/webapp2Route. Do I handle _ah/start/ or make a new endpoint for the backend? How do I handle the subdomain? It still seems like the wrong thing to do (I'm sticking a long-process directly into a request of sorts), but I'm at a loss otherwise.
So the root cause ended up being doing the following in the script itself:
models = MyModel.all()
for model in models:
# Magic happens
I was basically taking for granted that the query would automatically batch my Query.all() over many entities, but it was dying at the 1000th entry or so. I originally wrote it was computational only because I completely ignored the fact that the reads can fail.
The actual solution for solving the problem we wanted ended up being "Use the map-reduce library", since we were trying to look at each model for analysis.
I want to check users' subscribed dates for certain period. And send mail to users whose subscription is finishing (ex. reminds two days).
I think the best way is using thread and timer to check dates. But I have no idea how to call this function. I don't want to make a separate program or shell. I want to combine this procedure to my django code. I tried to call this function in my settings.py file. But it seems it is not a good idea. It calls the function and creates thread every time I imported settings.
That's case for manage.py command called periodically from cron. Oficial doc about creating those commands. Here bit more helpful.
If you want something simpler then django-command-extensions has commands for managing django jobs.
if you need more then only this one asynchronous job have a look at celery.
using Django-cron is much easier and simple
EDIT: Added a tip
from django_cron import cronScheduler, Job
class sendMail(Job):
# period run every 300 seconds (5 minutes)
run_every = 300
def job(self):
# This will be executed every 5 minutes
datatuple = check_subscription_finishing()
send_mass_mail(datatuple)
//and just register it
cronScheduler.register(sendMail)