I have a persistent background process currently running as a standalone python script on my ubuntu server, managed by supervisor. However, I am migrating to Heroku and wonder if anyone have any experiences setting up the same kind of environment.
The specifications of the script;
Fetch information from external API
Do calculations on the data
Store the data to the database
If the script used less than 5 seconds, sleep for the remaining time, else run again
I could run a cronjob every 5 seconds, but from time to time step 1-3 can take up to a full hour.
Any tips?
Thanks.
What you want to do is create a worker process. Simply define a command line script so that you can call it easily, then in your Procfile, add a new worker entry like so:
# Procfile
web: python manage.py runserver # example
worker: python manage.py start_cronjob # command to run your background process
Once you've got this defined in your Procfile, go ahead and push your app to Heroku, then scale up a worker process:
$ heroku scale worker=1
This will launch a single worker process.
To view the logs and ensure things are working as expected, you can say:
$ heroku logs --tail --ps worker
Related
I have a simply python script that I deploy on Heroku as worker:
Procfile:
worker: python main.py
The script is scheduled to run every day at a specific time with the Heroku Scheduler. I don't want it to run to other times.
Every time I push new changes to heroku (git push heroku master) the script is run automatically which I want to avoid.
How can I do that?
I looked into using another scheduler, that is set up from within the script like the APScheduler. Would this be a solution? Do I need to change my script?
Thanks!
You can scale the worker dyno formation down to zero:
If you want to stop running a particular process type entirely, simply scale it to 0
So:
heroku ps:scale worker=0
Jobs scheduled via Heroku Scheduler should still run as configured.
I have created an selenium bot that posts every 20 minutes on Instagram
I deployed my project to heroku and everything but i don't know how to make it run forever
I tried heroku run python mycode.py in the command promt but the program would stop working if i close command prompt
heroku run is for ad hoc interactive stuff.
For a long-running background process you should define a worker process in your Procfile:
worker: python mycode.py
Commit that change and redeploy. Then scale up a dyno to run it:
heroku ps:scale worker=1
This will either consume free dyno hours or, if you are using paid dynos, incur costs.
I would like to run a python script on Heroku, but I would like to run it only once and stop at the end of the script
Right now my script is running endlessly, so at the end of it, it restarts from the beginning.
How can I stop it at the end of the script ?
right now my procfile look the following;
web: python ValueAppScript.py
worker: python ValueAppScript.py
thx you
First of all, you probably don't want to declare the same command as both a web and a worker. If your script listens for HTTP requests it should be a web process, otherwise a worker makes more sense:
worker: python ValueAppScript.py
Since you don't want your worker running all the time, scale it down to zero dynos:
heroku ps:scale worker=0
If you wish to run it once interactively, you can use heroku run:
heroku run python ValueAppScript.py
If you want it to run on a schedule, e.g. once per day, you can use the Heroku Scheduler. Since you have defined this as a worker process you should be able to just use worker as the command.
I've deployed a python script on heroku and I can run that in local terminal by
heroku run python script.py
command, But when I close the local terminal the script has been stopped.
Is there a way to run deployed script through the heroku server and independent of local machine ?
You can check:
https://devcenter.heroku.com/articles/heroku-cli-commands#heroku-run-detached
heroku run:detached -t python script.py should do the trick for you
Reddy Abel Tintaya Conde's answer using heroku run:detached is good for ad hoc stuff.
But if your script should run continuously, automatically restarting when it fails, you should define a process for it in your Procfile. Such processes are often called worker processes:
worker: python script.py
Then you can scale your worker process up (or down) with heroku ps:scale:
heroku ps:scale worker=1
Whether you run your script this way or via heroku run:detached, remember that this consumes free dyno hours or, if you are using paid dynos, incurs costs.
I deployed my python script on Heroku and trying to setup scheduler to run it every hour.
My Procfile:
run: cd src && python3 main.py
I scale this dyno with:
heroku ps:scale run=1
And in heroku scheduler I set it for run every hour with command
run
Deploy and I think that it will runs every hour, but it runs more often than I expect, every 10-15 minutes, how can I solve it?
Scale your "run" process type down to 0 with heroku ps:scale run=0.
Thern, make sure you can run it in a one-off dyno, e.g. with heroku run python src/main.py.
Then in Heroku Scheduler configure the exact same command that you used to run it in a one-off dyno, and configure your required frequency (i.e. "Hourly").