I have a Python/Django application running on an Ubuntu Server. The application updates the stock for a webshop.
I have made a job in crontab to update the stock every night.
# daily stock update, starts at 23:30
30 23 * * * ./Cronjobs/stockUpdate.sh
stockUpdate.sh:
#!/bin/bash
cd Projecten
source booksVenv/bin/activate
cd Books/books
cat stockUpdate.py | python manage.py shell
stockUpdate.py:
from core.models import Stock
Stock().getAllStock()
Running Stock().getAllStock() by hand works fine.
Example: I login on the server via ssh, start the virtual environment, start django shell and run the getAllStock method.
However, contrab just seems to stop while running getAllStock without an error. A log is placed in /var/mail/.
When I open the file with nano here is what I get.
# more than 500 pages of prints
Go to next page.
>>>
Here is what I think could be going wrong:
* I use too many print statements in my code, they mess up the job.
* The job starts at 23:30 but takes a few hours, it is stopped the next day (after a half hour).
Can someone please tell me why this is occurring and give me some details as to how to debug and fix the issue.
Related
I have a question regarding one of my React apps that I recently developed. It's basically a landing page, which is using React frontend and Node+Express backend and its scraping data from various pages (scrapers are developed in Python).
Right now, the React app itself is hosted in Heroku and the execution of scrapers is working, but it's not scheduled automatically.
Current setup is the following:
EC2 for Python scrapers
AWS RDS MYSQL for database where I write the data to from the EC2 scrapers
I have created a separate file to execute all the others scrapers.
main.py
import time
import schedule
import os
from pathlib import Path
print('python script executed')
# make sure, what is the current working directory to add the right paths to scrapers
path = os.getcwd()
print(path)
#exec(open("/home/ec2-user/testing_python/lhvscraper.py").read())
filenames = [
#output table: fundsdata
Path("/home/ec2-user/testing_python/lhvscraper.py"),
Path("/home/ec2-user/testing_python/luminorscrapertest.py"),
Path("/home/ec2-user/testing_python/sebscraper.py"),
Path("/home/ec2-user/testing_python/swedscraper.py"),
Path("/home/ec2-user/testing_python/tulevascraper.py"),
#output table: feesdata
Path("/home/ec2-user/testing_python/feesscraper.py"),
#output table: yield_y1_data
Path("/home/ec2-user/testing_python/yield_1y_scraper.py"),
#output table: navdata
#Path("/home/ec2-user/testing_python/navscraper.py"),
]
def main_scraper_scheduler():
print("scheduler is working")
for filename in filenames:
print(filename)
with open(filename) as infile:
exec(infile.read())
time.sleep(11)
schedule.every(10).seconds.do(main_scraper_scheduler)
while True:
schedule.run_pending()
time.sleep(1)
I have successfully established a connection between MYSQL and EC2 and tested it on Putty -
which means, if I execute my main.py, all of the scrapers are working, inserting new data to the MYSQL database tables, and then repeat again (see the code above). The only thing is that when I close Putty (kill the connection), then the main.py function stops running.
So my question is: how to set it up like that, so that main.py file would always keep running (let's say, once a day at 12 PM) without me executing it?
I understand it's about setting up cron job or scheduler (or smth like that), but I didn't manage to set it up right now, so yours' help is very much needed.
Thanks in advance!
To avoid making crontab files overly long, Linux has canned entries that run things hourly, daily, weekly, or monthly. You don't have to modify any crontab to use this. ANY executable script that is located in /etc/cron.hourly will automatically be run once an hour. ANY executable script that is located in /etc/cron.daily will automatically be run once per day (usually at 6:30 AM), and so on. Just make sure to include a #! line for Python, and chmod +x to make it executable. Remember that it will run as root, and you can't necessarily predict which directory it will start in. Make no assumptions.
The alternative is to add a line to your own personal crontab. You can list your crontab with crontab -l, and you can edit it with crontab -e. To run something once a day at noon, you might add:
0 12 * * * /home/user/src/my_daily_script.py
I am looking to run a Python script every n days as part of a Django app.
I need this script to add new rows to the database for each attribute that a user has.
For example a User has many Sites each that have multiple Metric data points. I need to add a new Metric data point for each Site every n days by running a .py script.
I have already written the script itself, but it just works locally.
Is it the right approach to:
1) Get something like Celery or just a simple cron task running to run the python script every n days.
2) Have the python script run through all the Metrics for each Site and add a new data point by executing a SQL command from the python script itself?
You can use Django's manage commad, then use crontab to run this command circularly.
The first step is to write the script - which you have already done. This script should run the task which is to be repeated.
The second step is to schedule the execution.
The de-facto way of doing that is to write a cron entry for the script. This will let the system's cron daemon trigger the script at the interval you select.
To do so, create an entry in the crontab file for the user that owns the script; the command to do so is crontab -e, which will load a plain text file in your preferred editor.
Next, you need to add an entry to this file. The format is:
minutes hours day-of-month month day-of-week script-to-execute
So if you want to run your script every 5 days:
0 0 */5 * * /bin/bash /home/user/path/to/python.py
I have a shell script that I want to run automatically every day at 08 AM, and I am not authorised to use the crontab because I don't have root permission
My home directory is /home/user1/.
Any suggestions?
Ideally you should have your system administrator add your user account to /etc/cron.allow - without that you do not have permission to use the crontab command, as you probably discovered.
If that is not possible, then you could use a wrapper shell script along these lines (note: untested):
#!/bin/bash
TIME="tomorrow 8am"
while :; do
# Get the current time in seconds since the Epoch
CURRENT=$(date +%s)
# Get the next start time in seconds
NEXT=$(date +%s -d "$TIME")
# Sleep for the intervening time
sleep $((NEXT-CURRENT))
# Execute the command to repeat
/home/user1/mycommand.sh
done
You start the wrapper script in the background, or e.g. in a screen session, and as long as it's active it will regularly execute your script. Keep in mind that:
Much like cron there is no real accuracy w.r.t. the start time. If you care about timing to the second, then this is not the way to go.
If the script is interrupted for whatever reason, such as a server reboot, you will have to somehow restart. Normally I'd suggest an #reboot entry in crontab, but that seems to not be an option for you.
If there is some sort of process-cleaning mechanism that kills long-term user processed you are probably out of luck.
Your system administrator may have simply neglected to allow users access to cron - or it may have been an explicit decision. In the second case they might not take too well to you leaving a couple of processes overnight in order to bypass that restriction.
Even if you dont have root permission you can set cron job. Chcek these 2 commands as user1, if you can modify it or its throwing any error.
crontab -l
If you can see then try this as well:
crontab -e
If you can open and edit, then you can run that script with cron.
by adding this line:
* 08 * * * /path/to/your/script
I don't think root permission is required to create a cron job. Editing a cronjob that's not owned by you - there's where you'd need root.
In a pinch, you can use at(1). Make sure the program you run reschedules the at job. Warning: this goes to heck if the machine is down for any length of time.
I have correct python3 program looking like *.py.
I have Digital Ocean(DO) droplet with Ubuntu 14.04.
My program post message to my twitter account.
I just copy my *.py in some directory on DO droplet and run it with ssh and all works fine.
But I need to post message(rum my program) automatically every 15-30 min for example.
Iam newbie with this all.
What should i do? Step-by-step please!
cron is probably the way to go - it's built for this task. See this DigitalOcean tutorial for details.
This StackOverflow answer explicitly states what to put in for a 30 minute repeat interval.
If you don't want to use cron for some reason, you could do something like:
import time
# Repeat forever
while True:
post_to_twitter() # Call your function
# Sleep for 60 seconds / minute * 30 minutes
time.sleep(60 * 30)
First install and enable fcron. Then, sudo -s into root and run fcrontab -e. In the editor, enter */30 * * * /path/to/script.py and save the file. Change 30 to 15 if every 15 minutes is what you're after.
Im in trouble with Saltstack since I started 2 diferent developments with Python using its API. Sometimes the services crashes and when I try to start them again or reboot the servers, the time it takes to start is about more than 24 hours. Logs are empty and if i start salt-master in debug mode nothing happens.
# dpkg -l| grep salt
ii salt-common 2014.1.5+ds-1~bpo70+1
ii salt-master 2014.1.5+ds-1~bpo70+1
Note: It's happening to me in two different machines. OS Debian sid
Whoa, 24 hours is a ridiculous amount of time to start up.
Have you added any custom grains, modules or external pillars?
Have you tried upgrading? 2014.1.10 is now out.