We Keep Coding

crontab python script query

I have a python script on a Pi3 which sends sensor readings to a mysql database, which I would like to run at boot. I have tried several combinations of #boot within crontab, but the database table never gets any fresh data.
The first line of the script is...
#!/usr/bin/python
and the script runs with:
./distance2.py
#reboot /home/pi/distance2.py &
# #reboot cd /pyhome/pi/Pimoroni/VL53L1X/Examples && sudo python distance2.py
# #reboot /home/pi/Pimoroni/VL53L1X/Examples/distance2.py &
(I moved the script from the Pimoroni directory for the sake of simplicity.)
When run from terminal, the script works perfectly:
pi#raspberrypi:~ $ ./distance2.py
distance.py
Display the distance read from the sensor.
Uses the "Short Range" timing budget by default.
Press Ctrl+C to exit.
VL53L1X Start Ranging Address 0x29
VL53L0X_GetDeviceInfo:
Device Name : VL53L1 cut1.1
Device Type : VL53L1
Device ID :
ProductRevisionMajor : 1
ProductRevisionMinor : 15
Distance: 0mm
(1L, 'record inserted.')
Distance: 60mm
(1L, 'record inserted.')
Distance: 60mm
grep shows it's running OK (Unless the red colour of the script name text means something bad?)
ps aux | grep distance2.py
pi 1530 0.0 0.5 7332 2032 pts/0 S+ 16:20 0:00 grep --color=auto distance2.py
What's crontab #boot got against my humble project?

Try full path to the python and write the log for investigation:
#reboot /usr/bin/python /home/pi/distance2.py > /home/pi/distance2_cronjoblog 2>&1

rc.local python script can't load mysql python connector

I am running Raspbian on a Raspberry Pi. I have a python script set in /etc/rc.local to run when the device is booted. The python script was working as expected until I added mysql functionality to the script. Mysql is correctly installed and configured; the mysql functionality in the script works as expected when run manually from the shell. According to /var/log/syslog, the script is unable to find the mysql module:
Mar 17 10:18:42 PressPi rc.local[464]: Traceback (most recent call last):
Mar 17 10:18:42 PressPi rc.local[464]: File "/home/pi/python/switch_counter3.py", line 20, in <module>
Mar 17 10:18:42 PressPi rc.local[464]: import mysql.connector
Mar 17 10:18:42 PressPi rc.local[464]: ModuleNotFoundError: No module named 'mysql'
Python import statement :
import mysql.connector
/etc/rc.local entry :
python3 /home/pi/python/switch_counter3.py &
I also tried :
sudo python3 /home/pi/python/switch_counter3.py &
I tried to add a 60 second sleep in rc.local before the python script was called, in case the script was running before mysql was initialized. I can see in syslog that the script is waiting the 60 seconds before running, but I get the same error message.
I could try other methods to automate the script, but I am interested in why the current method is not working as expected.

As Tim Roberts helped me learn, the rc.local script was running as root, but I had installed mysql under my user account. Running from rc.local, the script couldn't find the mysql package. I added sudo -H -u username when calling the python script in /etc/rc.local, and the script runs as expected.
Thank you to Tim Roberts.
for those interested in the full /etc/rc.local. Everything before the sleep statement was default.
#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.
# Print the IP address
_IP=$(hostname -I) || true
if [ "$_IP" ]; then
printf "My IP address is %s\n" "$_IP"
fi
#make sure mysql has been initialized
sleep 15
#run python script as user pi with python3
#The ampersand allows the command to run in a separate process and continue booting with the process running.
sudo -H -u pi python3 /home/pi/python/switch_counter3.py &
exit 0

systemd service failed to start bash script

I am running a bash script as systemd service but it is giving me this error
Failed at step EXEC spawning /home/pipeline/entity-extraction/start_consumer.sh: Permission denied
Feb 8 11:59:58 irum systemd[1]: ee-consumer.service: main process exited, code=exited, status=203/EXEC
Feb 8 11:59:58 irum systemd[1]: Unit ee-consumer.service entered failed state.
My bash scrip is running 2 Python scripts and it runs fine when I run it from terminal as
sudo bash start_consumer.sh
start_consumer.sh
while true
do
echo "starting FIRST Consumer.py : $(date +"%T")"
python3 /home/irum/Desktop/Marketsyc/Consumer.py &
pid=$!
echo "pid:$pid"
sleep 60
echo "starting SECOND Consumer.py : $(date +"%T")"
python3 /home/irum/Desktop/Marketsyc/Consumer.py &
new_pid=$!
echo "new_pid:$new_pid"
# Here I want to kill FIRST Consumer.py
echo "killing first consumer"
kill "$pid"
sleep 60
# Here I want to kill SECOND Consumer.py
echo "killing second consumer"
kill "$new_pid"
done
code of my systemd service ee-consumer.service
[Unit]
Description=Entity extraction - consumer
After=default.target
[Service]
Type=simple
Restart=always
User=pipeline
ExecStart=/home/pipeline/entity-extraction/start_consumer.sh
how can I resolve this issue ?

You have to set the shebang line and permission to the script, for systemd to execute.
Add #!/bin/bash to the start of the bash script. And do the following,
chmod 755 /home/pipeline/entity-extraction/start_consumer.sh

Python3 script as deamon (via systemctl) and its dependencies

I am not a python developer.
I try to start a python web server script (https://github.com/rpiwalletui/qtum-ui) during startup of the machine (raspbian stretch).
I prepared a init-d script for that which seems to make a valid try but the python script reports missing dependencies.
If I run the python3 script directly, it works fine.
But trying to run it using the init.d systemctl script the script fails with the following log:
Jan 11 17:05:07 raspberrypi systemd[1]: Starting qtumui.service...
Jan 11 17:05:07 raspberrypi qtumui[12111]: Starting /home/pi/qtum-ui/app.py:Traceback (most recent call last):
Jan 11 17:05:07 raspberrypi qtumui[12111]: File "/home/pi/qtum-ui/app.py", line 3, in <module>
Jan 11 17:05:07 raspberrypi qtumui[12111]: from flask import Flask, render_template, request, flash, url_for, redirect, send_file
Jan 11 17:05:07 raspberrypi qtumui[12111]: ImportError: No module named 'flask'
Jan 11 17:05:07 raspberrypi qtumui[12111]: failed!
Jan 11 17:05:07 raspberrypi systemd[1]: Started qtumui.service.
This is the init.d script
#! /bin/sh
PATH=/bin:/usr/bin:/sbin:/usr/sbin
DAEMON=/home/pi/qtum-ui/app.py
PIDFILE=/var/run/qtumui.pid
test -x $DAEMON || exit 0
. /lib/lsb/init-functions
case "$1" in
start)
log_daemon_msg "Starting $DAEMON"
start_daemon -p $PIDFILE $DAEMON
log_end_msg $?
;;
stop)
log_daemon_msg "Stopping $DAEMON"
killproc -p $PIDFILE $DAEMON
PID=`ps x |grep qtum | head -1 | awk '{print $1}'`
kill -9 $PID
log_end_msg $?
;;
force-reload|restart)
$0 stop
$0 start
;;
status)
status_of_proc -p $PIDFILE $DAEMON atd && exit 0 || exit $?
;;
*)
echo "Usage: /etc/init.d/qtumui {start|stop|restart|force-reload|status}"
exit 1
;;
esac
exit 0
Is there something, which can be done to fix this?

OK, I managed to find out why the dependencies are missing.
I installed the dependencies as user "pi" using "pip3", in which case the modules are beeing installed into /home/pi/.local/lib/python3.5/dist-packages/.
The init script runs as some other default user, probably root, that is why the script can not find the dependencies.
To fix this I need to change the init-d script to run as user "pi", but the script works as well if I install the needed dependencies as root
sudo pip3 install flask Flask-WTF Flask-QRcode Flask-Bootstrap
In this case the modules can be found as well in /usr/local/lib/python3.5/dist-packages/ it kind of work, because of some ownership conflicts but the service starts at least.

Crontab not restarting process

I am trying to setup a crontab to run 6 python data scrapers. I am tired of having to restart them manually when one of them fails. When running the following:
> ps -ef | grep python
ubuntu 31537 1 0 13:09 ? 00:00:03 python /home/ubuntu/scrapers/datascraper1.py
etc... I get a list of the datascrapers 1-6 all in the same folder.
I edited my crontab like this:
sudo crontab -e
# m h dom mon dow command
* * * * * pgrep -f /home/ubuntu/scrapers/datascraper1.py || python /home/ubuntu/scrapers/datascraper1.py > test.out
Then I hit control+X to exit and hit yes to save as /tmp/crontab.M6sSxL/crontab .
However it does not work in restarting or even starting datascraper1.py whether I kill the process manually or if the process fails on its own. Next, I tried reloading cron but it still didn't work:
sudo cron reload
Finally I tried removing nohup from the cron statement and that also did not work.
How can I check if a cron.allow or cron.deny file exists?
Also, do I need to add a username before pgrep? I am also not sure what the "> test.out" is doing at the end of the cron statement.
After running
grep CRON /var/log/syslog
to check to see if cron ran at all, I get this output:
ubuntu#ip-172-31-29-12:~$ grep CRON /var/log/syslog
Jan 5 07:01:01 ip-172-31-29-12 CRON[31101]: (root) CMD (pgrep -f datascraper1.py ||
python /home/ubuntu/scrapers/datascraper1.py > test.out)
Jan 5 07:01:01 ip-172-31-29-12 CRON[31100]: (CRON) info (No MTA installed, discarding output)
Jan 5 07:17:01 ip-172-31-29-12 CRON[31115]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jan 5 08:01:01 ip-172-31-29-12 CRON[31140]: (root) CMD (pgrep -f datascraper1.py || python /home/ubuntu/scrapers/datascraper1.py > test.out)
Since there is evidence of Cron executing the command, there must be something wrong with this command, (note: I added the path to python):
pgrep -f datascraper1.py || /usr/bin/python /home/ubuntu/scrapers/datascraper1.py > test.out
Which is supposed to check to see if datascaper1.py is running, if not then restart it.
Since Cron is literally executing this statement:
(root) CMD (pgrep -f datascraper1.py || python /home/ubuntu/scrapers/datascraper1.py > test.out)
aka
root pgrep -f datascraper1.py
Running the above root command gives me:
The program 'root' is currently not installed. You can install it by typing:
sudo apt-get install root-system-bin
Is there a problem with Cron running commands from root?
Thanks for your help.

First of all, you need to see if cron is working at all.
Add this to your cron file (and ideally delete the python statement for now, to have a clear state)
* * * * * echo `date` >>/home/your_username/hello_cron
This will output the date in the file "hello_cron" every minute. Try this, and if this works, ie you see output every minute, write here and we can troubleshoot further.
You can also look in your system logs to see if cron has ran your command, like so:
grep CRON /var/log/syslog
Btw the >test.out part would redirect the output of the python program to the file test.out. I am not sure why you need the nohup part - this would let the python programs run even if you are logged out - is this what you want?
EDIT: After troubleshooting cron:
The message about no MTA installed means that cron is trying to send you an e-mail with the output of the job but cannot because you dont have an email programm installed:
Maybe this will fix it:
sudo apt-get install postfix
The line invoking the python program in cron is producing some output (an error) so it's in your best interests to see what happens. Look at this tutorial to see how to set your email address: http://www.cyberciti.biz/faq/linux-unix-crontab-change-mailto-settings/
Just in case the tutorial becomes unavailable:
MAILTO:youremail#example.com

You need to add python home to your path at the start of the job, however you have python set up. When you're running it yourself and you type python, it checks where you are, then one level down, then your $PATH. So, python home (where the python binary is) needs to be globally exported for the user that owns the cron (so, put it in a rc script in /etc/rc.d/) or, you need to append the python home to path at the start of the cron job. So,
export PATH=$PATH:<path to python>
Or, write the cron entry as
/usr/bin/python /home/ubuntu/etc/etc
to call it directly. It might not be /usr/bin, run the command
'which python'
to find out.
The 'No MTA' message means you are getting STDERR, which would normally get mailed to the user, but can't because you have no Mail Transfer Agent set up, like mailx, or mutt, so no mail for the user can get delivered from cron, so it is discarded. If you'd like STDERR to go into the log also, at the end, instead of
"command" > test.out
write
"command" 2>&1 > test.out
to redirect STDERR into STDOUT, then redirect both to test.out.