We Keep Coding

launching python script in terminal at boot and keeping open if/when exception

writing a data logging program that is intended to run when raspberry boots. I'm using lxsessions autostart to launch a shell script that has the command to launch my python program (my python script requires sudo)
while I continue to debug I would like the terminal window to stay open if/when it encounters an error.
I had done this successfully once before but lost my work.
my autostart file is:
#!/bin/bash
#lxpanel --profile LXDE-pi
#pcmanfm --desktop --profile LXDE-pi
#lxterminal -e sudo sh /home/pi/launcher.sh
#xscreensaver -no-splash
my script file is:
#!/bin/sh
echo Script is running
sudo /usr/bin/python3 /home/pi/hms/hms5-1.py
I thought something like this (in the autstart file) would work, but no:
#lxterminal -e -hold sudo sh /home/pi/launcher.sh
a simple internet search spit out of examples on how to execute command at boot, even launching scripts but nothing has helped so far. Thank you in advance.....

So I rebuilt my Raspberry Pi and had to go through this again. So after I was able to get it to work once, I followed my instructions from before, edited them to be clearer the posted here. NOTE - I think the mistake I made before was using sudo (sudo nano) when I should have just used nano....
Also note the python program I am launching is /home/pi/hms/hms2-v2.py
*** setting this up is a 4 step process ***
YOU MUST HAVE XTERM
STEP 1 - INSTALL XTERM:
sudo apt-get install xterm
STEP 2
read https://www.raspberrypi.org/forums/viewtopic.php?t=227191
FIRST CREATE autostart here: /home/pi/.config/lxsession/LXDE-pi/autostart
NOTE THE FOLDERS BELOW /home/pi/.config/ MAY NOT EXIST, IF NOT CREATE THEM EXACTLY AS ABOVE. NOTE: the directory must be LXDE-pi NOT LXDE
Then edit the autostart file
by using nano ~/.config/lxsession/LXDE-pi/autostart
NOTE: DO NOT use sudo in the above command
put the following in the file:
#!/bin/bash
#lxpanel --profile LXDE-pi
#pcmanfm --desktop --profile LXDE-pi
sh /home/pi/launcher.sh
#xscreensaver -no-splash
Step 3
create the script (.sh) file: launcher.sh in the directory /home/pi
include the following in the file launcher.sh:
#!/bin/sh
echo starting script
xterm -T "HMS" -geometry 100x70+10+35 -hold -e sudo /usr/bin/python3 /home/pi/hms/hms2-v2.py
Step 4
make the .sh file executable with: sudo chmod +x launcher.sh

How can I start a bash script 20 seconds after startup?

I got a bash script that stops a program, mounts the pi and starts the program again. I would like to start it on startup, but after the program itself started. So my idea was to simply wait some time (20 or 30 seconds) and start the script then (task.sh). Any idea how I can do that? Or any other idea how to solve this? (let the script wait for the program to start won't work i guess, cause then the script would restart after the program is restarted, right?)
Thanks and Greetings,
Elias

Thanks for all the help, but I solved it myself following this tutorial:
create a new file in /etc/init.d/, I'll call it example in this.
So:
sudo nano /etc/init.d/example
This will be a file that will be executed after raspberry pi startup.
The code for this file is the following:
#!/bin/sh
### BEGIN INIT INFO
# Provides: Für welches Programm ist das Script?
# Required-Start:
# Required-Stop:
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Kurze Beschreibung
# Description: Längere Beschreibung
### END INIT INFO
# Actions
case "$1" in
start)
# START
;;
stop)
# STOP
;;
restart)
# RESTART
;;
esac
exit 0
Because I wanted this script to start /home/pi/task.sh at startup and reboot, I simply put /home/pi/task.sh before the ;; after # START and # RESTART
If you've done that, save it and exit it. Then type
sudo chmod +x /etc/init.d/example
to make the script executable. The last step is to define the runlevels:
sudo update-rc.d example defaults
After that you can reboot and see if it works.
I hope that made it clear for everyone :)
(And just for me: https://jankarres.de/2014/07/raspberry-pi-autostart-von-programmen-einrichten/ :D)

Assuming that you are using a Debian / Debian derivative distro (Ubuntu / Mint / Etc) here's how to achieve the result you look for.
Create your script in the path you've suggested using any text editor (here I use the simple nano):
nano /home/pi/task.sh
Paste into your task.sh:
sleep 40
/home/pi/pi_video_looper/disable.sh
mount -a
/home/pi/pi_video_looper/install.sh
Make the script executable:
chmod +x /home/pi/task.sh
Make sure the script works running it:
/home/pi/task.sh
Once you're sure that the script works fine edit your rc.local:
sudo nano /etc/rc.local
A key concept here is that whatever you put in rc.local will be executed with root permissions.
For this reason there is no need to use sudo.
Add before exit 0 the following:
/home/pi/task.sh
Reboot and test

Sleepwatcher on OS X 10.11 not executing script on wake

In installed Sleepwatcher 2.2 on OS X 10.11 and launching it via LaunchD as an agent.
It launches okay and shows up in the activity monitor.
However, I want it to fire off a python script when the computer wakes up.
My installation commands are as follows.
sudo mkdir -p /usr/local/sbin /usr/local/share/man/man8
sudo cp ~/Desktop/sleepwatcher_2.2/sleepwatcher /usr/local/sbin
sudo cp ~/Desktop/sleepwatcher_2.2/sleepwatcher.8 /usr/local/share/man/man8
sudo cp ~/Desktop/sleepwatcher_2.2/sleepwatcher/config/rc.sleep /etc
sudo cp ~/Desktop/sleepwatcher_2.2/sleepwatcher/config/rc.wakeup /etc
sudo cp ~/Desktop/sleepwatcher_2.2/sleepwatcher/config/de.bernhard-baehr.sleepwatcher-20compatibility-localuser.plist /Library/LaunchAgents
chmod +x /etc/rc.sleep
chmod +x /etc/rc.wakeup
chmod +x /usr/local/bin/test.py
My rc.wakeup file is as follows.
#!/bin/sh
/usr/local/bin/python3 /usr/local/bin/test.py
When executing Sleepwatcher at the terimnal window by typing in the following, it seems to work.
/usr/local/sbin/sleepwatcher --verbose --wakeup /usr/local/bin/test.py
However, when trying to run it as a start-up item under LaunchD, it does not seem to work execute my python script.
I have search all over and cannot figure out why it is not working when being launched in LaunchD.
Has anybody ran into this type of problem?
Thanks in advance.

I encountered similar problems so I took a different approach using another open source tool called Hammerspoon. It can provide for automation of bunch of things on MacOS including sleep/wake events. It's quite simple to replicate sleepwatcher's functionality by adding the following to Hammerspoon's ~/.hammerspoon/init.lua (or create a 'spoon') script that triggers when the machine wakes or sleeps and calls the corresponding wake and sleep scripts (in e.g. /Users/username/scripts - ensure username is changed) from sleepwatcher:
function caffeinateWatcher(eventType)
if (eventType == hs.caffeinate.watcher.systemWillSleep or
eventType == hs.caffeinate.watcher.systemWillPowerOff) then
print ("WillSleep...")
-- Execute sleep script
hs.task.new("/Users/username/scripts/rc.sleep", nil):start()
elseif (eventType == hs.caffeinate.watcher.systemDidWake) then
print ("Woken...")
-- Execute wake script
hs.task.new("/Users/username/scripts/rc.wake", nil):start()
end
end
sleepWatcher = hs.caffeinate.watcher.new(caffeinateWatcher)
sleepWatcher:start()
Note if you want Hammerspoon to launch the shell scripts you need to ensure they start with the standard bash shell header #!/bin/bash.

Docker timeout for container?

For my dissertation at University, I'm working on a coding leaderboard system where users can compile / run untrusted code through temporary docker containers. The system seems to be working well so far, but one problem I'm facing is that when code for an infinite loop is submitted, E.g:
while True:
print "infinite loop"
the system goes haywire. The problem is that when I'm creating a new docker container, the Python interpreter prevents docker from killing the child container as data is still being printed to STDOUT (forever). This leads to the huge vulnerability of docker eating up all available system resources until the machine using the system completely freezes (shown below):
So my question is, is there a better way of setting a timeout on a docker container than my current method that will actually kill the docker container and make my system secure (code originally taken from here)?
#!/bin/bash
set -e
to=$1
shift
cont=$(docker run --rm "$#")
code=$(timeout "$to" docker wait "$cont" || true)
docker kill $cont &> /dev/null
echo -n 'status: '
if [ -z "$code" ]; then
echo timeout
else
echo exited: $code
fi
echo output:
# pipe to sed simply for pretty nice indentation
docker logs $cont | sed 's/^/\t/'
docker rm $cont &> /dev/null
Edit: The default timeout in my application (passed to the $to variable) is "10s" / 10 seconds.
I've tried looking into adding a timer and sys.exit() to the python source directly, but this isn't really a viable option as it seems rather insecure because the user could submit code to prevent it from executing, meaning the problem would still persist. Oh the joys of being stuck on a dissertation... :(

You could set up your container with a ulimit on the max CPU time, which will kill the looping process. A malicious user can get around this, though, if they're root inside the container.
There's another S.O. question, "Setting absolute limits on CPU for Docker containers" that describes how to limit the CPU consumption of containers. This would allow you to reduce the effect of malicious users.
I agree with Abdullah, though, that you ought to be able to docker kill the runaway from your supervisor.

If you want to run the containers without providing any protection inside them, you can use runtime constraints on resources.
In your case, -m 100M --cpu-quota 50000 might be reasonable.
That way it won't eat up the parent's system resources until you get around to killing it.

I have achieved a solution for this problem.
First you must kill docker container when time limit is achieved:
#!/bin/bash
set -e
did=$(docker run -it -d -v "/my_real_path/$1":/usercode virtual_machine ./usercode/compilerun.sh 2>> $1/error.txt)
sleep 10 && docker kill $did &> /dev/null && echo -n "timeout" >> $1/error.txt &
docker wait "$did" &> /dev/null
docker rm -f $ &> /dev/null
The container runs in detached mode (-d option), so it runs in the background.
Then you run sleep also in the background.
Then wait for the container to stop. If it doesnt stop in 10 seconds (sleep timer), the container will be killed.
As you can see, the docker run process calls a script named compilerun.sh:
#!/bin/bash
gcc -o /usercode/file /usercode/file.c 2> /usercode/error.txt && ./usercode/file < /usercode/input.txt | head -c 1M > /usercode/output.txt
maxsize=1048576
actualsize=$(wc -c <"/usercode/output.txt")
if [ $actualsize -ge $maxsize ]; then
echo -e "1MB file size limit exceeded\n\n$(cat /usercode/output.txt)" > /usercode/output.txt
fi
It starts by compiling and running a C program (its my use case, I am sure the same can be done for python compiller).
This part:
command | head -c 1M > /usercode/output.txt
Is responsible for the max output size thing. It allows output to be 1MB maximum.
After that, I just check if file is 1MB. If true, write a message inside (at the beginning of) the output file.

The --stop-timeout option is not killing the container if the timeout is exceeded.
Instead, use --ulimit --cpu=timeout to kill the container if the timeout is exceeded.
This is based on the CPU time for the process inside the container.

I guess, you can use signals in python like unix to set timeout. you can use alarm of specific time say 50 seconds and catch it. Following link might help you.
signals in python

Use --stop-timeout option while running your docker container. this will execute SIGKILL once the timeout occured

Crontab not restarting process

I am trying to setup a crontab to run 6 python data scrapers. I am tired of having to restart them manually when one of them fails. When running the following:
> ps -ef | grep python
ubuntu 31537 1 0 13:09 ? 00:00:03 python /home/ubuntu/scrapers/datascraper1.py
etc... I get a list of the datascrapers 1-6 all in the same folder.
I edited my crontab like this:
sudo crontab -e
# m h dom mon dow command
* * * * * pgrep -f /home/ubuntu/scrapers/datascraper1.py || python /home/ubuntu/scrapers/datascraper1.py > test.out
Then I hit control+X to exit and hit yes to save as /tmp/crontab.M6sSxL/crontab .
However it does not work in restarting or even starting datascraper1.py whether I kill the process manually or if the process fails on its own. Next, I tried reloading cron but it still didn't work:
sudo cron reload
Finally I tried removing nohup from the cron statement and that also did not work.
How can I check if a cron.allow or cron.deny file exists?
Also, do I need to add a username before pgrep? I am also not sure what the "> test.out" is doing at the end of the cron statement.
After running
grep CRON /var/log/syslog
to check to see if cron ran at all, I get this output:
ubuntu#ip-172-31-29-12:~$ grep CRON /var/log/syslog
Jan 5 07:01:01 ip-172-31-29-12 CRON[31101]: (root) CMD (pgrep -f datascraper1.py ||
python /home/ubuntu/scrapers/datascraper1.py > test.out)
Jan 5 07:01:01 ip-172-31-29-12 CRON[31100]: (CRON) info (No MTA installed, discarding output)
Jan 5 07:17:01 ip-172-31-29-12 CRON[31115]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jan 5 08:01:01 ip-172-31-29-12 CRON[31140]: (root) CMD (pgrep -f datascraper1.py || python /home/ubuntu/scrapers/datascraper1.py > test.out)
Since there is evidence of Cron executing the command, there must be something wrong with this command, (note: I added the path to python):
pgrep -f datascraper1.py || /usr/bin/python /home/ubuntu/scrapers/datascraper1.py > test.out
Which is supposed to check to see if datascaper1.py is running, if not then restart it.
Since Cron is literally executing this statement:
(root) CMD (pgrep -f datascraper1.py || python /home/ubuntu/scrapers/datascraper1.py > test.out)
aka
root pgrep -f datascraper1.py
Running the above root command gives me:
The program 'root' is currently not installed. You can install it by typing:
sudo apt-get install root-system-bin
Is there a problem with Cron running commands from root?
Thanks for your help.

First of all, you need to see if cron is working at all.
Add this to your cron file (and ideally delete the python statement for now, to have a clear state)
* * * * * echo `date` >>/home/your_username/hello_cron
This will output the date in the file "hello_cron" every minute. Try this, and if this works, ie you see output every minute, write here and we can troubleshoot further.
You can also look in your system logs to see if cron has ran your command, like so:
grep CRON /var/log/syslog
Btw the >test.out part would redirect the output of the python program to the file test.out. I am not sure why you need the nohup part - this would let the python programs run even if you are logged out - is this what you want?
EDIT: After troubleshooting cron:
The message about no MTA installed means that cron is trying to send you an e-mail with the output of the job but cannot because you dont have an email programm installed:
Maybe this will fix it:
sudo apt-get install postfix
The line invoking the python program in cron is producing some output (an error) so it's in your best interests to see what happens. Look at this tutorial to see how to set your email address: http://www.cyberciti.biz/faq/linux-unix-crontab-change-mailto-settings/
Just in case the tutorial becomes unavailable:
MAILTO:youremail#example.com

You need to add python home to your path at the start of the job, however you have python set up. When you're running it yourself and you type python, it checks where you are, then one level down, then your $PATH. So, python home (where the python binary is) needs to be globally exported for the user that owns the cron (so, put it in a rc script in /etc/rc.d/) or, you need to append the python home to path at the start of the cron job. So,
export PATH=$PATH:<path to python>
Or, write the cron entry as
/usr/bin/python /home/ubuntu/etc/etc
to call it directly. It might not be /usr/bin, run the command
'which python'
to find out.
The 'No MTA' message means you are getting STDERR, which would normally get mailed to the user, but can't because you have no Mail Transfer Agent set up, like mailx, or mutt, so no mail for the user can get delivered from cron, so it is discarded. If you'd like STDERR to go into the log also, at the end, instead of
"command" > test.out
write
"command" 2>&1 > test.out
to redirect STDERR into STDOUT, then redirect both to test.out.