We Keep Coding

Unable to trap TERM signal from 'docker stop'

This is my bash script used in CMD
#!/bin/bash
set -eo pipefail
echo "Setting trap"
echo $$
echo $BASHPID
trap 'cleanup' TERM
trap 'cleanup' KILL
cleanup() {
echo "Cleaning up..."
kill -TERM `jobs -p`
}
# To start the essential services
service ntp start
service awslogs start
cd /app
python -m job_manager &
wait
The Docker file is not very interesting
FROM ubuntu:16.04
RUN apt-get update --fix-missing && apt-get install -y \
git \
python \
python-pip \
ntp \
curl
ENV APP_HOME /app
RUN mkdir -p ${APP_HOME}
COPY src/ ${APP_HOME}/
# job-cmd.sh is kept here
COPY docker/helper-files/* /
CMD /job-cmd.sh
The idea is trap the TERM signal inside job-cmd.sh and then pass on to the python task.
I have tried a number of time and it did not work. After I add these call
echo $$
echo $BASHPID
I realised the pid of the CMD process is actually 7 instead of 1 as I would expect.
My questions:
1) Why the bash process is assigned PID 7?
2) How can I fix the my job script/dockerfile?

I think this is happening because you are using the shell form of the CMD instruction. From https://docs.docker.com/engine/reference/builder/#cmd:
If you want to run your command without a shell then you must express the command as a JSON array and give the full path to the executable. This array form is the preferred format of CMD.
So, replace your CMD instruction in Dockerfile with:
CMD ["/job-cmd.sh"]
Then your Bash process will be assigned PID 1. Your TERM handler will work, but you can't trap the KILL signal. From man trap:
Trapping SIGKILL or SIGSTOP is syntactically accepted by some historical implementations, but it has no effect. Portable POSIX applications cannot attempt to trap these signals.
FYI, I explained more about the PID 1 problem here: https://serverfault.com/questions/869543/bash-script-entrypoint-pid-1-kills-tail-sub-process-only-if-a-fake-trap-whi/870872#870872

You could use trap command in the bash to do this.
#!/bin/bash
#
function gracefulShutdown {
echo "Shutting down!"
# do something..
}
trap gracefulShutdown SIGTERM TERM INT
./subprocess.sh &
tail --pid=${!} -f /dev/null &
wait "${!}"
tail command just waits for subprocess to complete, while wait command waits for the tail to complete... Now, main process is the one which is waiting on.. so any docker signals directly reach the trap we set above...
Example is available at: https://github.com/iamdvr/docker-trap-subprocess

Sleepwatcher on OS X 10.11 not executing script on wake

In installed Sleepwatcher 2.2 on OS X 10.11 and launching it via LaunchD as an agent.
It launches okay and shows up in the activity monitor.
However, I want it to fire off a python script when the computer wakes up.
My installation commands are as follows.
sudo mkdir -p /usr/local/sbin /usr/local/share/man/man8
sudo cp ~/Desktop/sleepwatcher_2.2/sleepwatcher /usr/local/sbin
sudo cp ~/Desktop/sleepwatcher_2.2/sleepwatcher.8 /usr/local/share/man/man8
sudo cp ~/Desktop/sleepwatcher_2.2/sleepwatcher/config/rc.sleep /etc
sudo cp ~/Desktop/sleepwatcher_2.2/sleepwatcher/config/rc.wakeup /etc
sudo cp ~/Desktop/sleepwatcher_2.2/sleepwatcher/config/de.bernhard-baehr.sleepwatcher-20compatibility-localuser.plist /Library/LaunchAgents
chmod +x /etc/rc.sleep
chmod +x /etc/rc.wakeup
chmod +x /usr/local/bin/test.py
My rc.wakeup file is as follows.
#!/bin/sh
/usr/local/bin/python3 /usr/local/bin/test.py
When executing Sleepwatcher at the terimnal window by typing in the following, it seems to work.
/usr/local/sbin/sleepwatcher --verbose --wakeup /usr/local/bin/test.py
However, when trying to run it as a start-up item under LaunchD, it does not seem to work execute my python script.
I have search all over and cannot figure out why it is not working when being launched in LaunchD.
Has anybody ran into this type of problem?
Thanks in advance.

I encountered similar problems so I took a different approach using another open source tool called Hammerspoon. It can provide for automation of bunch of things on MacOS including sleep/wake events. It's quite simple to replicate sleepwatcher's functionality by adding the following to Hammerspoon's ~/.hammerspoon/init.lua (or create a 'spoon') script that triggers when the machine wakes or sleeps and calls the corresponding wake and sleep scripts (in e.g. /Users/username/scripts - ensure username is changed) from sleepwatcher:
function caffeinateWatcher(eventType)
if (eventType == hs.caffeinate.watcher.systemWillSleep or
eventType == hs.caffeinate.watcher.systemWillPowerOff) then
print ("WillSleep...")
-- Execute sleep script
hs.task.new("/Users/username/scripts/rc.sleep", nil):start()
elseif (eventType == hs.caffeinate.watcher.systemDidWake) then
print ("Woken...")
-- Execute wake script
hs.task.new("/Users/username/scripts/rc.wake", nil):start()
end
end
sleepWatcher = hs.caffeinate.watcher.new(caffeinateWatcher)
sleepWatcher:start()
Note if you want Hammerspoon to launch the shell scripts you need to ensure they start with the standard bash shell header #!/bin/bash.

Fabric run command - start sh script on the remote & leaving the remote script running [duplicate]

Through Fabric, I am trying to start a celerycam process using the below nohup command. Unfortunately, nothing is happening. Manually using the same command, I could start the process but not through Fabric. Any advice on how can I solve this?
def start_celerycam():
'''Start celerycam daemon'''
with cd(env.project_dir):
virtualenv('nohup bash -c "python manage.py celerycam --logfile=%scelerycam.log --pidfile=%scelerycam.pid &> %scelerycam.nohup &> %scelerycam.err" &' % (env.celery_log_dir,env.celery_log_dir,env.celery_log_dir,env.celery_log_dir))

I'm using Erich Heine's suggestion to use 'dtach' and it's working pretty well for me:
def runbg(cmd, sockname="dtach"):
return run('dtach -n `mktemp -u /tmp/%s.XXXX` %s' % (sockname, cmd))
This was found here.

As I have experimented, the solution is a combination of two factors:
run process as a daemon: nohup ./command &> /dev/null &
use pty=False for fabric run
So, your function should look like this:
def background_run(command):
command = 'nohup %s &> /dev/null &' % command
run(command, pty=False)
And you can launch it with:
execute(background_run, your_command)

This is an instance of this issue. Background processes will be killed when the command ends. Unfortunately on CentOS 6 doesn't support pty-less sudo commands.
The final entry in the issue mentions using sudo('set -m; service servicename start'). This turns on Job Control and therefore background processes are put in their own process group. As a result they are not terminated when the command ends.
For even more information see this link.

you just need to run
run("(nohup yourcommand >& /dev/null < /dev/null &) && sleep 1")

DTACH is the way to go. It's a software you need to install like a lite version of screen.
This is a better version of the "dtach"-method found above, it will install dtach if necessary. It's to be found here where you can also learn how to get the output of the process which is running in the background:
from fabric.api import run
from fabric.api import sudo
from fabric.contrib.files import exists
def run_bg(cmd, before=None, sockname="dtach", use_sudo=False):
"""Run a command in the background using dtach
:param cmd: The command to run
:param output_file: The file to send all of the output to.
:param before: The command to run before the dtach. E.g. exporting
environment variable
:param sockname: The socket name to use for the temp file
:param use_sudo: Whether or not to use sudo
"""
if not exists("/usr/bin/dtach"):
sudo("apt-get install dtach")
if before:
cmd = "{}; dtach -n `mktemp -u /tmp/{}.XXXX` {}".format(
before, sockname, cmd)
else:
cmd = "dtach -n `mktemp -u /tmp/{}.XXXX` {}".format(sockname, cmd)
if use_sudo:
return sudo(cmd)
else:
return run(cmd)
May this help you, like it helped me to run omxplayer via fabric on a remote rasberry pi!

You can use :
run('nohup /home/ubuntu/spider/bin/python3 /home/ubuntu/spider/Desktop/baidu_index/baidu_index.py > /home/ubuntu/spider/Desktop/baidu_index/baidu_index.py.log 2>&1 &', pty=False)

nohup did not work for me and I did not have tmux or dtach installed on all the boxes I wanted to use this on so I ended up using screen like so:
run("screen -d -m bash -c '{}'".format(command), pty=False)
This tells screen to start a bash shell in a detached terminal that runs your command

You could be running into this issue
Try adding 'pty=False' to the sudo command (I assume virtualenv is calling sudo or run somewhere?)

This worked for me:
sudo('python %s/manage.py celerycam --detach --pidfile=celerycam.pid' % siteDir)
Edit: I had to make sure the pid file was removed first so this was the full code:
# Create new celerycam
sudo('rm celerycam.pid', warn_only=True)
sudo('python %s/manage.py celerycam --detach --pidfile=celerycam.pid' % siteDir)

I was able to circumvent this issue by running nohup ... & over ssh in a separate local shell script. In fabfile.py:
#task
def startup():
local('./do-stuff-in-background.sh {0}'.format(env.host))
and in do-stuff-in-background.sh:
#!/bin/sh
set -e
set -o nounset
HOST=$1
ssh $HOST -T << HERE
nohup df -h 1>>~/df.log 2>>~/df.err &
HERE
Of course, you could also pass in the command and standard output / error log files as arguments to make this script more generally useful.
(In my case, I didn't have admin rights to install dtach, and neither screen -d -m nor pty=False / sleep 1 worked properly for me. YMMV, especially as I have no idea why this works...)

ubuntu ec2 - run python script at startup with arguments

I have a python script i'd like to start on startup on an ubuntu ec2 instance but im running into troubles.
The script runs in a loop and takes care or exiting when its ready so i shouldn't need to start or stop it after its running.
I've read and tried a lot of approaches with various degrees of success and honestly im confused about whats the best approach. I've tried putting a shell script that starts the python script in /etc/init.d, making it executable and doing update-rc.d to try to get it to run but its failed at every stage.
here's the contents of the script ive tried:
#!/bin/bash
cd ~/Dropbox/Render\ Farm\ 1/appleseed/bin
while :
do
python ./watchfolder18.py -t ./appleseed.cli -u ec2 ../../data/
done
i then did
sudo chmod +x /etc/init.d/script_name
sudo sudo update-rc.d /etc/init.d/script_name defaults
This doesn't seem to run on startup and i cant see why, if i run the command manually it works as expected.
I also tried adding a line to rc.local to start the script but that doesn't seem to work either
Can anybody share what they have found is the simplest way to run a python script in the background with arguments on startup of an ec2 instance.
UPDATE: ----------------------
I've since moved this code to a file called /home/ubuntu/bin/watch_folder_start
#!/bin/bash
cd /home/ubuntu/Dropbox/Render\ Farm\ 1/appleseed/bin
while :
do
python ./watchfolder18.py -t ./appleseed.cli -u ec2 ../../data/
done
and changed my rc.local file to this:
nohup /home/ubuntu/bin/watch_folder_start &
exit 0
Which works when i manually run rc.local but wont fire on startup, i did chmod +x rc.local but that didn't change anything,

Your /etc/init.d/script_name is missing the plumbing that update-rc.d and so on use, and won't properly handle stop, start, and other init-variety commands, so...
For initial experimentation, take advantage of the /etc/init.d/rc.local script (which should be linked to by default from /etc/rc2/S99rc.local). The gets you out of having to worry about the init.d conventions and just add things to /etc/rc.local before the exit 0 at its end.
Additionally, that ~ isn't going to be defined, you'll need to use a full pathname - and furthermore the script will run as root. We'll address how to avoid this if desired in a bit. In any of these, you'll need to replace "whoeveryouare" with something more useful. Also be warned that you may need to prefix the python command with a su command and some arguments to get the process to run with the user id you might need.
You might try (in /etc/rc.local):
( if cd '/home/whoeveryouare/Dropbox/Render Farm 1/appleseed/bin' ; then
while : ; do
# This loop should respawn watchfolder18.py if it dies, but
# ideally one should fix watchfolder18.py and remove this loop.
python ./watchfolder18.py -t ./appleseed.cli -u ec2 ../../data/
done
else
echo warning: could not find watchfolder 1>&2
fi
) &
You could also put all that in a script and just call it from /etc/rc.local.
The first pass is roughly what you had, but if we assume that watchfolder18.py will arrange to avoid dying we can cut it down to:
( cd '/home/whoeveryouare/Dropbox/Render Farm 1/appleseed/bin' \
&& exec python ./watchfolder18.py -t ./appleseed.cli -u ec2 ../../data/ ) &
These aren't all that pretty, but it should let you get your daemon sorted out so you can debug it and so on, then come back to making a proper /etc/init.d or /etc/init script later. Something like this might work in /etc/init/watchfolder.conf, but I'm not yet facile enough to claim this is anything other than a rough stab at it:
# watchfolder - spawner for watchfolder18.py
description "watchfolder program"
start on runlevel [2345]
stop on runlevel [!2345]
script
if cd '/home/whoeveryouare/Dropbox/Render Farm 1/appleseed/bin' ; then
exec python ./watchfolder18.py -t ./appleseed.cli -u ec2 ../../data/0
fi
end script

I found that the best solution in the end was to use 'upstart' and create a file in etc/init called myfile.conf that contained the following
description "watch folder service"
author "Jonathan Topf"
start on startup
stop on shutdown
# Automatically Respawn:
respawn
respawn limit 99 5
script
HOST=`hostname`
chdir /home/ubuntu/Dropbox/Render\ Farm\ 1/appleseed/bin
exec /usr/bin/python ./watchfolder.py -t ./appleseed.cli -u $HOST ../../data/ >> /home/ubuntu/bin/ec2_server.log 2>&1
echo "watch_folder started"
end script
More info on using the upstart system here
http://upstart.ubuntu.com/
https://help.ubuntu.com/community/UbuntuBootupHowto
http://blog.joshsoftware.com/2012/02/14/upstart-scripts-in-ubuntu/

Python script as linux service/daemon

Hallo,
I'm trying to let a python script run as service (daemon) on (ubuntu) linux.
On the web there exist several solutions like:
http://pypi.python.org/pypi/python-daemon/
A well-behaved Unix daemon process is tricky to get right, but the required steps are much the same for every daemon program. A DaemonContext instance holds the behaviour and configured process environment for the program; use the instance as a context manager to enter a daemon state.
http://www.jejik.com/articles/2007/02/a_simple_unix_linux_daemon_in_python/
However as I want to integrate my python script specifically with ubuntu linux my solution is a combination with an init.d script
#!/bin/bash
WORK_DIR="/var/lib/foo"
DAEMON="/usr/bin/python"
ARGS="/opt/foo/linux_service.py"
PIDFILE="/var/run/foo.pid"
USER="foo"
case "$1" in
start)
echo "Starting server"
mkdir -p "$WORK_DIR"
/sbin/start-stop-daemon --start --pidfile $PIDFILE \
--user $USER --group $USER \
-b --make-pidfile \
--chuid $USER \
--exec $DAEMON $ARGS
;;
stop)
echo "Stopping server"
/sbin/start-stop-daemon --stop --pidfile $PIDFILE --verbose
;;
*)
echo "Usage: /etc/init.d/$USER {start|stop}"
exit 1
;;
esac
exit 0
and in python:
import signal
import time
import multiprocessing
stop_event = multiprocessing.Event()
def stop(signum, frame):
stop_event.set()
signal.signal(signal.SIGTERM, stop)
if __name__ == '__main__':
while not stop_event.is_set():
time.sleep(3)
My question now is if this approach is correct. Do I have to handle any additional signals? Will it be a "well-behaved Unix daemon process"?

Assuming your daemon has some way of continually running (some event loop, twisted, whatever), you can try to use upstart.
Here's an example upstart config for a hypothetical Python service:
description "My service"
author "Some Dude <blah#foo.com>"
start on runlevel [234]
stop on runlevel [0156]
chdir /some/dir
exec /some/dir/script.py
respawn
If you save this as script.conf to /etc/init you simple do a one-time
$ sudo initctl reload-configuration
$ sudo start script
You can stop it with stop script. What the above upstart conf says is to start this service on reboots and also restart it if it dies.
As for signal handling - your process should naturally respond to SIGTERM. By default this should be handled unless you've specifically installed your own signal handler.

Rloton's answer is good. Here is a light refinement, just because I spent a ton of time debugging. And I need to do a new answer so I can format properly.
A couple other points that took me forever to debug:
When it fails, first check /var/log/upstart/.log
If your script implements a daemon with python-daemon, you do NOT use the 'expect daemon' stanza. Having no 'expect' works. I don't know why. (If anyone knows why - please post!)
Also, keep checking "initctl status script" to make sure you are up (start/running). (and do a reload when you update your conf file)
Here is my version:
description "My service"
author "Some Dude <blah#foo.com>"
env PYTHON_HOME=/<pathtovirtualenv>
env PATH=$PYTHON_HOME:$PATH
start on runlevel [2345]
stop on runlevel [016]
chdir <directory>
# NO expect stanza if your script uses python-daemon
exec $PYTHON_HOME/bin/python script.py
# Only turn on respawn after you've debugged getting it to start and stop properly
respawn