We Keep Coding

Unable to trap TERM signal from 'docker stop'

This is my bash script used in CMD
#!/bin/bash
set -eo pipefail
echo "Setting trap"
echo $$
echo $BASHPID
trap 'cleanup' TERM
trap 'cleanup' KILL
cleanup() {
echo "Cleaning up..."
kill -TERM `jobs -p`
}
# To start the essential services
service ntp start
service awslogs start
cd /app
python -m job_manager &
wait
The Docker file is not very interesting
FROM ubuntu:16.04
RUN apt-get update --fix-missing && apt-get install -y \
git \
python \
python-pip \
ntp \
curl
ENV APP_HOME /app
RUN mkdir -p ${APP_HOME}
COPY src/ ${APP_HOME}/
# job-cmd.sh is kept here
COPY docker/helper-files/* /
CMD /job-cmd.sh
The idea is trap the TERM signal inside job-cmd.sh and then pass on to the python task.
I have tried a number of time and it did not work. After I add these call
echo $$
echo $BASHPID
I realised the pid of the CMD process is actually 7 instead of 1 as I would expect.
My questions:
1) Why the bash process is assigned PID 7?
2) How can I fix the my job script/dockerfile?

I think this is happening because you are using the shell form of the CMD instruction. From https://docs.docker.com/engine/reference/builder/#cmd:
If you want to run your command without a shell then you must express the command as a JSON array and give the full path to the executable. This array form is the preferred format of CMD.
So, replace your CMD instruction in Dockerfile with:
CMD ["/job-cmd.sh"]
Then your Bash process will be assigned PID 1. Your TERM handler will work, but you can't trap the KILL signal. From man trap:
Trapping SIGKILL or SIGSTOP is syntactically accepted by some historical implementations, but it has no effect. Portable POSIX applications cannot attempt to trap these signals.
FYI, I explained more about the PID 1 problem here: https://serverfault.com/questions/869543/bash-script-entrypoint-pid-1-kills-tail-sub-process-only-if-a-fake-trap-whi/870872#870872

You could use trap command in the bash to do this.
#!/bin/bash
#
function gracefulShutdown {
echo "Shutting down!"
# do something..
}
trap gracefulShutdown SIGTERM TERM INT
./subprocess.sh &
tail --pid=${!} -f /dev/null &
wait "${!}"
tail command just waits for subprocess to complete, while wait command waits for the tail to complete... Now, main process is the one which is waiting on.. so any docker signals directly reach the trap we set above...
Example is available at: https://github.com/iamdvr/docker-trap-subprocess

Python: run service after hostapd

I have been working on a project for a few weeks now and I encountered something (probably stupidly simple) I can't figure out!
import os
os.system("service hostapd start && hostapd /etc/hostapd/hostapd.conf")
os.system("service someservicethatIuse start")
When I start hostapd the script pauses because it enables an access point. I tried running it with xfce4-terminal --tab -e "hostapd /etc/hostapd/hostapd.conf" --tab -e "service someservicethatIuser start" but it doesn't seem to work :-/
(Language: Python 2.6)

I don't know about hostapd, but usually it's enough to run service foo start to start a service and it does not block.
Anyways, you could run shell processes in prallel using sh & operator:
import os
os.system("service hostapd start && hostapd /etc/hostapd/hostapd.conf &")
os.system("service someservicethatIuse start")

Use the -B option when you start hostapd, it should run it in the background
$~/hostapd -B /etc/hostapd/hostapd.conf

How to make Python script run as service?

I want to run a python script in a CENTOS server:
#!/usr/bin/env python
import socket
try:
import thread
except ImportError:
import _thread as thread #Py3K changed it.
class Polserv(object):
def __init__(self):
self.numthreads = 0
self.tidcount = 0
self.port = 843
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self.sock.bind(('100.100.100.100', self.port))
self.sock.listen(5)
def run(self):
while True:
thread.start_new_thread(self.handle, self.sock.accept())
def handle(self,conn,addr):
self.numthreads += 1
self.tidcount += 1
tid=self.tidcount
while True:
data=conn.recv(2048)
if not data:
conn.close()
self.numthreads-=1
break
#if "<policy-file-request/>\0" in data:
conn.sendall(b"<?xml version='1.0'?><cross-domain-policy><allow-access-from domain='*' to-ports='*'/></cross-domain-policy>")
conn.close()
self.numthreads-=1
break
#conn.sendall(b"[#%d (%d running)] %s" % (tid,self.numthreads,data) )
Polserv().run()
Im using $ python flashpolicyd.py and it works fine...
The question is: How to keep this script running even after I close the terminal(console)?

I offer two recommendations:
supervisord
1) Install the supervisor package (more verbose instructions here):
sudo apt-get install supervisor
2) Create a config file for your daemon at /etc/supervisor/conf.d/flashpolicyd.conf:
[program:flashpolicyd]
directory=/path/to/project/root
environment=ENV_VARIABLE=example,OTHER_ENV_VARIABLE=example2
command=python flashpolicyd.py
autostart=true
autorestart=true
3) Restart supervisor to load your new .conf
supervisorctl update
supervisorctl restart flashpolicyd
systemd (if currently used by your Linux distro)
[Unit]
Description=My Python daemon
[Service]
Type=simple
ExecStart=/usr/bin/python3 /opt/project/main.py
WorkingDirectory=/opt/project/
Environment=API_KEY=123456789
Environment=API_PASS=password
Restart=always
RestartSec=2
[Install]
WantedBy=sysinit.target
Place this file into /etc/systemd/system/my_daemon.service and enable it using systemctl daemon-reload && systemctl enable my_daemon && systemctl start my_daemon --no-block.
To view logs:
systemctl status my_daemon

I use this code to daemonize my applications. It allows you start/stop/restart the script using the following commands.
python myscript.py start
python myscript.py stop
python myscript.py restart
In addition to this I also have an init.d script for controlling my service. This allows you to automatically start the service when your operating system boots-up.
Here is a simple example to get your going. Simply move your code inside a class, and call it from the run function inside MyDeamon.
import sys
import time
from daemon import Daemon
class YourCode(object):
def run(self):
while True:
time.sleep(1)
class MyDaemon(Daemon):
def run(self):
# Or simply merge your code with MyDaemon.
your_code = YourCode()
your_code.run()
if __name__ == "__main__":
daemon = MyDaemon('/tmp/daemon-example.pid')
if len(sys.argv) == 2:
if 'start' == sys.argv[1]:
daemon.start()
elif 'stop' == sys.argv[1]:
daemon.stop()
elif 'restart' == sys.argv[1]:
daemon.restart()
else:
print "Unknown command"
sys.exit(2)
sys.exit(0)
else:
print "usage: %s start|stop|restart" % sys.argv[0]
sys.exit(2)
Upstart
If you are running an operating system that is using Upstart (e.g. CentOS 6) - you can also use Upstart to manage the service. If you use Upstart you can keep your script as is, and simply add something like this under /etc/init/my-service.conf
start on started sshd
stop on runlevel [!2345]
exec /usr/bin/python /opt/my_service.py
respawn
You can then use start/stop/restart to manage your service.
e.g.
start my-service
stop my-service
restart my-service
A more detailed example of working with upstart is available here.
Systemd
If you are running an operating system that uses Systemd (e.g. CentOS 7) you can take a look at the following Stackoverflow answer.

My non pythonic approach would be using & suffix. That is:
python flashpolicyd.py &
To stop the script
killall flashpolicyd.py
also piping & suffix with disown would put the process under superparent (upper):
python flashpolicyd.pi & disown

first import os module in your app than with use from getpid function get pid's app and save in a file.for example :
import os
pid = os.getpid()
op = open("/var/us.pid","w")
op.write("%s" % pid)
op.close()
and create a bash file in /etc/init.d path:
/etc/init.d/servername
PATHAPP="/etc/bin/userscript.py &"
PIDAPP="/var/us.pid"
case $1 in
start)
echo "starting"
$(python $PATHAPP)
;;
stop)
echo "stoping"
PID=$(cat $PIDAPP)
kill $PID
;;
esac
now , u can start and stop ur app with down command:
service servername stop
service servername start
or
/etc/init.d/servername stop
/etc/init.d/servername start

for my script of python, I use...
To START python script :
start-stop-daemon --start --background --pidfile $PIDFILE --make-pidfile --exec $DAEMON
To STOP python script :
PID=$(cat $PIDFILE)
kill -9 $PID
rm -f $PIDFILE
P.S.: sorry for poor English, I'm from CHILE :D

Run python script as daemon at boot time (Ubuntu)

I've created small web server using werkzeug and I'm able to run it in usual python way with python my_server.py. Pages load, everything works fine. Now I want to start it when my pc boots. What's the easiest way to do that? I've been struggling with upstart but it doesn't seem to "live in a background" cuz after I execute start my_server I immediately receive kernel: [ 8799.793942] init: my_server main process (7274) terminated with status 1
my_server.py:
...
if __name__ == '__main__':
from werkzeug.serving import run_simple
app = create_app()
run_simple('0.0.0.0', 4000, app)
upstart configuration file my_server.conf:
description "My service"
author "Some Dude <blah#foo.com>"
start on runlevel [2345]
stop on runlevel [016]
exec /path/to/my_server.py
start on startup
Any Ideas how to make it work? Or any other better way to daemonize the script?
Update:
I believe the problem lies within my_server.py. It doesn't seem to initiate the webserver (method run_simple()) in the first place. What steps should be taken to make .py file be run by task handler such as upstart?
Place shebang as first line #!/usr/bin/env python
Allow execution permissions chmod 755
Start the daemon with superuser rights (to be absolutely sure no permission restrictions prevents it from starting)
Make sure all python libraries are there!
Something else?
Solved:
The problem was with missing python dependencies. When starting the script through task manager (e.g. upstart or start-stop-daemon) no errors are thrown. Need to be absolutely sure that pythonpath contains everything you need.

In addition to gg.kaspersky method, you could also turn your script into a "service", so that you can start or stop it using:
$ sudo service myserver start
* Starting system myserver.py Daemon [ OK ]
$ sudo service myserver status
* /path/to/myserver.py is running
$ sudo service myserver stop
* Stopping system myserver.py Daemon [ OK ]
and define it as a startup service using:
$ sudo update-rc.d myserver defaults
To do this, you must create this file and save it in /etc/init.d/.
#!/bin/sh -e
DAEMON="/path/to/myserver.py"
DAEMONUSER="myuser"
DAEMON_NAME="myserver.py"
PATH="/sbin:/bin:/usr/sbin:/usr/bin"
test -x $DAEMON || exit 0
. /lib/lsb/init-functions
d_start () {
log_daemon_msg "Starting system $DAEMON_NAME Daemon"
start-stop-daemon --background --name $DAEMON_NAME --start --user $DAEMONUSER --exec $DAEMON
log_end_msg $?
}
d_stop () {
log_daemon_msg "Stopping system $DAEMON_NAME Daemon"
start-stop-daemon --name $DAEMON_NAME --stop --retry 5 --name $DAEMON_NAME
log_end_msg $?
}
case "$1" in
start|stop)
d_${1}
;;
restart|reload|force-reload)
d_stop
d_start
;;
force-stop)
d_stop
killall -q $DAEMON_NAME || true
sleep 2
killall -q -9 $DAEMON_NAME || true
;;
status)
status_of_proc "$DAEMON_NAME" "$DAEMON" "system-wide $DAEMON_NAME" && exit 0 || exit $?
;;
*)
echo "Usage: /etc/init.d/$DAEMON_NAME {start|stop|force-stop|restart|reload|force-reload|status}"
exit 1
;;
esac
exit 0
In this example, I assume you have a shebang like #!/usr/bin/python at the head of your python file, so that you can execute it directly.
Last but not least, do not forget to give execution rights to your python server and to the service script :
$ sudo chmod 755 /etc/init.d/myserver
$ sudo chmod 755 /path/to/mserver.py
Here's the page where I learned this originally (french).
Cheers.

One simple way to do is using crontab:
$ crontab -e
A crontab file will appear for editing, write the line at the end:
#reboot python myserver.py
and quit. Now, after each reboot, the cron daemon will run your myserver python script.

If you have supervisor service that starts at boot, write a supervisor service is much, much simpler.
You can even set autorestart if your program fails.

Python Service / Daemon

I'm writing a Python script that I need to have start at boot (it should continue running forever once it is started). I would like to be able to manage the service via running a command like:
sudo service my-service (and either start, stop, restart, etc.)
I have been reading a lot into it and would really like to finish my project. I saw this but would need help implementing it if it were to work Linux Start-up Script. I'm not even sure where to start, I don't really know how to program in bash but I'm open to all solutions. Thank you ahead of time, and I appreciate all responses!

Take a look at zdaemon. It provides a simple way to daemonize a Python process.
Then you can write an init.d script for it - based on your OS.
Or you can use a tool like Upstart, supervisord to control the daemon.
My init.d script (on centos 5.8) looks like this:
. /etc/rc.d/init.d/functions
. /etc/sysconfig/network
APP_PATH=/path/to/your/app
PYTHON=/usr/local/bin/python
USER=user
start() {
cd $APP_PATH
zdaemon -C app.zdconf start
}
stop() {
cd $APP_PATH
zdaemon -C app.zdconf stop
}
check_status() {
cd $APP_PATH
zdaemon -C app.zdconf status
}
case "$1" in
start)
start
;;
stop)
stop
;;
status)
check_status
;;
restart)
stop
start
;;
*)
esac
exit 0
Where app.zdconf is the zdaemon configuration file.