I want to run a python script in a CENTOS server:
#!/usr/bin/env python
import socket
try:
import thread
except ImportError:
import _thread as thread #Py3K changed it.
class Polserv(object):
def __init__(self):
self.numthreads = 0
self.tidcount = 0
self.port = 843
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self.sock.bind(('100.100.100.100', self.port))
self.sock.listen(5)
def run(self):
while True:
thread.start_new_thread(self.handle, self.sock.accept())
def handle(self,conn,addr):
self.numthreads += 1
self.tidcount += 1
tid=self.tidcount
while True:
data=conn.recv(2048)
if not data:
conn.close()
self.numthreads-=1
break
#if "<policy-file-request/>\0" in data:
conn.sendall(b"<?xml version='1.0'?><cross-domain-policy><allow-access-from domain='*' to-ports='*'/></cross-domain-policy>")
conn.close()
self.numthreads-=1
break
#conn.sendall(b"[#%d (%d running)] %s" % (tid,self.numthreads,data) )
Polserv().run()
Im using $ python flashpolicyd.py and it works fine...
The question is: How to keep this script running even after I close the terminal(console)?
I offer two recommendations:
supervisord
1) Install the supervisor package (more verbose instructions here):
sudo apt-get install supervisor
2) Create a config file for your daemon at /etc/supervisor/conf.d/flashpolicyd.conf:
[program:flashpolicyd]
directory=/path/to/project/root
environment=ENV_VARIABLE=example,OTHER_ENV_VARIABLE=example2
command=python flashpolicyd.py
autostart=true
autorestart=true
3) Restart supervisor to load your new .conf
supervisorctl update
supervisorctl restart flashpolicyd
systemd (if currently used by your Linux distro)
[Unit]
Description=My Python daemon
[Service]
Type=simple
ExecStart=/usr/bin/python3 /opt/project/main.py
WorkingDirectory=/opt/project/
Environment=API_KEY=123456789
Environment=API_PASS=password
Restart=always
RestartSec=2
[Install]
WantedBy=sysinit.target
Place this file into /etc/systemd/system/my_daemon.service and enable it using systemctl daemon-reload && systemctl enable my_daemon && systemctl start my_daemon --no-block.
To view logs:
systemctl status my_daemon
I use this code to daemonize my applications. It allows you start/stop/restart the script using the following commands.
python myscript.py start
python myscript.py stop
python myscript.py restart
In addition to this I also have an init.d script for controlling my service. This allows you to automatically start the service when your operating system boots-up.
Here is a simple example to get your going. Simply move your code inside a class, and call it from the run function inside MyDeamon.
import sys
import time
from daemon import Daemon
class YourCode(object):
def run(self):
while True:
time.sleep(1)
class MyDaemon(Daemon):
def run(self):
# Or simply merge your code with MyDaemon.
your_code = YourCode()
your_code.run()
if __name__ == "__main__":
daemon = MyDaemon('/tmp/daemon-example.pid')
if len(sys.argv) == 2:
if 'start' == sys.argv[1]:
daemon.start()
elif 'stop' == sys.argv[1]:
daemon.stop()
elif 'restart' == sys.argv[1]:
daemon.restart()
else:
print "Unknown command"
sys.exit(2)
sys.exit(0)
else:
print "usage: %s start|stop|restart" % sys.argv[0]
sys.exit(2)
Upstart
If you are running an operating system that is using Upstart (e.g. CentOS 6) - you can also use Upstart to manage the service. If you use Upstart you can keep your script as is, and simply add something like this under /etc/init/my-service.conf
start on started sshd
stop on runlevel [!2345]
exec /usr/bin/python /opt/my_service.py
respawn
You can then use start/stop/restart to manage your service.
e.g.
start my-service
stop my-service
restart my-service
A more detailed example of working with upstart is available here.
Systemd
If you are running an operating system that uses Systemd (e.g. CentOS 7) you can take a look at the following Stackoverflow answer.
My non pythonic approach would be using & suffix. That is:
python flashpolicyd.py &
To stop the script
killall flashpolicyd.py
also piping & suffix with disown would put the process under superparent (upper):
python flashpolicyd.pi & disown
first import os module in your app than with use from getpid function get pid's app and save in a file.for example :
import os
pid = os.getpid()
op = open("/var/us.pid","w")
op.write("%s" % pid)
op.close()
and create a bash file in /etc/init.d path:
/etc/init.d/servername
PATHAPP="/etc/bin/userscript.py &"
PIDAPP="/var/us.pid"
case $1 in
start)
echo "starting"
$(python $PATHAPP)
;;
stop)
echo "stoping"
PID=$(cat $PIDAPP)
kill $PID
;;
esac
now , u can start and stop ur app with down command:
service servername stop
service servername start
or
/etc/init.d/servername stop
/etc/init.d/servername start
for my script of python, I use...
To START python script :
start-stop-daemon --start --background --pidfile $PIDFILE --make-pidfile --exec $DAEMON
To STOP python script :
PID=$(cat $PIDFILE)
kill -9 $PID
rm -f $PIDFILE
P.S.: sorry for poor English, I'm from CHILE :D
Related
I made a simple python webserver based on this example, which runs inside Docker
FROM python:3-alpine
WORKDIR /app
COPY entrypoint.sh .
RUN chmod +x entrypoint.sh
COPY src src
CMD ["python", "/app/src/api.py"]
ENTRYPOINT ["/app/entrypoint.sh"]
Entrypoint:
#!/bin/sh
echo starting entrypoint
set -x
exec "$#"
Stopping the container took very long, altough the exec statement with the JSON array syntax should pass it to the python process. I assumed a problem with SIGTERM no being passed to the container. I added the following to my api.pyscript to detect SIGTERM
def terminate(signal,frame):
print("TERMINATING")
if __name__ == "__main__":
signal.signal(signal.SIGTERM, terminate)
webServer = HTTPServer((hostName, serverPort), MyServer)
print("Server started http://%s:%s" % (hostName, serverPort))
webServer.serve_forever()
Executed without Docker python3 api/src/api.py, I tried
kill -15 $(ps -guaxf | grep python | grep -v grep | awk '{print $2}')
to send SIGTERM (15 is the number code of it). The script prints TERMINATING, so my event handler works. Now I run the Docker container using docker-compose and press CTRL + C. Docker says gracefully stopping... (press Ctrl+C again to force) but doesn't print my terminating message from the event handler.
I also tried to run docker-compose in detached mode, then run docker-compose kill -s SIGTERM api and view the logs. Still no message from the event handler.
Since the script runs as pid 1 as desired and setting init: true in docker-compose.yml doesn't seem to change anything, I took a deeper drive in this topic. This leads me figuring out multiple mistakes I did:
Logging
The approach of printing a message when SIGTERM is catched was designed as simple test case to see if this basically works before I care about stopping the server. But I noticed that no message appears for two reasons:
Output buffering
When running a long term process in python like the HTTP server (or any while True loop for example), there is no output displayed when starting the container attached with docker-compose up (no -d flag). To receive live logs, we need to start python with the -u flag or set the env variable PYTHONUNBUFFERED=TRUE.
No log piping after stop
But the main problem was not the output buffering (this is just a notice since I wonder why there was no log output from the container). When canceling the container, docker-compose stops piping logs to the console. This means that from a logical perspective it can't display anything that happens AFTER CTRL + C is pressed.
To fetch those logs, we need to wait until docker-compose has stopped the container and run docker-compose logs. It will print all, including those generated after CTRL + C is pressed. Using docker-compose logs I found out that SIGTERM is passed to the container and my event handler works.
Stopping the webserver
With those knowledge I tried to stop the webserver instance. First this doesn't work because it's not enough to just call webServer.server_close(). Its required to exit explicitely after any cleanup work is done like this:
def terminate(signal,frame):
print("Start Terminating: %s" % datetime.now())
webServer.server_close()
sys.exit(0)
When sys.exit() is not called, the process keeps running which results in ~10s waiting time before Docker kills it.
Full working example
Here a demo script that implement everything I've learned:
from http.server import BaseHTTPRequestHandler, HTTPServer
import signal
from datetime import datetime
import sys, os
hostName = "0.0.0.0"
serverPort = 80
class MyServer(BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(200)
self.send_header("Content-Type", "text/html")
self.end_headers()
self.wfile.write(bytes("Hello from Python Webserver", "utf-8"))
webServer = None
def terminate(signal,frame):
print("Start Terminating: %s" % datetime.now())
webServer.server_close()
sys.exit(0)
if __name__ == "__main__":
signal.signal(signal.SIGTERM, terminate)
webServer = HTTPServer((hostName, serverPort), MyServer)
print("Server started http://%s:%s with pid %i" % ("0.0.0.0", 80, os.getpid()))
webServer.serve_forever()
Running in a container, it could be stopped very fast without waiting for Docker to kill the process:
$ docker-compose up --build -d
$ time docker-compose down
Stopping python-test_app_1 ... done
Removing python-test_app_1 ... done
Removing network python-test_default
real 0m1,063s
user 0m0,424s
sys 0m0,077s
Docker runs your application, per default, in foreground, so, as PID 1, this said, the process with the PID 1 as a special meaning and specific protections in Linux.
This is highlighted in docker run documentation:
Note
A process running as PID 1 inside a container is treated specially by Linux: it ignores any signal with the default action. As a result, the process will not terminate on SIGINT or SIGTERM unless it is coded to do so.
Source: https://docs.docker.com/engine/reference/run/#foreground
In order to fix this, you can run the container, in a single container mode, with the flag --init of docker run:
You can use the --init flag to indicate that an init process should be used as the PID 1 in the container. Specifying an init process ensures the usual responsibilities of an init system, such as reaping zombie processes, are performed inside the created container.
Source: https://docs.docker.com/engine/reference/run/#specify-an-init-process
The same configuration is possible in docker-compose, simply by specifying init: true on the container.
An example would be:
version: "3.8"
services:
web:
image: alpine:latest
init: true
Source: https://docs.docker.com/compose/compose-file/#init
I am running a script that works on sockets.. It requires sudo to run..
however,
Inside the script i call another script that requires not to be run as sudo
here is the code:
import subprocess
import socket
s=socket.socket(socket.AF_INET, socket.SOCK_STREAM)
#s.settimeout(5.0)
host='192.168.1.148'
port=1022
s.bind((host, port))
s.listen(5)
while True:
c, addr = s.accept()
subprocess.call("python bluetooth2.py",shell=True)
print 'got connection from',addr
c.send('Thank you for connecting')
#c.settimeout(5.0)
c.recv(1022)
c.close()
bluetooth2.py runs pulseaudio which is run as root for some reason and doesn't work. any help greatly appreciated!
Here is what the bluetooth2.py script looks like for reference (the one that is calling on pulseaudio)
import time
import pexpect
from sh import bluetoothctl
import subprocess
mac = "C8:84:47:26:E6:3C"
print ("stuck here")
#bluetoothctl("connect", mac)
def connect():
child = pexpect.spawn('bluetoothctl')
child.sendline('power on')
child.sendline('agent on')
child.sendline('default-agent')
child.sendline('pair C8:84:47:26:E6:3C')
time.sleep(1)
child.sendline('trust C8:84:47:26:E6:3C')
time.sleep(1)
child.sendline('connect C8:84:47:26:E6:3C')
print("connecting...")
time.sleep(5)
subprocess.call("pulseaudio --start",shell=True)
subprocess.call("pacmd set-default-sink
bluez_sink.C8_84_47_26_E6_3C",shell=True)
subprocess.call("aplay /home/pi/bleep_01.wav", shell=True)
Solution run PulseAudio for all your users
Add bellow lines into /etc/systemd/system/pulseaudio.service file and save
[Unit]
Description=PulseAudio system server
[Service]
Type=notify
ExecStart=pulseaudio --daemonize=no --system --realtime --log-target=journal
[Install]
WantedBy=multi-user.target
Enable service
sudo systemctl --system enable pulseaudio.service
sudo systemctl --system start pulseaudio.service
sudo systemctl --system status pulseaudio.service
Edit Client conf /etc/pulse/client.conf and replace ass bellow
default-server = /var/run/pulse/native
autospawn = no
Add root to pulse group
sudo adduser root pulse-access
And finally reboot the system
I have been working on a project for a few weeks now and I encountered something (probably stupidly simple) I can't figure out!
import os
os.system("service hostapd start && hostapd /etc/hostapd/hostapd.conf")
os.system("service someservicethatIuse start")
When I start hostapd the script pauses because it enables an access point. I tried running it with xfce4-terminal --tab -e "hostapd /etc/hostapd/hostapd.conf" --tab -e "service someservicethatIuser start" but it doesn't seem to work :-/
(Language: Python 2.6)
I don't know about hostapd, but usually it's enough to run service foo start to start a service and it does not block.
Anyways, you could run shell processes in prallel using sh & operator:
import os
os.system("service hostapd start && hostapd /etc/hostapd/hostapd.conf &")
os.system("service someservicethatIuse start")
Use the -B option when you start hostapd, it should run it in the background
$~/hostapd -B /etc/hostapd/hostapd.conf
I've created small web server using werkzeug and I'm able to run it in usual python way with python my_server.py. Pages load, everything works fine. Now I want to start it when my pc boots. What's the easiest way to do that? I've been struggling with upstart but it doesn't seem to "live in a background" cuz after I execute start my_server I immediately receive kernel: [ 8799.793942] init: my_server main process (7274) terminated with status 1
my_server.py:
...
if __name__ == '__main__':
from werkzeug.serving import run_simple
app = create_app()
run_simple('0.0.0.0', 4000, app)
upstart configuration file my_server.conf:
description "My service"
author "Some Dude <blah#foo.com>"
start on runlevel [2345]
stop on runlevel [016]
exec /path/to/my_server.py
start on startup
Any Ideas how to make it work? Or any other better way to daemonize the script?
Update:
I believe the problem lies within my_server.py. It doesn't seem to initiate the webserver (method run_simple()) in the first place. What steps should be taken to make .py file be run by task handler such as upstart?
Place shebang as first line #!/usr/bin/env python
Allow execution permissions chmod 755
Start the daemon with superuser rights (to be absolutely sure no permission restrictions prevents it from starting)
Make sure all python libraries are there!
Something else?
Solved:
The problem was with missing python dependencies. When starting the script through task manager (e.g. upstart or start-stop-daemon) no errors are thrown. Need to be absolutely sure that pythonpath contains everything you need.
In addition to gg.kaspersky method, you could also turn your script into a "service", so that you can start or stop it using:
$ sudo service myserver start
* Starting system myserver.py Daemon [ OK ]
$ sudo service myserver status
* /path/to/myserver.py is running
$ sudo service myserver stop
* Stopping system myserver.py Daemon [ OK ]
and define it as a startup service using:
$ sudo update-rc.d myserver defaults
To do this, you must create this file and save it in /etc/init.d/.
#!/bin/sh -e
DAEMON="/path/to/myserver.py"
DAEMONUSER="myuser"
DAEMON_NAME="myserver.py"
PATH="/sbin:/bin:/usr/sbin:/usr/bin"
test -x $DAEMON || exit 0
. /lib/lsb/init-functions
d_start () {
log_daemon_msg "Starting system $DAEMON_NAME Daemon"
start-stop-daemon --background --name $DAEMON_NAME --start --user $DAEMONUSER --exec $DAEMON
log_end_msg $?
}
d_stop () {
log_daemon_msg "Stopping system $DAEMON_NAME Daemon"
start-stop-daemon --name $DAEMON_NAME --stop --retry 5 --name $DAEMON_NAME
log_end_msg $?
}
case "$1" in
start|stop)
d_${1}
;;
restart|reload|force-reload)
d_stop
d_start
;;
force-stop)
d_stop
killall -q $DAEMON_NAME || true
sleep 2
killall -q -9 $DAEMON_NAME || true
;;
status)
status_of_proc "$DAEMON_NAME" "$DAEMON" "system-wide $DAEMON_NAME" && exit 0 || exit $?
;;
*)
echo "Usage: /etc/init.d/$DAEMON_NAME {start|stop|force-stop|restart|reload|force-reload|status}"
exit 1
;;
esac
exit 0
In this example, I assume you have a shebang like #!/usr/bin/python at the head of your python file, so that you can execute it directly.
Last but not least, do not forget to give execution rights to your python server and to the service script :
$ sudo chmod 755 /etc/init.d/myserver
$ sudo chmod 755 /path/to/mserver.py
Here's the page where I learned this originally (french).
Cheers.
One simple way to do is using crontab:
$ crontab -e
A crontab file will appear for editing, write the line at the end:
#reboot python myserver.py
and quit. Now, after each reboot, the cron daemon will run your myserver python script.
If you have supervisor service that starts at boot, write a supervisor service is much, much simpler.
You can even set autorestart if your program fails.
Hallo,
I'm trying to let a python script run as service (daemon) on (ubuntu) linux.
On the web there exist several solutions like:
http://pypi.python.org/pypi/python-daemon/
A well-behaved Unix daemon process is tricky to get right, but the required steps are much the same for every daemon program. A DaemonContext instance holds the behaviour and configured process environment for the program; use the instance as a context manager to enter a daemon state.
http://www.jejik.com/articles/2007/02/a_simple_unix_linux_daemon_in_python/
However as I want to integrate my python script specifically with ubuntu linux my solution is a combination with an init.d script
#!/bin/bash
WORK_DIR="/var/lib/foo"
DAEMON="/usr/bin/python"
ARGS="/opt/foo/linux_service.py"
PIDFILE="/var/run/foo.pid"
USER="foo"
case "$1" in
start)
echo "Starting server"
mkdir -p "$WORK_DIR"
/sbin/start-stop-daemon --start --pidfile $PIDFILE \
--user $USER --group $USER \
-b --make-pidfile \
--chuid $USER \
--exec $DAEMON $ARGS
;;
stop)
echo "Stopping server"
/sbin/start-stop-daemon --stop --pidfile $PIDFILE --verbose
;;
*)
echo "Usage: /etc/init.d/$USER {start|stop}"
exit 1
;;
esac
exit 0
and in python:
import signal
import time
import multiprocessing
stop_event = multiprocessing.Event()
def stop(signum, frame):
stop_event.set()
signal.signal(signal.SIGTERM, stop)
if __name__ == '__main__':
while not stop_event.is_set():
time.sleep(3)
My question now is if this approach is correct. Do I have to handle any additional signals? Will it be a "well-behaved Unix daemon process"?
Assuming your daemon has some way of continually running (some event loop, twisted, whatever), you can try to use upstart.
Here's an example upstart config for a hypothetical Python service:
description "My service"
author "Some Dude <blah#foo.com>"
start on runlevel [234]
stop on runlevel [0156]
chdir /some/dir
exec /some/dir/script.py
respawn
If you save this as script.conf to /etc/init you simple do a one-time
$ sudo initctl reload-configuration
$ sudo start script
You can stop it with stop script. What the above upstart conf says is to start this service on reboots and also restart it if it dies.
As for signal handling - your process should naturally respond to SIGTERM. By default this should be handled unless you've specifically installed your own signal handler.
Rloton's answer is good. Here is a light refinement, just because I spent a ton of time debugging. And I need to do a new answer so I can format properly.
A couple other points that took me forever to debug:
When it fails, first check /var/log/upstart/.log
If your script implements a daemon with python-daemon, you do NOT use the 'expect daemon' stanza. Having no 'expect' works. I don't know why. (If anyone knows why - please post!)
Also, keep checking "initctl status script" to make sure you are up (start/running). (and do a reload when you update your conf file)
Here is my version:
description "My service"
author "Some Dude <blah#foo.com>"
env PYTHON_HOME=/<pathtovirtualenv>
env PATH=$PYTHON_HOME:$PATH
start on runlevel [2345]
stop on runlevel [016]
chdir <directory>
# NO expect stanza if your script uses python-daemon
exec $PYTHON_HOME/bin/python script.py
# Only turn on respawn after you've debugged getting it to start and stop properly
respawn