We Keep Coding

How can I keep my python-daemon process running or restart it on fail?

I have a python3.9 script I want to have running 24/7. In it, I use python-daemon to keep it running like so:
import daemon
with daemon.DaemonContext():
%%script%%
And it works fine but after a few hours or days, it just crashes randomly. I always start it with sudo but I can't seem to figure out where to find the log file of the daemon process for debugging. What can I do to ensure logging? How can I keep the script running or auto-restart it after crashing?
You can find the full code here.

If you really want to run a script 24/7 in background, the cleanest and easiest way to do it would surely be to create a systemd service.
There are already many descriptions of how to do that, for example here.
One of the advantages of systemd, in addition to being able to launch a service at startup, is to be able to restart it after failure.
Restart=on-failure
If all you want to do is automatically restart the program after a crash, the easiest method would probably be to use a bash script.
You can use the until loop, which is used to execute a given set of commands as long as the given condition evaluates to false.
#!/bin/bash
until python /path/to/script.py; do
echo "The program crashed at `date +%H:%M:%S`. Restarting the script..."
done
If the command returns a non zero exit-status, then the script is restarted.

I would start with familiarizing myself with those two questions:
How to make a Python script run like a service or daemon in Linux
Run a python script with supervisor
Looks like you need a supervisor that will make sure that your script/daemon is still running. You can take a look at supervisord.

Python Script that was called from bash startup script stops after 20mins of running,

I have a python script that has a While True: in it that I would like to have run on startup on a raspberry pi running Jessie.
So far I have a startup bash script in /etc/init.d called startup.sh which contains
sudo python3 /home/pi/Desktop/Scripts/bluez3.py &
When the raspberry pi starts up, the script does run but after 20 minutes the script seems to stop. I have logging in my script and the time-stamp stops exactly 20 mins in.
I did some reading and I think the best option would be to create the python script as a service on the raspberry pi. However, I have not been able to find a decent tutorial about how to do this (and my lack of python knowledge).
My question is, is there another way to resolve my problem or does anyone know of a good tutorial on how to make the python script into a service.
Thanks!

given the name of your script, I'm guessing it's related to some bluetooth stuff. It's likely that after 20 min, whatever you're checking/needing in your script gets unaccessible and throws an exception or something like that. Like a resource being locked, or a bt device being disconnected or a module being unloaded or unavailable or [insert edge case reason here]…
that being said, in between creating a systemd service, you can first play with supervisorctl which is just an apt install supervisor away.
then if you really want to launch it as a service, you can find plenty of examples in /lib/systemd/system/*.service, like the following:
[Unit]
Description=Your service
Wants=
After=bluetooth.target # I guess you need bluetooth initialised first
[Service]
ExecStart=/usr/bin/python3 /home/pi/Desktop/Scripts/bluez3.py
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=always
[Install]
WantedBy=multi-user.target
which I customized from the sshd.service file 😉

Run python script with supervisord

I have a simple python script (discord bot) and it works well when I run it with command python3 discord_bot.py & or sh start_bot.sh.
But how can I run it with supervisord?
Update:
I have installed supervisord. But when I try to run process, I have error:
exit status 0; not expected
My supervisord config:
[program:AFI]
command=/home/maksymov/www/Bots/discord_bots/afi/start_bot.sh
autostart=true
autorestart=true
stderr_logfile=/var/log/afi.err.log
stdout_logfile=/var/log/afi.out.log
user=www-data

Probably you need to use one of the "supervisors". Like system.d or ramona
The first one is more general. The second is more "python-specific"

I guess your program tries to run as a daemon. I pasted the most relevant part from documentation:
Supervisord subprocess
Programs meant to be run under supervisor should not daemonize themselves. Instead, they should run in the foreground. They should not detach from the terminal from which they are started.
The easiest way to tell if a program will run in the foreground is to run the command that invokes the program from a shell prompt. If it gives you control of the terminal back, but continues running, it’s daemonizing itself and that will almost certainly be the wrong way to run it under supervisor.

How do I write a python program to get my Linux machine to restart at a signal?

I have one main program "main.py" that may freeze occasionally. Whenever I detect this happens, I want to have a separate program "watch.py" get my Linux machine to restart. These scripts start at bootup automatically since I edited /etc/rc.local.
Right now /etc/rc.local looks like this -
python watch.py &
python main.py &
This should let both programs run simultaneously. When I notice main.py has frozen, I'll give a signal manually to watch.py (using a remote TCP connection) to restart. What should my python code or shell script be to actually restart the system when "watch.py" receives a signal?
Possible answers could be writing some python code to restart a Linux machine, writing some python code to exit with a certain argument, and upon noticing the argument, execute "sudo reboot".

Run Python script forever, logging errors and restarting when crashes

I have a python script that continuously process new data and writes to a mongodb. In the script, its a while loop and a sleep that runs the code continuously.
What is the recommended way to run the Python script forever, logging errors when they occur, and restarting when it crashes?
Will node.js's forever be suitable? I'm also running node/meteor on the same Ubuntu server.

supervisord is perfect for this sort of thing. While I used to check that programs were still running every couple of minutes with a cron job, supervisord runs all programs in an in-process thread, so in the event your program terminates, supervisord will automatically restart the process. I no longer need to parse the output of ps to see if a program crashed.
It has a simple declaritive config file and configurable logging. By default it creates a log file for your-program-name-stderr.log your-program-name-stdout.log which are automatically handled by logrotate when supervisord is installed from an OS package manager (Debian for me).
If you don't want to configure supervisord's logging, you should look at logging in python so you can control what goes into those files.
if you're on a debian derivative you should be able to install and start the daemon simply by executing apt-get install supervisord as root.
The config file is very straightforward too:
[program:myprogram]
command=/path/to/my/program/script
directory=/path/to/my/program/base
user=myuser
autostart=true
autorestart=true
redirect_stderr=True
supervisorctl also allows you to see what your program is doing interactively and can start and stop multiple programs with supervisorctl start myprogram etc

Recently wrote something similar. The basic pattern I follow is
while True:
try:
#functionality
except SpecificError:
#log exception
except: #catch everything else
finally:
time.sleep(600)
to handle reboots you can use init.d or cron jobs.

If you are writing a daemon, you should probably do it with this command:
http://manpages.ubuntu.com/manpages/lucid/man8/start-stop-daemon.8.html
You can spawn this from a System V /etc/init.d/ script, or use Upstart which is slowly replacing it.
Upstart: http://upstart.ubuntu.com/getting-started.html
System V: http://www.cyberciti.biz/tips/linux-write-sys-v-init-script-to-start-stop-service.html
I find System V easier to write, but if this will ever be packaged and distributed in a debian file, I recommend writing an Upstart conf.
Definitely keep the sleep so it won't keep a grip on CPU load.

I don't know if this is still relevant to you, but I have been reading forever about how to do this and want to share somewhere what I did.
For me, the goal was to have a python script running always (on my Linux computer). The python script also has a "while True " loop in it which should theoretically run forever, but if it for any reason I cannot think of would crash, I want the script to restart. Also, when I restart the computer it should run the script.
I am not an expert but for me the best and most understandable was to use systemd (assuming you use Linux).
There are two nice examples of how to do this given here and here, showing how to write your .service files in either /etc/systemd/system or /lib/systemd/system. If you want to be completely correct you should take the former:
" /etc/systemd/system/: units installed by the system administrator" 1
The documentation of systemd here is actually nice to read, even if you are not an expert.
Hope this helps someone!