We Keep Coding

How to Properly Exit Airflow Standalone?

I am running airflow standalone as a local development environment. I followed the instructions provided by Airflow to setup the environment, but now I'd like to shut it down in the most graceful way possible.
I ran the standalone command in a terminal, and so my first attempt was to simply use Ctrl+C. It looks promising:
triggerer | [2022-02-02 10:44:06,771] {triggerer_job.py:251} INFO - 0 triggers currently running
^Cstandalone | Shutting down components
However, even 10 minutes later, the shutdown is still in progress, with no more messages in the terminal. I used Ctrl+C again and got a KeyboardInterrupt. Did I do this the wrong way? Is there a better way to shut down the standalone environment?

You could try the following (in bash):
pkill --signal 2 -u $USER airflow
or
pkill --signal 15 -u $USER airflow
or
pkill --signal 9 -u $USER airflow
Say what?
Here's more description of each part:
pkill - Process kill function.
--signal - Tells what 'signal' to send to the process
2 | 15 | 9 - Is the id for the terminal signal to send.
2 = SIGINT, which is like CTRL + C.
15 = SIGTERM, the default for pkill.
9 = SIGKILL, which doesn't messaround with gracefully ending a process.
For more info, run kill -L in your bash terminal.
-u - Tells the functon to only match processes whose real user ID is listed.
$USER - The current session user environment variable. This may be different on your system, so adjust accordingly.
airflow - The name of the selection criteria or pattern to match.
prep info page for more detail on the options available.

Run a python script with supervisor

I copied from here to run my Python code as a daemon.
For extra uptime. I thought it would be a better Idea to use supervisor to keep this daemon running.
I did this.
python_deamon.conf
[program:python_deamon]
directory=/usr/local/python_deamon/
command=/usr/local/python_venv/bin/python daemon_runnner.py start
stderr_logfile=/var/log/gunicorn.log
stdout_logfile=/var/log/gunicorn.log
autostart=true
autorestart=true
The problem is that though supervisor successfully starts the python_daemon it keeps retrying.
2015-09-23 16:10:45,592 CRIT Supervisor running as root (no user in config file)
2015-09-23 16:10:45,592 WARN Included extra file "/etc/supervisor/conf.d/python_daemon.conf" during parsing
2015-09-23 16:10:45,592 INFO RPC interface 'supervisor' initialized
2015-09-23 16:10:45,592 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2015-09-23 16:10:45,592 INFO supervisord started with pid 13880
2015-09-23 16:10:46,595 INFO spawned: 'python_deamon' with pid 17884
2015-09-23 16:10:46,611 INFO exited: python_deamon (exit status 1; not expected)
2015-09-23 16:10:47,614 INFO spawned: 'python_deamon' with pid 17885
2015-09-23 16:10:47,630 INFO exited: python_deamon (exit status 1; not expected)
2015-09-23 16:10:49,635 INFO spawned: 'python_deamon' with pid 17888
2015-09-23 16:10:49,656 INFO exited: python_deamon (exit status 1; not expected)
2015-09-23 16:10:52,662 INFO spawned: 'python_deamon' with pid 17891
2015-09-23 16:10:52,680 INFO exited: python_deamon (exit status 1; not expected)
2015-09-23 16:10:53,681 INFO gave up: python_deamon entered FATAL state, too many start retries too quickly
Just for the record the after overriding run() method I never return anything.
Is it possible to do what I am trying to do or am I being dumb ?
P.S: I know that the root cause of the whole problem is that since run() never return anything supervisor keeps trying to start it and hence thinks that the process failed and gives the status as FATAL Exited too quickly (process log may have details). My actual question is am I doing it right ? or can this be done this way ?
P.P.S: Stand alone(without supervisor) daemon_runnner.py runs fine with and without sudo permissions.

try to set startsecs = 0:
[program:foo]
command = ls
startsecs = 0
autorestart = false
http://supervisord.org/configuration.html
startsecs
The total number of seconds which the program needs to stay running after a startup to consider the start successful. If the program does not stay up for this many seconds after it has started, even if it exits with an “expected” exit code (see exitcodes), the startup will be considered a failure. Set to 0 to indicate that the program needn’t stay running for any particular amount of time.

This is what I normally do:
Create a service.conf file which describes the new Python script. This script references the shell script which is the one in reality launching the Python script. This .conf file lives inside /etc/supervisor/conf.d/
Create a shell script which launches the Python script. Change permissions to executable. chmod 755 service.sh. In this script we actually launch the Python script.
Configure log_stderr and stderr_logfile to verify issue.
Update supervisor using reload and then check status:
supervisor> status
alexad RUNNING pid 32657, uptime 0:21:05
service.conf
[program:alexad]
; Set full path to celery program if using virtualenv
command=sh /usr/local/src/gonzo/supervisorctl/alexad.sh
directory=/usr/local/src/gonzo/services/alexa
log_stdout=true ; if true, log program stdout (default true)
log_stderr=true ; if true, log program stderr (default false)
stderr_logfile=/usr/local/src/gonzo/log/alexad.err
logfile=/usr/local/src/gonzo/log/alexad.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; Set Celery priority higher than default (999)
priority=500
service.sh
#!/bin/bash
cd /usr/local/src/gonzo/services/alexa
exec python reader.py

You should check your supervisor program logs which live here normally /var/log/supervisor/
In my case, I had a ModuleNotFoundError: No module named 'somemodule'
I found this odd cause when I ran the script directly it worked fine.
After attempting to run the script with sudo I realized what was happening.
Python imports are specific to the user, not to the folder. So if you python or pip install with your current logged-in user and try running the same script as sudo or some other user it will probably return ModuleNotFoundError: No module named 'somemodule'
Supervisor by default runs as root
I solved this by setting the user in the supervisor config file to the current user which in my case ubuntu:
[program:some_program]
directory=/home/ubuntu/scripts/
command=/usr/bin/python3 myscript.py
autostart=true
autorestart=true
user=ubuntu
stderr_logfile=/var/log/supervisor/myscriptlogs.err.log
stdout_logfile=/var/log/supervisor/myscriptlogs.out.log
As a side note, it's also important to make sure your Supervisor command= is calling the script from the version of Python you intend.
cd to the folder where your script lives and run which python and use that

Your script is failing with an exit status. Supervisor is simply trying to restart the script.
Supervisor is started with root permissions, perhaps it is giving those permissions to your script and this is causing it to fail (a change in the source directory or something). Check what happens when you run your daemon as root without Supervisor.
We really need more information to know why it is failing.

Not sure if the issue is the same with daemon runner, but if you use the daemon context directly and use supervisord, you need to set context.detach_process to False

How to add a delay to supervised process in supervisor - linux

I added a bottle server that uses python's cassandra library, but it exits with this error: Bottle FATAL Exited too quickly (process log may have details) log shows this: File "/usr/local/lib/python2.7/dist-packages/cassandra/cluster.py", line 1765, in _reconnect_internal
raise NoHostAvailable("Unable to connect to any servers", errors)So I tried to run it manually using supervisorctl start Bottle ,and then it started with no issue. The conclusion= Bottle service starts too fast (before the needed cassandra supervised service does): a delay is needed!

This is what I use:
[program:uwsgi]
command=bash -c 'sleep 5 && uwsgi /etc/uwsgi.ini'

Not happy enough with the sleep hack I created a startup script and launched supervisorctl start processname from there.
[program:startup]
command=/startup.sh
startsecs = 0
autostart = true
autorestart = false
startretries = 1
priority=1
[program:myapp]
command=/home/website/venv/bin/gunicorn /home/website/myapp/app.py
autostart=false
autorestart=true
process_name=myapp
startup.sh
#!/bin/bash
sleep 5
supervisorctrl start myapp
This way supervisor will fire the startup script once and this will start myapp after 5 seconds, mind the autostart=false and autorestart=true on myapp.

I had a similar issue where, starting 64 python rq-worker processes using supervisorctl was raising CPU and RAM alert at every restart. What I did was the following:
command=/bin/bash -c "sleep %(process_num)02d && virtualenv/bin/python3 manage.py rqworker --name %(program_name)s_my-rq-worker_%(process_num)02d default low"
Basically, before running the python command, I sleep for N second, where N is the process number, which basically means I supervisor will start one rq-worker process every second.

How can I run uWsgi as a service in CentOs?

I am in a hurry, I can find out how to do this but I need some help to achieve this without loosing too much time.
Currently what I do to run a uWsgi instance along with my ini file is just:
uwsgi --ini /home/myonlinesite/uwsgi.ini --pidfile /var/run/uwsgi_serv.pid
and then to stop:
uwsgi --stop /var/run/uwsgi_serv.pid.
By the way, I have this code inside a uwsgi init file in my /etc/init.d/uwsgi.
so when I run /etc/init.d/uwsgi start it executes the ini config file and when I execute /etc/init.d/uwsgi stop it stops the uwsgi process id.
The problem is that when I start the uWsgi service it runs normally and logs every http request, any debug print and so on, but when I close putty which is where I run my Vps it kills all uWsgi process and quits the site from being displayed.
I do not know if I have to touch the pid file only, or what do I need to do leave the uWsgi process executing and I can close putty.
Thanks in advance.

If you are setting the parameters in the command line, add the flag -d file.log to your command (-d stands for daemonize):
uwsgi --ini /home/myonlinesite/uwsgi.ini --pidfile /var/run/uwsgi_serv.pid -d file.log
If you are setting the parameters in a config file, add the following line in your config:
daemonize = /absolute/path/to/file.log
In both cases, uWsgi will run in the background and log everything in file.log. Given these options, there is no need using nohup et al.

Using nohup to start the uWsgi process should solve your problem of the process stopping when you log out.
A tutorial

Be sure to add
daemon = logfile
to your config

Need help running Python app as service in Ubuntu with Upstart

I have written a logging application in Python that is meant to start at boot, but I've been unable to start the app with Ubuntu's Upstart init daemon. When run from the terminal with sudo /usr/local/greeenlog/main.pyw, the application works perfectly. Here is what I've tried for the Upstart job:
/etc/init/greeenlog.conf
# greeenlog
description "I log stuff."
start on startup
stop on shutdown
script
exec /usr/local/greeenlog/main.pyw
end script
My application starts one child thread, in case that is important. I've tried the job with the expect fork stanza without any change in the results. I've also tried this with sudo and without the script statements (just a lone exec statement). In all cases, after boot, running status greeenlog returns greeenlog stop/waiting and running start greeenlog returns:
start: Rejected send message, 1 matched rules; type="method_call", sender=":1.61" (uid=1000 pid=2496 comm="start) interface="com.ubuntu.Upstart0_6.Job" member="Start" error name="(unset)" requested_reply=0 destination="com.ubuntu.Upstart" (uid=0 pid=1 comm="/sbin/init"))
Can anyone see what I'm doing wrong? I appreciate any help you can give. Thanks.

Thanks to unutbu's help, I have been able to correct my job. Apparently, these are the only environment variables that Upstart sets (retrieved in Python with os.environ):
{'TERM': 'linux', 'PWD': '/', 'UPSTART_INSTANCE': '', 'UPSTART_JOB': 'greeenlog', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin'}
My program relies on a couple of these variables being set, so here is the revised job with the right environment variables:
# greeenlog
description "I log stuff."
start on startup
stop on shutdown
env DISPLAY=:0.0
env GTK_RC_FILES=/etc/gtk/gtkrc:/home/greeenguru/.gtkrc-1.2-gnome2
script
exec /usr/local/greeenlog/main.pyw > /tmp/greeenlog.out 2>&1
end script
Thank you!