We Keep Coding

Linux/pm2 is killing my Flask service using Python's multiprocessing library

I have a Flask service running on a particular port xxxx. Inside this flask service is an endpoint:
/buildGlobalIdsPool
This endpoint uses Python's multiprocessing library's Pool object to run parallel processes of a function:
with Pool() as p:
p.starmap(api.build_global_ids_with_recordlinkage, args)
We use pm2 process manager on a Linux server to manage our services. I am hitting the endpoint from Postman and everything works fine up until the code above is reached. As soon as processes are supposed to spawn, pm2 will kill the main Flask process, but the spawned processes will persist (I check using lsof -i:xxxx and I see multiple python3 processes running on this port). This happens whether I run the service using pm2 or if I simply run python3 app.py. My program works on my local Windows 10 machine.
Just curious what I could be missing that is native to Linux or pm2 that is killing this process or not allowing multiple processes on the same port, while my local machine handles the program just fine.
Thanks!

When using the python script as CGI, Subprocess Popen not able to run background process

I am trying to run another Python script in background from a CGI python script and want this script to run the process in background without waiting for the other script to complete. Somehow when I am running the same from Linux shell, I can run the other python script in background. But when I tried doing the same through CGI, The front end keeps on loading until the other script completes and not just make it run in background.
I have tried running the same on the Linux Shell and it works. When I shifted to CGI that is when the script waits for the other process to complete.
python1.py:
command = [sys.executable,'python2.py', str(senddata)]
proc=subprocess.Popen(command,shell=False,stdin=None,stdout=None,stderr=None,close_fds=True)
print("Content-type: text/html\r\n\r\n")
print("The script is running in background! Expect an email in 10 minutes.")
python2.py:
This script takes 2-5 minutes to execute and then sends an email to the group.
The expected output is to have this message:
The script is running in background! Expect an email in 10 minutes.
And run python2.py in background without waiting for it to complete.

The webserver will keep the client response active (causing the client to stay in a "loading" state) until all of the output from the CGI program has been collected and forwarded to the client. The way the webserver knows that all output has been collected is that it sees the standard output stream of the CGI process being closed. That normally happens when the CGI process exits.
The reason why you're having this problem is that when subprocess.Popen is told to execute a program with stdout=None, the spawned program will share its parent's standard output stream. Here that means that your background program shares the CGI process's standard output. That means that from the webserver's point of view that stream remains open until both of the processes exit.
To fix, launch the background process with stdout=subprocess.PIPE. If the background process misbehaves if its stdout gets closed when the CGI process dies, try launching it with stdout=open('/dev/null') instead.
The background process's stdin and stderr will have the same issue; they will be shared with the CGI process. AFAIK sharing those will not cause trouble as long as the background process does not attempt to read from its standard input, but if it does do that (or if you just want to be cautious) you can treat them the same way as you treat stdout and either set them to subprocess.PIPE or associate them with /dev/null.
More detail is at https://docs.python.org/2/library/subprocess.html#popen-constructor

Running same Pexpect program simultaneously on the same server but differnt terminals

I have a python pexpect code lets say Program1.py which logs-in to one router from each program using pexpect.spawn and does required operations using sendline and expect modules in pexpect.
If I try to run this same program, from multiple prompts on my server with each program login to two diffrerent routers only one program seems to get the expect() input while the other instance of the programs times out atchild.expect() -> at read non_blocing()
Example:
In prompt-1 on my RHEL Server I execute the program to login to router X with ARGs
bash$ python program1.py 10.11.12.13/2001 configure_MGMT
In Prompt-2 on my RHEL Server I execute the program to login to router Y with ARGs
bash$ python program1.py 20.20.12.13/2020 configure_MGMT
one of the Programs runs successfully, while the other hits TIMEOUT at the first child.expect() call.
Is this due to the GIL?
Is there a workaround for this?
(I wish to avoid multiprocessing here, because my webserver handles the multiprocessing aspect and executes the same program multiple times.)

The GIL has nothing to do with this, because independent processes do not share a GIL. The most likely cause of this is that your router only supports one log-in at a time. The easiest way to verify that this is the problem is to remove Python from the equation by manually logging in to the router from two different terminal sessions at the same time.

Found the answer to this. Since python programs will always get executed synchronously and not in parallel, the pexpect will timeout waiting for input while the process is waiting to get scheduled.
to run all of them as background processes and in parallel, we need to use the '&' at the end of execution CLI.
Example:
bash$ python program1.py 20.20.12.13/2020 configure_MGMT &
thanks

How to check if remote process is executed - Long polling?

I am using python 2.7.
I am using paramiko module to trigger a remote action. The process is long and typically runs for 15-20 minutes. At the end it generates a file locally.
I would not like to block a thread until the process is finished. Is there a way by which I can trigger another action once the file is generated?

Python: How to Run multiple programs on same interpreter

How to start an always on Python Interpreter on a server?
If bash starts multiple python programs, how can I run it on just one interpreter?
And how can I start a new interpreter after tracking number of bash requests, say after X requests to python programs, a new interpreter should start.
EDIT: Not a copy of https://stackoverflow.com/questions/16372590/should-i-run-1000-python-scripts-at-once?rq=1
Requests may come pouring in sequentially

You cannot have new Python programs started through bash run on the same interpreter, each program will always have its own. If you want to limit the number of Python programs running the best approach would be to have a Python daemon process running on your server and instead of creating a new program through bash on each request you would signal the daemon process to create a thread to handle the task.

To run a program forever in python:
while True :
do_work()
You could look at spawning threads for incoming request. Look at threading.Thread class.
from threading import Thread
task = new Thread(target=do_work, args={})
task.start()

You probably want to take a look at http://docs.python.org/3/library/threading.html and http://docs.python.org/3/library/multiprocessing.html; threading would be more lightweight but only allows one thread to execute at a time (meaning it won't take advantage of multicore/hyperthreaded systems), while multiprocessing allows for true simultaneous execution but can be a bit less lightweight than threading if you're on a system that doesn't utilize lightweight subprocesses and may not be as necessary if the threads/processes spend lots of time doing I/O requests.