I am running an HPC simulation on amazon AWS with spot instances. Spot instances can be terminated with 2 minutes notice by AWS. In order to check for termination you need to exectute curl on a spefiic URL every 5 seconds. It is a simple request that returns a json with the termination time, if AWS have initiated the termination process.
Currently I am using subprocess to run the script:
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, bufsize=1, universal_newlines=True)
for line in p.stdout:
if "Floating point exception" in line:
print(line.rstrip())
log.write(line)
log.flush()
p.wait()
status = p.returncode
print(status)
Is it possible to add a callback that is called every 5 seconds?
The callback would check the return of the curl command, if it finds a termination time it would set a flag in a file and exit. The main process will then end gracefully because of this flag.
To clarify, I do not want to interact or kill the main process. This particular process (not written by me) checks continuously the content of a file and exits gracefully if it finds a specific keyword. The callback would set this keyword.
Is this the right approach?
Write a function that runs the following loop;
launches curl in a subprocess and processes the returned JSON.
If the sim should terminate, it writes the required file and returns.
Otherwise sleep for 4.5 minutes.
Start that function in a threading.Thread before you launch the simulation.
You'd have to test what happens to your for line in p.stdout loop if the program running in p exits.
Maybe it will generate an exception. Or you might want to check p.poll() in the loop to handle that gracefully.
Related
I am using the following code to launch a subprocess :
# Run the program
subprocess_result = subprocess.run(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
check=False,
timeout=timeout,
cwd=directory,
env=env,
preexec_fn=set_memory_limits,
)
The launched subprocess is also a Python program, with a shebang.
This subprocess may last for longer than the specified timeout.
The subprocess does heavy computations and write results in a file and does not contain any signal handler.
According to the documentation https://docs.python.org/3/library/subprocess.html#subprocess.run, subprocess.run kills a child that timesout :
The timeout argument is passed to Popen.communicate(). If the timeout
expires, the child process will be killed and waited for. The
TimeoutExpired exception will be re-raised after the child process has
terminated.
When my subprocess timesout, I always receive the subprocess.TimeoutExpired exception, but from time to time the subprocess is not killed, hence still consuming resources on my machine.
So my question is, am I doing something wrong here ? If yes, what and if no, why do I have this issue and how can I solve it ?
Note : I am using Python 3.10 on Ubuntu 22_04
The most likely culprit for the behaviour you see is that the subprocess you are spawning is probably using multiprocessing and spawning its own child processes. Killing the parent process does not automatically kill the whole set of descendants. The granchildren are inherited by the init process (i.e. the process with PID 1) and will continue to run.
You can verify from the source code of suprocess.run :
with Popen(*popenargs, **kwargs) as process:
try:
stdout, stderr = process.communicate(input, timeout=timeout)
except TimeoutExpired as exc:
process.kill()
if _mswindows:
# Windows accumulates the output in a single blocking
# read() call run on child threads, with the timeout
# being done in a join() on those threads. communicate()
# _after_ kill() is required to collect that and add it
# to the exception.
exc.stdout, exc.stderr = process.communicate()
else:
# POSIX _communicate already populated the output so
# far into the TimeoutExpired exception.
process.wait()
raise
except: # Including KeyboardInterrupt, communicate handled that.
process.kill()
# We don't call process.wait() as .__exit__ does that for us.
raise
Here you can see at line 550 the timeout is set on the communicate call, if it fires at line 552 the subprocess is .kill()ed. The kill method sends a SIGKILL which immediately kills the subprocess without any cleanup. It's a signal that cannot be caught by the subprocess, so it's not possible that the child is somehow ignoring it.
The TimeoutException is then re-raised at line 564, so if your parent process sees this exception the subprocess is already dead.
This however says nothing of granchildren processes. Those will continue to run as children of PID 1.
I don't see any way in which you can customize how subprocess.run handles subprocess termination. For example, if it used SIGTERM instead of SIGKILL you could modify your child process or write a wrapper process that will catch the signal and properly kill all its descendants. But SIGKILL doesn't give you this luxury.
So I believe that for your use case you cannot use the subprocess.run facade but you should use Popen directly. You can look at the subprocess.run implementation and take just the things that you need, maybe dropping support for platforms you don't use.
Note: There are extremely rare situations in which the subprocesses won't die immediately on SIGKILL. I believe the only situation in which this happens is if the subprocess is performing a very long system call or other kernel operation, which might not be interrupted immediately. If the operation is in deadlock this might prevent the process from terminating forever. However I don't think that this is your case, since you did not mention that the process is stuck doing nothing, but from what you said the process simply seems to continue running.
Let's say I have a main.py script which does different stuff inside a while loop, among which it calls a binary via terminal command with subprocess.Popen(). The binary is meant to control an external piece of hardware. At execution, first it sets the configuration and then it enters an infinite while loop waiting for the user to input control commands or exit, both via terminal.
The goal is to create the subprocess in main.py, wait a few seconds until the configuration is done (for example, 10 seconds) and then keep it alive waiting until it is required to send a command while the main code continues with the 'other stuff'.
After the while loop in 'main.py' is done, the subprocess should be killed by typing 'exit'.
Can this be done without the parent and child processes impacting each other in a way one messes up with the other?
The current approach:
main.py
import subprocess
# initiate subprocess to set configuration
p = subprocess.Popen(path/to/binary,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
stdin=subprocess.PIPE,
universal_newlines=True)
# sleep 10 seconds until setting is done
sleep(10)
while True:
# do stuff
# send command to subprocess
(stdout, stderr) = p.communicate('cmd_to_subprocess\n')
# continue with stuff
# while loop finished
# kill subprocessed
p.communicate('exit\n')
I have a python script that creates a subprocess to run indexing operations (logstash to elasticsearch).
the code snippet is as follows,
process = subprocess.Popen([logstash, '-f', sample.conf],
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
I do not call process.wait(), as the subprocess I'm creating needs to run independent of the rest of the script.
But I do have to update the database record when the subprocess is complete. The indexing operation that I am running does not allow me to create a post job call that will allow me to update the database.
How can I handle this with python subprocess? I do store the PIDs of the jobs in a text file, but I'd like to have a trigger in place that knows when the subprocess is complete to execute the next script.
Since you appear to stash the process variable somewhere, later you can check its returncode attribute after calling its poll method.
If the process has completed then its returncode value won't be None and you can update your database.
You could create your process in a thread. You can wait from the thread, so you get the output, and it's non-blocking
import threading
import subprocess
def run_command():
p = subprocess.Popen([logstash, '-f', sample.conf],
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
output = p.stdout.read()
p.wait()
# now your command has ended, do whatever you like
and in your main thread:
t = threading.Thread(target=run_command)
t.start()
# continue with main processing
I am using Python 2.7.8 to coordinate and automate the running of several application many times over in a Windows environment. During each run, I use subprocess.Popen to launch several child process, passing subprocess.PIPE for stdin and stdout to each as follows:
proc = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
where cmd is the list of arguments.
The script waits for an external trigger to know when a given run is done, and then terminates each application it is currently running by writing a string to the stdin of each Popen object. The applications read this string, and perform its own graceful shutdown (which is why I don't simply call kill() or terminate()).
# Try to shutdown process
timeout = 5
try:
if proc.poll() is None:
proc.stdin.write(cmd)
# Wait to see if proc shuts down gracefully
while timeout > 0:
if proc.poll() is not None:
break
else:
time.sleep(1)
timeout -= 1
else:
# Kill it the old fashioned way
proc.kill()
except Error:
pass # Process as necessary...
Once the applications are complete, I'm left with a Popen object. If I inspect the stdin or stdout members of that object, I get something like the following:
<open file '<fdopen>', mode 'wb' at 0x0277C758>
The script then loops to perform the next run, relaunching the necessary applications.
My question is, do I need to explicitly call close() for the stdin and stdout file descriptors each time, in order to avoid leaks, i.e. in the finally statement above? I am wondering this because it is possible for the loop to occur hundreds, or even thousands of times during a given script.
I've looked through the subprocess.py code, but the file handles for the pipes are created by an apparent Windows(-only) call in the _subprocess module, so I can't get any further detail.
The pipes might eventually be closed during the garbage collection but you should not rely on it and close the pipes explicitly.
def kill_process(process):
if process.poll() is None: # don't send the signal unless it seems it is necessary
try:
process.kill()
except OSError: # ignore
pass
# shutdown process in `timeout` seconds
t = Timer(timeout, kill_process, [proc])
t.start()
proc.communicate(cmd)
t.cancel()
.communicate() method closes the pipes and waits for the child process to exit.
I'm running a tool via Python in cmd. For each sample in a given directory I want that tool to do something. However, when I use process = subprocess.Popen(command) in the loop, the commands does not wait untill its finished, resulting in 10 prompts at once. And when I use subprocess.Popen(command, stdout=subprocess.PIPE) the command remains black and I can't see the progress, although it does wait untill the command is finished.
Does anyone know a way how to call an external tool via Python in cmd, that does wait untill the command is finished and thats able to show the progress of the tool in the cmd?
#main.py
for sample in os.listdir(os.getcwd()):
if ".fastq" in sample and '_R1_' in sample and "Temp" not in sample:
print time.strftime("%H:%M:%S")
DNA_Bowtie2.DNA_Bowtie2(os.getcwd()+'\\'+sample+'\\'+sample)
#DNA_Bowtie2.py
# Run Bowtie2 command and wait for process to be finished.
process = subprocess.Popen(command, stdout=subprocess.PIPE)
process.wait()
process.stdout.read()
Edit: command = a perl or java command. With above make-up I cannot see tool output since the prompt (perl window, or java window) remains black.
It seems like your subprocess forks otherwise there is no way the wait() would return before the process has finished.
The order is important here: first read the output, then wait.
If you do it this way:
process.wait()
process.stdout.read()
you can experience a deadlock if the pipe buffer is completely full: the subprocess blocks on waiting on stdout and never reaches the end, your program blocks on wait() and never reaches the read().
Do instead
process.stdout.read()
process.wait()
which will read until EOF.
This holds for if you want the stdout of the process at all.
If you don't want that, you should omit the stdout=PIPE stuff. Then the output is directed into that prompt window. Then you can omit process.stdout.read() as well.
Normally, the process.wait() should then prevent that 10 instances run at once. If that doesn't work, I don't know why not...