We Keep Coding

Random behaviour while reading from subprocess stdout in a thread

I am writing a script that to start a process and check's its stdout (while it's being run, not at the end of execution).
The obvious choice seemed to have a thread that will be blocked reading lines from the process stdout.
I have tested it with WSL2 bash using:
python __main__.py 'echo ok'
The outcome is random, resulting in one of the following cases:
Execution terminated without any output
"ok" printed as expected
"ok" printed follow by a 'ValueError: readline of closed file' exception
Any idea on what might be the problem ?
The code:
import argparse
from subprocess import Popen, PIPE
import sys
import threading
class ReadlineThread(threading.Thread):
def __init__(self, proc):
threading.Thread.__init__(self)
self._proc = proc
def run(self):
while self._proc.poll() is None:
line = self._proc.stdout.readline()
sys.stdout.buffer.write(line)
sys.stdout.flush()
def main():
parser = argparse.ArgumentParser()
parser.add_argument('command', nargs='+', help='bar help')
args = parser.parse_args()
with Popen(args.command, stdout=PIPE, stderr=PIPE, shell=True) as proc:
stdout_thread = ReadlineThread(proc)
stdout_thread.start()
if __name__ == "__main__":
main()

When you create a thread, it becomes part of the parent process. The parent process is the thread that runs your main function. In your main function, you call stdout_thread.start(), which begins the process of starting a thread and then immediately returns. Aftfer that, there is no more code in your main function, which results in python shutting down the main process. Since your thread is part of the main process, it will be taken down when the main process terminates. Meanwhile, the thread you've started up is still being created.
Here we have what is called a race condition. Your thread is starting while simultaneously the process it belongs to is shutting down. If your thread manages to start up and complete its work before the process terminates, you get your expected result. If the process terminates before the thread has started, you get no output. In the third situation, the process closes its stdout before the thread has finished reading it, resulting in an error.
To fix this, in your main function you should wait for your spawned thread to finish, which could be achieved by calling stdout_thread.join().

How check if a process has finished but without waiting?

I'm doing a small project in python/tkinter and I have been looking for a way to check if a process has finished but "without waiting". I have tried with:
process = subprocess.Popen(command)
while process.poll() is None:
print('Running!')
print('Finished!')
or:
process = subprocess.Popen(command)
stdoutdata, stderrdata = process.communicate()
print('Finished!')
Both codes execute the command and print "Finished!" when the process ends, but the main program freezes (waiting) and that's what I want to avoid. I need the GUI to stay functional while the process is running and then run some code right after it finishes. Any help?

It's common that you use the Thread module for that purpose:
For example:
# import Thread
from threading import Thread
import time
# create a function that checks if the process has finished
process = True
def check():
while process:
print('Running')
time.sleep(1) # here you can wait as much as you want without freezing the program
else:
print('Finished')
# call the function with the use of Thread
Thread(target=check).start()
# or if you want to keep a reference to it
t = Thread(target=check)
# you might also want to set thread daemon to True so as the Thread ends when the program closes
t.deamon = True
t.start()
This way when you do process=False the program will end and the output will show 'Finished'

How to keep sub-process running after main process has exited?

I have a requirement to use python to start a totally independent process. That means even the main process exited, the sub-process can still run.
Just like the shell in Linux:
#./a.out &
then even if the ssh connection is lost, then a.out can still keep running.
I need a similar but unified way across Linux and Windows
I have tried the multiprocessing module
import multiprocessing
import time
def fun():
while True:
print("Hello")
time.sleep(3)
if __name__ == '__main__':
p = multiprocessing.Process(name="Fun", target=fun)
p.daemon = True
p.start()
time.sleep(6)
If I set the p.daemon = True, then the print("Hello") will stop in 6s, just after the main process exited.
But if I set the p.daemon = False, the main process won't exit on time, and if I CTRL+C to force quit the main process, the print("Hello") will also be stopped.
So, is there any way the keep print this "Hello" even the main process has exited?

The multiprocessing module is generally used to split a huge task into multiple sub tasks and run them in parallel to improve performance.
In this case, you would want to use the subprocess module.
You can put your fun function in a seperate file(sub.py):
import time
while True:
print("Hello")
time.sleep(3)
Then you can call it from the main file(main.py):
from subprocess import Popen
import time
if __name__ == '__main__':
Popen(["python", "./sub.py"])
time.sleep(6)
print('Parent Exiting')

The subprocess module can do it. If you have a .py file like this:
from subprocess import Popen
p = Popen([r'C:\Program Files\VideoLAN\VLC\vlc.exe'])
The file will end its run pretty quickly and exit, but vlc.exe will stay open.
In your case, because you want to use another function, you could in principle separate that into another .py file

os.sytem() in Python gives infinite loop

My main Python script imports 2 other scripts; Test1.py and Test2.py.
Test1.py does multiprocessing, and Test2.py does a simple os.system('ls') command. When Test1.py is finished and Test.py is called, os.system(ls) is going crazy and creates infinite new processes. Does anyone know why this happens?
# Main
import multiprocessing
import Test1.py
import Test2.py
def doSomething():
# Function 1, file1...file10 contain [name, path]
data = [file1, file2, file3, file4, file5, file6, file7, file8, file9, file10]
if __name__ == '__main__':
pool = multiprocessing.Pool(processes=min(len(data), 5))
print pool.map(Test1.worker, data)
# Function 2
Test2.worker()
Test1.py; calls perl commands
def worker(data):
command = 'perl '+data[1].split('data_')[0]+'methods_FastQC\\fastqc '+data[1]+'\\'+data[0]+'\\'+data[0]+' --outdir='+data[1]+'\\_IlluminaResults\\_fastqcAnalysis'
process = subprocess.Popen(command, stdout=subprocess.PIPE)
process.wait()
process.stdout.read()
Test2.py should do ONE simple ls command, instead it never stops making new commands;
def worker():
command = 'ls'
os.system(command)
When looking at the processes if script is started, it seems like the processes after function1 also don't close properly. Via the Taskmanager I still see 5 extra pythonw.exe which don't seem to do anything. Only when I close the opened shell they go away. Thats probably related to why os.system(command) goes crazy in function 2? Does anyone have a solution, since I can't close the shell because the script is not finished since it still has to do function2?
Edit: When trying to find a solution, it also happened that function1 started with executing the commands from function(2) multiple times, and after that the perl commands. Which is even more weird.

It seems doSomething() is executed every time your main module is imported and it can be imported several times by multiprocessing during the workers initialization. You could check it by printing process pid: print(os.getpid()) in Test2.worker().
You should use if __name__ == '__main__': at the module level. It is error-prone to do it inside a function as your code shows.
import multiprocessing
# ...
if __name__ == '__main__': # at global level
multiprocessing.freeze_support()
main() # it calls do_something() and everything else
See the very first note in the introduction to multiprocessing.

python parallel programming

hi i need some guidelines how to write programm that executes other python programm but in maximum for example 6 times at once and always try to be 6 process be running even if one ends up
also i would like to know what is happening to those processes right know but i dont want to wait to any process to finish
what is the pid of just created process? and its still running? or there has been an error? or it finish sucessfully?
some job manager ...
import subprocess
def start():
proc = {}
for i in range (0,6):
proc[i] = subprocess.Popen(
['python', 'someprogramm.py', '--env', 'DEVELOPMENT', '-l'],
shell = True,
stdout = subprocess.PIPE,
stderr = subprocess.STDOUT
)
if __name__ == '__main__':
start()

Use celery.

Have a look at supervisord. It sounds like it will do what you want.

Maybe you can try with the multiprocessing module, it can handles pool of worker processes that seems similar to what you try to achieve.

You can use the poll() method to check if a process is still running. You'd have to loop through each process and check if it runs, and otherwise, run a new process.