I need to convert a lot of files using mediapipe compiler which runs on bazel. There are hundreds of files so this process has to be automated. The command to be executed normally would be something like:
GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/multi_hand_tracking/multi_hand_tracking_cpu --input_video_path=/home/tony/Videos/HandWashDataset/Step_1/HandWash_001_A_01_G01.mp4
Where GLOG_logtostderr is the logger attached to the bazel program to output the log(result). I used redirect (2>a.txt) at the end to get the results to be written as txt file.
I have got the program working by writing a Python script using subprocess module.
import glob
import os
import time
import subprocess
files = glob.glob("/home/tony/Videos/HandWashDataset/Step_1/*.mp4")
os.chdir("/home/tony/mediapipe")
output = ""
for i in files:
print("process file {}".format(i))
output = (i[:len(i)-4]) + ".txt"
inputf = "--input_video_path=" + i
outputf = "2>" + output
f = open("blah.txt", "w")
sp = subprocess.call(["GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/multi_hand_tracking/multi_hand_tracking_cpu " + inputf], shell=True, stderr=f)
time.sleep(180)
f.close()
print("process finished")
The problem I am having is currently it seems to have no control on the process in each iteration. Since it is invoking another program in the script. The call to sp seems to be nearly instant but the actual conversion actually takes a few minutes. Without time.sleep, all instances of the bazel are launched at once and killed my computer. Is there a way to monitor process so I can convert one file at a time?
You should use subprocess.run :
That runs the command described by args,
waits for the command to complete,
then returns a CompletedProcess instance.
see also : https://docs.python.org/3/library/subprocess.html
Related
I want to run a compiled Fortran numerical model from Python. It is too complex to compile it using F2PY without implement several changes in the Fortran routines. This is why I am just calling its executable using the subprocess module.
The problem is that I have to call it few thousands of times, and I have the feeling that generating soo many shells is slowing the whole thing.
My implememtation (It is difficult to provide a reproducible example, sorry) looks like:
import os
import subprocess
foo_path = '/path/to/compiled/program/'
program_dir = os.path.join(foo_path, "FOO") #FOO is the Fortran executable
instruction = program_dir + " < nlst" #It is necesary to provide FOO a text file (nlst)
#with the configuration to the program
subprocess.call(instruction, shell=True, cwd=foo_path) #run the executable
Running it in this way (inside a loop), it works well and FOO generates a text file output that I can read from python. But I'd like to do the same keeping the shell active and just providing to it the "nlst" file path. Another nice option may be start an empty shell and keep it waiting for the instruction string, that will look like "./FOO < nlst". But I am not sure about how to do it, any ideas?
Thanks!
[Edited] Something like this should work but .comunicate ends process and a second call does not work:
from subprocess import Popen, PIPE
foo_path = '/path/to/FOO/'
process = Popen(['bash'], stdin=PIPE, cwd=foo_path)
process.communicate(b'./FOO < nlst')
I found this solution using the pexpect module,
import pexpect
import os.path
foo_path = '/path/to/FOO/'
out_path = '/path/to/FOO/foo_out_file' #path to output file
child = pexpect.spawn('bash', cwd=foo_path)
child.sendline('./FOO < nlst')
while not os.path.exists(out_path): #wait until out_path is created
continue
To extend my comment, here is an example for threading with your code:
import os
import subprocess
from concurrent.futures import ThreadPoolExecutor
foo_path = '/path/to/compiled/program/'
program_dir = os.path.join(foo_path, "FOO") #FOO is the Fortran executable
instruction = program_dir + " < nlst" #It is necesary to provide FOO a text file (nlst)
#with the configuration to the program
def your_function():
subprocess.call(instruction, shell=True, cwd=foo_path) #run the executable
# create executor object
executor = ThreadPoolExecutor(max_workers=4) # uncertain of how many workers you might need/want
# specify how often you want to run the function
for i in range(10):
# start your function as thread
executor.submit(your_function)
What I meant in my comment was something like the following Python script:
from subprocess import Popen, PIPE
foo_path = '/home/ronald/tmp'
process = Popen(['/home/ronald/tmp/shexecutor'], stdin=PIPE, cwd=foo_path)
process.stdin.write("ls\n")
process.stdin.write("echo hello\n")
process.stdin.write("quit\n")
And the shell script that executes the commands:
#!/bin/bash
while read cmdline; do
if [ "$cmdline" == "quit" ]; then
exit 0
fi
eval "$cmdline" >> x.output
done
Instead of doing an "eval", you can do virtually anything.
Note that this is just an outline of a real implementation.
You'd need to do some error handling. And if you are going to use this in a production environment, be sure to harden the code to the limit.
Here is my problem. I have an application which prints some traces to the standard output using logging module. Now, I want to be able to read those traces at the same time in order to wait for specific trace I need.
This is for the testing purpose. So for example, if wanted trace does not occur in about 2 seconds, test fails.
I know I can read output of another scripts by using something like this:
import subprocess
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while True:
line = p.stdout.readline()
print line
if line == '' and p.poll() != None:
break
But, how can I do something similar from the script itself?
Thanks in advance.
EDIT
So, since my problem was expecting certain trace to appear while the Python application is running, and since I couldn't find a simple way to do so from the application itself, I decided to start the application (as suggested in comments) from another script.
The module I found very helpful, and easier to use than subprocess module, is pexpect module.
If you want to do some pre-processing of the logger messages you can do something like:
#!/usr/bin/python
import sys
import logging
import time
import types
def debug_wrapper(self,msg):
if( hasattr(self,'last_time_seen') and 'message' in msg):
print("INFO: seconds past since last time seen "+str(time.time()-self.last_time_seen))
self.last_time_seen = time.time()
self.debug_original(msg)
logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
logger = logging.getLogger("test")
logger.debug_original = logger.debug
logger.debug = types.MethodType(debug_wrapper, logger)
while True:
logger.debug("INFO: some message.")
time.sleep(1)
This works by replacing the original debug function of the logger object with your custom debug_wrapper function, in which you can do whatever processing you want, like for example, storing the last time you have seen a message.
You can store the script output to a file in real-time and then read its content within the script in real-time(as the contents in the output file is updating dynamically).
To store the script output to a file in real-time, you may use unbuffer which comes with the expect package.
sudo apt-get install expect
Then, while running the script use:
unbuffer python script.py > output.txt
You have to just print the output in the script , which will be dynamically updating to the output file. And hence, read that file each time.
Also, use > for overwriting old or creating new file and >> for appending the contents in previously created output.txt file.
If you want to record the output from print statement in other Python code, you can redirect sys.stdout to string like file object as follows:
import io
import sys
def foo():
print("hello world, what else ?")
stream = io.StringIO()
sys.stdout = stream
try:
foo()
finally:
sys.stdout = sys.__stdout__
print(stream.getvalue())
I want to make a background process that displays a file with an external viewer. When the process is stopped, it should delete the file.
The following piece of code does what I want to do, but it is ugly and I guess there is a more idiomatic way.
It would be perfect, if it is even OS independent.
subprocess.Popen(viewer + ' ' + file + ' && rm ' + file, shell=True)
Using subprocess.call() to open the viewer and view the file will exactly do that. Subsequently, run the command to delete the file.
If you want the script to continue while the process is running, use threading
An example:
from threading import Thread
import subprocess
import os
def test():
file = "/path/to/somefile.jpg"
subprocess.call(["eog", file])
os.remove(file)
Thread(target = test).start()
# the print command runs, no matter if the process above is finished or not
print("Banana")
This will do exactly what you describe:
open the file with eog (viewer), wait for it to finish (closeeog) and remove the file.
In the meantime continue the script and print "Banana".
There is an excel file which contains the paths of multiple scenarios. I am using os.system(command) in a for loop. In each iteration, a path is read from excel file and executes the scenario for that path.
My problem is that every time, by using os.system(), a CMD opens, execute one command and close. In next iteration, again second path is read and execute it and CMD close. Here CMD pop-ups again and again. And the system is busy during that period and not able to do other task. I want to execute all the commands(scenario) in one CMD because I would like to minimize it and use the system for other task.
In each iteration, there are two main steps:
os.chdir(PATH)
os.system(path of exe+" "+name of config file that is present at PATH")
Can it be done by using subprocess. If yes please give me some example how it can be implemented?
If you want to use the subprocess module, try something like this :
from subprocess import call
import os.path
def call_scenario(path, config_file):
retcode = call(["path/of/exe", os.path.join(path,config_file)])
if retcode != 0:
print "Something bad happened : %s"%retcode
When using subprocess.call, the shell=False parameter will avoid to launch a cmd to do something.
it can be done - here is a quick example using multiprocessing (Python 2.6 or newer )
The example bellow uses a Unix command ("ls") and unixes paths ("/usr, etc...) but just replace those with your needed commands and paths.
from multiprocessing import Process
import os
paths = ["/tmp", "/usr", "/usr/include"]
def exec_(path):
p = Process()
p.run = lambda: os.system("ls %s" % path)
p.start()
for path in paths:
exec_(path)
Another option, if you need some sophisticated control on what is running, return codes, etc... is to make use of the Fabric project -
Although it is aimed at executingmultiple commands on different hosts using ssh - I think it culd be used for different paths on the same host.
URL for fabric:
http://docs.fabfile.org/en/1.3.3/index.html
To run c:\path\to\exe for all config.ini from each path simultaneously and to change current directory to cwd before it is executed:
from subprocess import Popen
processes = [Popen([r"c:\path\to\exe", "config.ini"], cwd=path) for path in paths]
for p in processes: p.wait()
If you don't want to run all commands in parallel then use subprocess.call() with the same arguments as for subprocess.Popen().
i have a java program which runs on a particular port in ubuntu. While running the program i need to take the output from the program and needs to save it in the log file. I use nohub to run them currently.when they fail I don't know why they fail.Then the process restart the nohub get overwritten . I want the process to restart and update the log file, I can check it at a later date. Currently I don't know the state of it, is it running or failed.
I heard that it is pretty easy to do it using python scripts .
Anyone please help me to do this?
Thanks in advance
Renjith Raj
You should use the subprocess module of python.
If your logs are not too big, you can simply use :
# for python >=2.7
result = subprocess.check_output(["/path/to/process_to_lauch", "arg1"])
# for python < 2.7
process = subprocess.Popen(["/path/to/process_to_lauch", "arg1"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
str_out, str_errr = process.communicate()
# in str_out you will find the standard output of your process
# in str_err you will find the standard output of your process
But if your outputs are really big (let's talk in Mo, not in Ko), this may cause some memory overflow...
In case of big output, use file handles for stdout and stderr:
out_file = open(out_file_name, "w")
err_file = open(out_file_name, "w")
process = subprocess.Popen(["/path/to/process_to_lauch", "arg1"], stdout=out_file, stderr=err_file)
return_code = process.wait()
out_file.close()
err_file.close()
And then, in out_file you'll find the output of the process, and in err_file the error output.
Of course, if you want to relaunch the process when it die, put this code in a loop ;)