I have to write the time it takes for several C programs to run on several files using:
time ./program filename
to a spread sheet and am using subprocess.check_output to get the stdout as a string. I should get something along the lines of:
real 0m0.001s
user 0m0.001s
sys 0m0.000s
but I get:
b'0.00user 0.00system 0:00.00elapsed ?%CPU (0avgtext+0avgdata
1388maxresident)k\n0inputs+0outputs (0major+60minor)pagefaults
0swaps\n'
I see the user and system time but they get cut off after two decimal places. Is there a way to make sure the output reads all 3 decimal places?
Here is my code:
import xlwt
import subprocess
files = ('100KB.txt', '1MB.txt', '10MB.txt', '100MB.txt')
programs = ('./10kBuffer', './step2', './step3', './step4')
command = ['time', programs[0], files[0]]
out = subprocess.check_output(command, stderr=subprocess.STDOUT)
print(out)
that's because GNU time uses the default format string, more detailed, but you need -p option.
Quoting the manual:
The default format string is:
%Uuser %Ssystem %Eelapsed %PCPU (%Xtext+%Ddata %Mmax)k
%Iinputs+%Ooutputs (%Fmajor+%Rminor)pagefaults %Wswaps
When the -p option is given, the (portable) output format is used:
real %e
user %U
sys %S
You also need to decode the output or you'll get bytes instead of str, and newlines won't be interpreted. ex:
>>> print(b'hello\nworld\n')
b'hello\nworld\n'
>>> print('hello\nworld\n')
hello
world
So I would fix your code as is:
command = ['time', '-p', programs[0], files[0]]
out = subprocess.check_output(command, stderr=subprocess.STDOUT)
print(out.decode())
EDIT: the other answer seems to help fixing the missing decimals by using shell built-in. You can mix both answers to get the output you need as string, with enough decimals.
Note that it doesn't seem you can do much better, unless you want to use a profiler for your command (see How do I get time of a Python program's execution?)
It looks like you are running into confusion between GNU time getting used by your python script and the time shell built-in being used on command line.
This comes from the man page for GNU time:
Note: some shells (e.g., bash(1)) have a built-in time command that provides less functionality than the command described here. To access the
real command, you may need to specify its pathname (something like /usr/bin/time).
Based on the output you were expecting, it looks like you are wanting the bash built-in, which gives 3 decimal places:
$ bash -c time time
real 0m0.000s
user 0m0.000s
sys 0m0.000s
$ sh -c time time
user 0m0.00s
sys 0m0.00s
$ ksh -c time time
user 0m0.00s
sys 0m0.00s
$ tcsh -c time time
0.016u 0.011s 0:00.02 100.0% 0+0k 0+0io 0pf+0w
So in order to specify the bash built-in rather than GNU time, you can change your command to:
command = ['bash', '-c', 'time', programs[0], files[0]]
and you should get the output you were expecting.
Related
I'm trying to run a Python script in PowerShell. While using sys module I stumble upon a problem of getting the return value of a PowerShell function. The function in question is Date -f 'dd\/MM\/yyyy' that returns 14/03/2019 and I'd like that value to be used as an argument in the Python script. I've been able so far to get the literal arguments from the command line, e.g. text.
This Python script (see sys.argv docs):
import sys
print(sys.argv)
called like this in Powershell:
python .\test.py -date $(Date -f "dd\/MM\/yyyy")
outputs:
['.\\test.py', '-date', '14/03/2019']
On a general note, I recommend using better date formats than dd/MM/yyyy - ISO 8061 yyyy-MM-dd would be a good one.
If I understand correctly, you want to have the output of the command Date -f 'dd\/MM\/yyyy' to be fed into your python script (script.py).
What you are looking for are so called "pipes" or output redirects. Looking at the officical documentation (https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/write-output?view=powershell-6) the following might work:
$DATE=Date -f 'dd/\MM\/yyyy'
Write-Output $DATE | python script.py
I m want to extract the scene change timestamp using the scene change detection from ffmpeg. I have to run it on a few hundreds of videos , so i wanted to use a python subprocess to loop over all the content of a folder.
My problem is that the command that i was using for getting these values on a single video involve piping the output to a file which seems to not be an option from inside a subprocess call.
this is my code :
p=subprocess.check_output(["ffmpeg", "-i", sourcedir+"/"+name+".mpg","-filter:v", "select='gt(scene,0.4)',showinfo\"","-f","null","-","2>","output"])
this one tell ffmpeg need an output
output = "./result/"+name
p=subprocess.check_output(["ffmpeg", "-i", sourcedir+"/"+name+".mpg","-filter:v", "select='gt(scene,0.4)',metadata=print:file=output","-an","-f","null","-"])
this one give me no error but doesn't create the file
this is the original command that i use directly with ffmpeg:
ffmpeg -i input.flv -filter:v "select='gt(scene,0.4)',showinfo" -f null - 2> ffout
I just need the ouput of this command to be written to a file, anyone see how i could make it work?
is there a better way then subprocess ? or just another way ? will it be easier in C?
You can redirect the stderr output directly from Python without any need for shell=True which can lead to shell injection.
It's as simple as:
with open(output_path, 'w') as f:
subprocess.check_call(cmd, stderr=f)
Things are easier in your case if you use the shell argument of the subprocess command and it should behave the same. When using the shell command, you can pass in a string as the command rather then a list of args.
cmd = "ffmpeg -i {0} -filter:v \"select='gt(scene,0.4)',showinfo\" -f {1} - 2> ffout".format(inputName, outputFile)
p=subprocess.check_output(cmd, shell=True)
If you want to pass arguments, you can easily format your string
I'm looking for a way to monitor a file that is written to by a program on Linux. I found the tail -F command in here, and also recommended was less +FG. I tested it by running tail -F file in one terminal, and a simple python script:
import time
for i in range(20):
print i
time.sleep(0.5)
in another. I redirected the output to the file:
python script.py >> file
I expected that tail would track the file contents and update the display in fixed intervals, instead it only shows what was written to the file after the command terminates.
The same thing happens with less +FG and also if I watch the output from cat. I've also tried using the usual redirect which truncates the file > instead of >>. Here it says the file was truncated, but still does not track it in real time.
Any idea why this doesn't work? (It's suggested here that it might be due to buffered writes, but since my script runs over 10 seconds, I suspect this might not be the cause)
Edit: In case it matters, I'm running Linux Mint 18.1
Python's standard out is buffered. If when you close the script / script is done, you see all the output - that's definitely buffer issue.
You can use this instead:
import time
import sys
for i in range(20):
sys.stdout.write('%d\n' % i)
sys.stdout.flush()
time.sleep(0.5)
I've tested it and it prints values in real time. To overcome buffer issue, after each .write() method I use .flush() force "flushing" the buffer.
Additional options from the comments:
Use the original print statement with sys.stdout.flush() afterwords
Run the python script with python -u for unbuffered binary stdout and stderr
Regarding jon1467 answer (sorry can't comment your answer), your understanding of redirection is wrong.
Try this :
dd if=/dev/urandom > test.txt
while looking at the file size with :
ls -l test.txt
You'll see the file grow while dd is running.
Vinny's answer is correct, python standard output is buffered.
The more common way to the "buffering effect" you notice is by flushing the stdout as Vinny showed you.
You could also use -u option to disable buffering for the whole python process, or you could just reopen standard output with a buffer size of 0 as below (in python2 at least):
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
I want to run a program on Python on macOS Sierra that checks Terminal for its outputs after I automatically enter a command on it. For example, I would write in Terminal:
$ pwd
and then Terminal would output something like:
/Users/username
How would I have Python scan what Terminal outputs and set it to a variable as a string?
>>>output = (whatever Terminal outputs)
>>>print (output)
"/Users/username"
By the way, the other forums do not explain in much detail how one would do this in macOS. Therefore, this is not a duplicate of any forum.
You could pipe the output to a file and read the file.
$ pwd > output.txt
Then read the file and take further actions based on its contents.
Use the subprocess module, it has some shortcut methods to make things easier and less complicated than using Popen.
>>> import subprocess
>>> output = subprocess.check_output("pwd")
>>> print(output)
b'L:\\\r\n'
You can decode this using output.decode("UTF-8") if you like or you can use the universal_newlines keyword argument to have it done automatically as well as sorting out newlines.
>>> subprocess.check_output("pwd", universal_newlines=True)
'L:\\\n'
Edit: With #Silvio's sensible suggestion, passing all arguments you can do the following:
subprocess.check_output(["ls", "-l"])
Or if you have a string sourced from elsewhere you can call .split() which will generate a list of substrings separated by a space.
subprocess.check_output("ls -l /".split())
Note: I'm using Python3 on Windows and Gnu on Windows so I have \r\n line endings and pwd.
I've got a webserver that I'm presently benchmarking for CPU usage. What I'm doing is essentially running one process to slam the server with requests, then running the following bash script to determine the CPU usage:
#! /bin/bash
for (( ;; ))
do
echo "`python -c 'import time; print time.time()'`, `ps -p $1 -o '%cpu' | grep -vi '%CPU'`"
sleep 5
done
It would be nice to be able to do this in Python so I can run it in one script instead of having to run two. I can't seem to find any platform independent (or at least platform independent to linux and OS X) way to get the ps output in Python without actually launching another process to run the command. I can do that, but it would be really nice if there were an API for doing this.
Is there a way to do this, or am I going to have to launch the external script?
You could check out this question about parsing ps output using Python.
One of the answers suggests using the PSI python module. It's an extension though, so I don't really know how suitable that is for you.
It also shows in the question how you can call a ps subprocess using python :)
My preference is to do something like this.
collection.sh
for (( ;; ))
do
date; ps -p $1 -o '%cpu'
done
Then run collection.sh >someFile while you "slam the server with requests".
Then kill this collection.sh operation after the server has been slammed.
At the end, you'll have file with your log of date stamps and CPU values.
analysis.py
import datetime
with( "someFile", "r" ) as source:
for line in source:
if line.strip() == "%CPU": continue
try:
date= datetime.datetime.strptime( line, "%a %b %d %H:%M:%S %Z %Y" )
except ValueError:
cpu= float(line)
print date, cpu # or whatever else you want to do with this data.
You could query the CPU usage with PySNMP. This has the added benefit of being able to take measurements from a remote computer. For that matter, you could install a VM of Zenoss or its kin, and let it do the monitoring for you.
if you don't want to invoke PS then why don't you try with /proc file system.I think you can write you python program and read the files from /proc file system and extract the data you want.I did this using perl,by writing inlined C code in perl script.I think you can find similar way in python as well.I think its doable,but you need to go through /prof file system and need to figure out what you want and how you can get it.
http://www.faqs.org/docs/kernel/x716.html
above URL might give some initial push.