We Keep Coding

python subprocess with ffmpeg give no output

I m want to extract the scene change timestamp using the scene change detection from ffmpeg. I have to run it on a few hundreds of videos , so i wanted to use a python subprocess to loop over all the content of a folder.
My problem is that the command that i was using for getting these values on a single video involve piping the output to a file which seems to not be an option from inside a subprocess call.
this is my code :
p=subprocess.check_output(["ffmpeg", "-i", sourcedir+"/"+name+".mpg","-filter:v", "select='gt(scene,0.4)',showinfo\"","-f","null","-","2>","output"])
this one tell ffmpeg need an output
output = "./result/"+name
p=subprocess.check_output(["ffmpeg", "-i", sourcedir+"/"+name+".mpg","-filter:v", "select='gt(scene,0.4)',metadata=print:file=output","-an","-f","null","-"])
this one give me no error but doesn't create the file
this is the original command that i use directly with ffmpeg:
ffmpeg -i input.flv -filter:v "select='gt(scene,0.4)',showinfo" -f null - 2> ffout
I just need the ouput of this command to be written to a file, anyone see how i could make it work?
is there a better way then subprocess ? or just another way ? will it be easier in C?

You can redirect the stderr output directly from Python without any need for shell=True which can lead to shell injection.
It's as simple as:
with open(output_path, 'w') as f:
subprocess.check_call(cmd, stderr=f)

Things are easier in your case if you use the shell argument of the subprocess command and it should behave the same. When using the shell command, you can pass in a string as the command rather then a list of args.
cmd = "ffmpeg -i {0} -filter:v \"select='gt(scene,0.4)',showinfo\" -f {1} - 2> ffout".format(inputName, outputFile)
p=subprocess.check_output(cmd, shell=True)
If you want to pass arguments, you can easily format your string

Chain of UNIX commands within Python

I'd like to execute the following UNIX command in Python:
cd 2017-02-10; pwd; echo missing > 123.txt
The date directory DATE = 2017-02-10 and OUT = 123.txt are already variables in Python so I have tried variations of
call("cd", DATE, "; pwd; echo missing > ", OUT)
using the subprocess.call function, but I’m struggling to find documentation for multiple UNIX commands at once, which are normally separated by ; or piping with >
Doing the commands on separate lines in Python doesn’t work either because it “forgets” what was executed on the previous line and essentiality resets.

You can pass a shell script as a single argument, with strings to be substituted as out-of-band arguments, as follows:
date='2017-02-10'
out='123.txt'
subprocess.call(
['cd "$1"; pwd; echo missing >"$2"', # shell script to run
'_', # $0 for that script
date, # $1 for that script
out, # $2 for that script
], shell=True)
This is much more secure than substituting your date and out values into a string which is evaluated by the shell as code, because these values are treated as literals: A date of $(rm -rf ~) will not in fact try to delete your home directory. :)

Doing the commands on separate lines in Python doesn’t work either
because it “forgets” what was executed on the previous line and
essentiality resets.
This is because if you have separate calls to subprocess.call it will run each command in its own shell, and the cd call has no effect on the later shells.
One way around that would be to change the directory in the Python script itself before doing the rest. Whether or not this is a good idea depends on what the rest of the script does. Do you really need to change directory? Why not just write "missing" to 2017-02-10/123.txt from Python directly? Why do you need the pwd call?
Assuming you're looping through a list of directories and want to output the full path of each and also create files with "missing" in them, you could perhaps do this instead:
import os
base = "/path/to/parent"
for DATE, OUT in [["2017-02-10", "123.txt"], ["2017-02-11", "456.txt"]]:
date_dir = os.path.join(base, DATE)
print(date_dir)
out_path = os.path.join(date_dir, OUT)
out = open(out_path, "w")
out.write("missing\n")
out.flush()
out.close()
The above could use some error handling in case you don't have permission to write to the file or the directory doesn't exist, but your shell commands don't have any error handling either.

>>> date = "2017-02-10"
>>> command = "cd " + date + "; pwd; echo missing > 123.txt"
>>> import os
>>> os.system(command)

Bash process substitution in Python with Popen

I'm attempting to create a looped video file by calling ffmpeg from the python subprocess library. Here's the part that's giving me problems:
import subprocess as sp
sp.Popen(['ffmpeg', '-f', 'concat', '-i', "<(for f in ~/Desktop/*.mp4; do echo \"file \'$f\'\"; done)", "-c", "copy", "~/Desktop/sample3.mp4"])
With the above code I'm getting the following error:
<(for f in /home/delta/Desktop/*.mp4; do echo "file '$f'"; done): No such file or directory
I did find a similarly phrased question here. But I'm not sure how the solution might apply to solving my issue.

Following the advice in the comments and looking elsewhere I ended up changing the code to this:
sp.Popen("ffmpeg -f concat -i <(for f in ~/Desktop/*.mp4; do echo \"file \'$f\'\"; done) -c copy ~/Desktop/sample3.mp4",
shell=True, executable="/bin/bash")
--which works fine. – moorej
If you need to parameterize input and output files, consider breaking out your parameters:
# sample variables
inputDirectory = os.path.expanduser('~/Desktop')
outputDirectory = os.path.expanduser('~/dest.mp4')
sp.Popen(['''ffmpef -f concat -i <(for f in "$1"/*; do
echo "file '$f'";
done) -c copy "$2" ''',
bash, # this becomes $0
inputDirectory, # this becomes $1
outputDirectory, # this becomes $2
], shell=True, executable="/bin/bash")
...as this ensures that your code won't do untoward things even when given an input directory with a hostile name like /uploads/$(rm -rf ~)'$(rm -rf ~)'. (ffmpeg is likely to fail to parse an input file with such a name, and if there's any video you don't want included in the current working directory but we'd need to know the escaping rules it uses to avoid that; but it's far better for ffmpeg to fail than to execute arbitrary code).

Python: Write stdout to log file; output is hexadecimal not ascii

I'm working on a script to call an executable for i/o files. I'm using subprocess and trying to shell out the exe and the stdout to a log file. Problem is I would like to output a simple ascii file and I'm getting a hexadecimal file. Just really learning to program python (or any language for that matter) so, I'm assuming there some type of formatting I can do but I just don't get it. I've done a fair bit of searching on this site and others but I haven't anything like what I'm using subprocess for. The "outRadcorr" is what I need help on the most...Any ideas? More code on request.
Import system modules
import os, sys, string, traceback, time, datetime
import params
from subprocess import Popen, PIPE, STDOUT
...some code here.....
Write stdout to log file
rad_log_file = open(dsFolder + '\\radcorr.log', 'w')
# loop through the files in raw file list to run radiometric correction
for rawfiles in rawFolderList:
# Define the file base
rawBase = rawfiles.split(".")[0]
print ('\nProcessing file: %s \n')%( rawBase )
# define variables from raw file to process radcorr
radFile = rawfiles
pixFile = ('%s.pix')%( rawBase )
attFile = ('%s.att')%( rawBase )
# create windose bat file function
def rad_bat_writer( radcorr_bat ):
with open(dsFolder + '\\radcorr.bat', 'a') as rad_bat_file:
rad_bat_file.write(radcorr_bat + '\n')
# grab radcor input/output files and run radcorr.exe
radcorr = ('C:\\itres\\bin\\radcorr.exe -f 1 -j 100 -g 50 -s -1 -n -1 %s %s %s -i '+ rawFolder + '\%s,rb -o ' + radFolder + '\%s -a ' \
+ radFolder + '\%s -c C:\\itres\\rad_cal_files\\%s -I 0 -v 0 -r Y -R Y -^^ 2') %( sum,scatter,shift,radFile,pixFile,attFile,rad_prefix )
# print out radcorr command
print radcorr
# Execute radcorr and write stdout
outRadcorr = Popen("{};".format(radcorr), stdout=PIPE, stderr=STDOUT)
for line in outRadcorr.stdout:
sys.stdout.write(line)
rad_log_file.write(line)
# write output to log
rad_bat_writer( radcorr )
# Close out exe and log files
outRadcorr.wait()
rad_log_file.close()

I was using UltraEdit to view the file, which you can view hex files in. I was not looking in hex mode. I might be confusing the terminology here. File looks normal in NotePad/NotePad++/WordPad etc. As I ran the script in debug mode, I could hit the loop file by file. First two files (1GB/file), the output log looked fine. Once the radcorr.log file was over 10kb, I change from a normal ascii output to this binary file viewed that looked hex. I can't post images yet, but just google ultraedit hexadecimal.
Still not sure why it moved to this format. Finally, output size was 45kb. I change to view/edit hex mode in UltraEdit and it looks fine. Just wanted to get it out there to see if others had any ideas why when I specified the log to be 'w' and not 'wb', for instance.
I do appreciate all you help. #J.F. Sebastian I'll have to test the code you posted, probably help fix potential bugs down the road.

Passing a filename with an apostrophe into scp using python

I'm trying to write a python script to copy files from a remote server to a local directory via scp.
Because I'm running this on an OpenELEC distribution (minimal HTPC linux distro, read-only filesystem except for userhome makes it impractical to install python ssh module), I'm doing this ugly and just passing the filename to the scp command via os.system.
SCPCopy = "scp -c blowfish -C user#host:\"" + pipes.quote(file) + "\" /storage/downloads/incoming/"
SCPCopy = SCPCopy.replace('\n','')
os.system(SCPCopy)
This works, except for filenames containing an apostrophe.
Below is an example of what gets passed to os.system in a file with an apostrophe:
scp -c blowfish -C user#host:"'/media/sdi1/home/data/bob'"'"'s file.avi'" /storage/downloads/incoming/
And the error:
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
It looks pipes.quote(x) is escaping the apostrophe (as it should), but obviously the syntax is still incorrect. I've experimented ditching pipes.quote(x) and replacing apostrophes with /' but that isn't getting me anywhere either.

As scp is based on SSH, the filenames you give to it are subject to shell escaping on the remote side as well. Thus you need to escape twice.
A correctly escaped cmdline for the shell:
scp -c blowfish -C user#host:"\"/media/sdi1/home/data/bob's file\"" /storage/.../
To make a python string, we have to add one more level of escaping. To stay sane, we could use triple-quotes:
"""scp -c blowfish -C user#host:"\"/media/sdi1/home/data/bob's file\"" /storage/.../"""
If you do it programmatically (e.g. using the deprecated pipes.quote), then don't touch the filename at all (in your example above, you added apostrophes around the filename).
fp = "/media/sdi1/home/data/bob's file.avi"
fp = "user#host:" + pipes.quote(pipes.quote(fp))
cmdline = "scp -c blowfish -C " + fp + " /storage/downloads/incoming/"
os.system(cmdline)
This is admittedly confusing. For a simple model, the whole point of pipes.quote is to escape the input so that the input will be parsed by the shell as exactly one word, which is equal to the input.
The following is a more generally correct way (and yields the same result):
fp = "/media/sdi1/home/data/bob's file.avi"
# the filepath argument escaped for ssh/scp on the remote side
fp = pipes.quote(fp)
commandargs = ["scp", "-c", "blowfish", "-C", "user#host:"+fp, "/storage/downloads/incoming/"]
# escape all words for the local shell, and then concatenate space-separated
cmdline = " ".join(map(pipes.quote, commandargs))
os.system(cmdline)
It expresses more clearly the intent: Controlling what words exactly the shell will parse.
But why start with a shell in the first place? We don't need one and can save the escaping on the local side. To spawn a process with our args, directly, use commands from the os.exec* family.
fp = pipes.quote("/media/sdi1/home/data/bob's file.avi")
commandargs = ["scp", "-c", "blowfish", "-C", "user#host:"+fp, "/storage/downloads/incoming/"]
if os.fork() == 0:
os.execvp("scp", commandargs)