We Keep Coding

Using ffmpeg with subprocess python

I am trying to download videos from Wistia and I managed to download them but in .bin&amp format ; I'd like to convert them to .mp4 in order to use OpenCV. For this I am calling ffmpeg with subprocess on Python but I get 1 as the value for the return code, meaning the process has failed. Any idea why, and how I can change this...?
Code is the following:
import subprocess
infile = filename #a bin&amp file
outfile = filename[:-7]+'mp4'
subprocess.run(['ffmpeg', '-i', infile, outfile],shell=True)
I get :
CompletedProcess(args=['ffmpeg', '-i', '58c63bccfcc1c150646c261caad97a58ced4b5e3.bin&amp', '58c63bccfcc1c150646c261caad97a58ced4b5e3.mp4'], returncode=1)
Also, it works in the command prompt...
Thank you for your help,
Sincerely,

Try to use Popen. I I'm using FFmpeg in python and it works well
process_ = subprocess.Popen(ffmpeg_cmd, start_new_session=True)

Subprocess writing stdin and reading stdout python 3.4

I am writing a script which would run a Linux command and write a string (up to EOL) to stdin and read a string (until EOL) from stdout. The easiest illustration would be cat - command:
p=subprocess.Popen(['cat', '-'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
stringin="String of text\n"
p.stdin.write=(stringin)
stringout=p.stout.read()
print(stringout)
I aim to open the cat - process once and use it to write a string multiple times to its stdin every time getting a string from its stdout.
I googled quite a bit and a lot of recipes don't work, because the syntax is incompatible through different python versions (I use 3.4). That is my first python script from scratch and I find the python documentation to be quite confusing so far.

Thank you for your solution Salva.
Unfortunately communicate() closes the cat - process. I did not find any solution with subprocess to communicate with the cat - without having to open a new cat - for every call. I found an easy solution with pexpect though:
import pexpect
p = pexpect.spawn('cat -')
p.setecho(False)
def echoback(stringin):
p.sendline(stringin)
echoback = p.readline()
return echoback.decode();
i = 1
while (i < 11):
print(echoback("Test no: "+str(i)))
i = i + 1
In order to use pexpect Ubuntu users will have to install it through pip. If you wish to install it for python3.x, you will have to install pip3 (python3-pip) first from the Ubuntu repo.

Well you need to communicate with the process:
from subprocess import Popen, PIPE
s = Popen(['cat', '-'], stdin=PIPE, stdout=PIPE, stderr=PIPE)
input = b'hello!' # notice the input data are actually bytes and not text
output, errs = s.communicate(input)
To use unicode strings, you would need to encode() the input and decode() the output:
from subprocess import Popen, PIPE
s = Popen(['cat', '-'], stdin=PIPE, stdout=PIPE, stderr=PIPE)
input = 'España'
output, errs = s.communicate(input.encode())
output, errs = output.decode(), errs.decode()

Python: Write stdout to log file; output is hexadecimal not ascii

I'm working on a script to call an executable for i/o files. I'm using subprocess and trying to shell out the exe and the stdout to a log file. Problem is I would like to output a simple ascii file and I'm getting a hexadecimal file. Just really learning to program python (or any language for that matter) so, I'm assuming there some type of formatting I can do but I just don't get it. I've done a fair bit of searching on this site and others but I haven't anything like what I'm using subprocess for. The "outRadcorr" is what I need help on the most...Any ideas? More code on request.
Import system modules
import os, sys, string, traceback, time, datetime
import params
from subprocess import Popen, PIPE, STDOUT
...some code here.....
Write stdout to log file
rad_log_file = open(dsFolder + '\\radcorr.log', 'w')
# loop through the files in raw file list to run radiometric correction
for rawfiles in rawFolderList:
# Define the file base
rawBase = rawfiles.split(".")[0]
print ('\nProcessing file: %s \n')%( rawBase )
# define variables from raw file to process radcorr
radFile = rawfiles
pixFile = ('%s.pix')%( rawBase )
attFile = ('%s.att')%( rawBase )
# create windose bat file function
def rad_bat_writer( radcorr_bat ):
with open(dsFolder + '\\radcorr.bat', 'a') as rad_bat_file:
rad_bat_file.write(radcorr_bat + '\n')
# grab radcor input/output files and run radcorr.exe
radcorr = ('C:\\itres\\bin\\radcorr.exe -f 1 -j 100 -g 50 -s -1 -n -1 %s %s %s -i '+ rawFolder + '\%s,rb -o ' + radFolder + '\%s -a ' \
+ radFolder + '\%s -c C:\\itres\\rad_cal_files\\%s -I 0 -v 0 -r Y -R Y -^^ 2') %( sum,scatter,shift,radFile,pixFile,attFile,rad_prefix )
# print out radcorr command
print radcorr
# Execute radcorr and write stdout
outRadcorr = Popen("{};".format(radcorr), stdout=PIPE, stderr=STDOUT)
for line in outRadcorr.stdout:
sys.stdout.write(line)
rad_log_file.write(line)
# write output to log
rad_bat_writer( radcorr )
# Close out exe and log files
outRadcorr.wait()
rad_log_file.close()

I was using UltraEdit to view the file, which you can view hex files in. I was not looking in hex mode. I might be confusing the terminology here. File looks normal in NotePad/NotePad++/WordPad etc. As I ran the script in debug mode, I could hit the loop file by file. First two files (1GB/file), the output log looked fine. Once the radcorr.log file was over 10kb, I change from a normal ascii output to this binary file viewed that looked hex. I can't post images yet, but just google ultraedit hexadecimal.
Still not sure why it moved to this format. Finally, output size was 45kb. I change to view/edit hex mode in UltraEdit and it looks fine. Just wanted to get it out there to see if others had any ideas why when I specified the log to be 'w' and not 'wb', for instance.
I do appreciate all you help. #J.F. Sebastian I'll have to test the code you posted, probably help fix potential bugs down the road.

Convert mp3 to wav on the fly using ffmpeg in Python

I'm trying to convert a mp3 file on the fly in Python to wav file using ffmpeg.
I call it using subprocess, how can I get it's output to play it as wav on the fly wthout saving it as the file (or playing it while its converting) and then playing it?
This is what I have so far:
I'm using aplay just for a example.
FileLocation = "/home/file.mp3"
subprocess.call(["ffmpeg", "-i", FileLocation etc etc "newfail.wav"])
os.system("aplay ... ") #play it on the fly here
As far as I understand, if I put "-" as file name, it will output it instead to stdout, but I don't know how to read stdout...

To emulate source arg1 arg2 | sink shell command without the shell:
from subprocess import Popen, PIPE
source = Popen(['source', 'arg1', 'arg2'], stdout=PIPE)
sink = Popen(['sink'], stdin=source.stdout)
source.stdout.close()
source.wait()
sink.wait()

Python equivalent of piping file output to gzip in Perl using a pipe

I need to figure out how to write file output to a compressed file in Python, similar to the two-liner below:
open ZIPPED, "| gzip -c > zipped.gz";
print ZIPPED "Hello world\n";
In Perl, this uses Unix gzip to compress whatever you print to the ZIPPED filehandle to the file "zipped.gz".
I know how to use "import gzip" to do this in Python like this:
import gzip
zipped = gzip.open("zipped.gz", 'wb')
zipped.write("Hello world\n")
However, that is extremely slow. According to the profiler, using that method takes up 90% of my run time since I am writing 200GB of uncompressed data to various output files. I am aware that the file system could be part of the problem here, but I want to rule it out by using Unix/Linux compression instead. This is partially because I have heard that decompressing using this same module is slow as well.

ChristopheD's suggestion of using the subprocess module is an appropriate answer to this question. However, it's not clear to me that it will solve your performance problems. You would have to measure the performance of the new code to be sure.
To convert your sample code:
import subprocess
p = subprocess.Popen("gzip -c > zipped.gz", shell=True, stdin=subprocess.PIPE)
p.communicate("Hello World\n")
Since you need to send large amounts of data to the sub-process, you should consider using the stdin attribute of the Popen object. For example:
import subprocess
p = subprocess.Popen("gzip -c > zipped.gz", shell=True, stdin=subprocess.PIPE)
p.stdin.write("Some data")
# Write more data here...
p.communicate() # Finish writing data and wait for subprocess to finish
You may also find the discussion at this question helpful.

Try something like this:
from subprocess import Popen, PIPE
f = open('zipped.gz', 'w')
pipe = Popen('gzip', stdin=PIPE, stdout=f)
pipe.communicate('Hello world\n')
f.close()

Using the gzip module is the official one-way-to-do-it and it's unlikely that any other pure python approach will go faster. This is especially true because the size of your data rules out in-memory options. Most likely, the fastest way is to write the full file to disk and use subprocess to call gz on that file.

Make sure you use the same compression level when comparing speeds. By default, linux gzip uses level 6, while python uses level 9. I tested this in Python 3.6.8 using gzip version 1.5, compressing 600MB of data from MySQL dump. With default settings:
python module takes 9.24 seconds and makes a file 47.1 MB
subprocess gzip takes 8.61 seconds and makes a file 48.5 MB
After changing it to level 6 so they match:
python module takes 8.09 seconds and makes a file 48.6 MB
subprocess gzip takes 8.55 seconds and makes a file 48.5 MB
# subprocess method
start = time.time()
with open(outfile, 'wb') as f:
subprocess.run(['gzip'], input=dump, stdout=f, check=True)
print('subprocess finished after {:.2f} seconds'.format(time.time() - start))
# gzip method
start = time.time()
with gzip.open(outfile2, 'wb', compresslevel=6) as z:
z.write(dump)
print('gzip module finished after {:.2f} seconds'.format(time.time() - start))

In addition to #srgerg's answer I want to apply same approach by disabling shell option shell=False, which is also done on #Moishe Lettvin's answer and recommended on (https://stackoverflow.com/a/3172488/2402577).
import subprocess
def zip():
f = open("zipped.gz", "w")
p1 = subprocess.Popen(["echo", "Hello World"], stdout=subprocess.PIPE)
p2 = subprocess.Popen(["gzip", "-9c"], stdin=p1.stdout, stdout=f)
p1.stdout.close()
p2.communicate()
f.close()
Please not that originally I am using this p1s output for git diff as:
p1 = subprocess.Popen(["git", "diff"], stdout=subprocess.PIPE)