We Keep Coding

subprocess.run simple scenario fails

I am trying to run python subprocess.run function to execute following command:
pdftoppm -jpeg -f 1 -scale-to 200 data/andromeda.pdf and-page
pdftoppm - is part of poppler utility and it generates images from pdf files.
File data/andromeda.pdf exists. Folder data is on same level with python script and/or where I run command from.
Command basically will generate a jpeg file, from page 1 (-f 1) 200px wide (-scale-to) from given file of and-page-1.jpeg format (so called ppmtroot).
Long story short: from command line it works as expected i.e. if I call the above command either from zsh or bash shell, manually - it generates thumbnail as expected. However if I run it from python subprocess module - it fails it returns 99 error code!
Following is python code (file name is sc_02_thumbnails.py):
import subprocess
import sys
def main(filename, ppmroot):
cmd = [
'pdftoppm',
'-f 1',
'-scale-to 200',
'-jpeg',
filename,
ppmroot
]
result = subprocess.run(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE
)
if result.returncode:
print("Failed to generate thumbnail. Return code: {}. stderr: {}".format(
result.returncode,
result.stderr
))
print("Used cmd: {}".format(' '.join(cmd)))
sys.exit(1)
else:
print("Success!")
if __name__ == "__main__":
if len(sys.argv) > 2:
filename = sys.argv[1]
ppmroot = sys.argv[2]
else:
print("Usage: {} <pdffile> <ppmroot>".format(sys.argv[0]))
sys.exit(1)
main(filename, ppmroot)
And here is repo which includes data/andromeda.pdf file as well.
I call my script with as (from zsh):
$ chmod +x ./sc_02_thumbnauils.py
$ ./sc_02_thumbnails.py data/andromeda.pdf and-page
and ... thumbnail generating fails!
I have tried executing python script from both, from zsh and bash shells :(
What I am doing wrong?

The quoting is wrong, you should have '-f', '1', etc

/bin/sh: line 62: to: command not found

I have a python code in which I am calling a shell command. The part of the code where I did the shell command is:
try:
def parse(text_list):
text = '\n'.join(text_list)
cwd = os.getcwd()
os.chdir("/var/www/html/alenza/hdfs/user/alenza/sree_account/sree_project/src/core/data_analysis/syntaxnet/models/syntaxnet")
synnet_output = subprocess.check_output(["echo '%s' | syntaxnet/demo.sh 2>/dev/null"%text], shell = True)
os.chdir(cwd)
return synnet_output
except Exception as e:
sys.stdout.write(str(e))
Now, when i run this code on a local file with some sample input (I did cat /home/sree/example.json | python parse.py) it works fine and I get the required output. But I am trying to run the code with an input on my HDFS (the same cat command but input file path is from HDFS) which contains exactly the same type of json entries and it fails with an error:
/bin/sh: line 62: to: command not found
list index out of range
I read similar questions on Stack Overflow and the solution was to include a Shebang line for the shell script that is being called. I do have the shebang line #!/usr/bin/bash in demo.sh script.
Also, which bash gives /usr/bin/bash.
Someone please elaborate.

You rarely, if ever, want to combine passing a list argument with shell=True. Just pass the string:
synnet_output = subprocess.check_output("echo '%s' | syntaxnet/demo.sh 2>/dev/null"%(text,), shell=True)
However, you don't really need a shell pipeline here.
from subprocess import check_output
from StringIO import StringIO # from io import StringIO in Python 3
synnet_output = check_output(["syntaxnet/demo.sh"],
stdin=StringIO(text),
stderr=os.devnull)

There was a problem with some special characters appearing in the text string that i was inputting to demo.sh. I solved this by storing text into a temporary file and sending the contents of that file to demo.sh.
That is:
try:
def parse(text_list):
text = '\n'.join(text_list)
cwd = os.getcwd()
with open('/tmp/data', 'w') as f:
f.write(text)
os.chdir("/var/www/html/alenza/hdfs/user/alenza/sree_account/sree_project/src/core/data_analysis/syntaxnet/models/syntaxnet")
synnet_output = subprocess.check_output(["cat /tmp/data | syntaxnet/demo.sh 2>/dev/null"%text], shell = True)
os.chdir(cwd)
return synnet_output
except Exception as e:
sys.stdout.write(str(e))

Logging last Bash command to file from script

I write lots of small scripts to manipulate files on a Bash-based server. I would like to have a mechanism by which to log which commands created which files in a given directory. However, I don't just want to capture every input command, all the time.
Approach 1: a wrapper script that uses a Bash builtin (a la history or fc -ln -1) to grab the last command and write it to a log file. I have not been able to figure out any way to do this, as the shell builtin commands do not appear to be recognized outside of the interactive shell.
Approach 2: a wrapper script that pulls from ~/.bash_history to get the last command. This, however, requires setting up the Bash shell to flush every command to history immediately (as per this comment) and seems also to require that the history be allowed to grow inexorably. If this is the only way, so be it, but it would be great to avoid having to edit the ~/.bashrc file on every system where this might be implemented.
Approach 3: use script. My problem with this is that it requires multiple commands to start and stop the logging, and because it launches its own shell it is not callable from within another script (or at least, doing so complicates things significantly).
I am trying to figure out an implementation that's of the form log_this.script other_script other_arg1 other_arg2 > file, where everything after the first argument is logged. The emphasis here is on efficiency and minimizing syntax overhead.
EDIT: iLoveTux and I both came up with similar solutions. For those interested, my own implementation follows. It is somewhat more constrained in its functionality than the accepted answer, but it also auto-updates any existing logfile entries with changes (though not deletions).
Sample usage:
$ cmdlog.py "python3 test_script.py > test_file.txt"
creates a log file in the parent directory of the output file with the following:
2015-10-12#10:47:09 test_file.txt "python3 test_script.py > test_file.txt"
Additional file changes are added to the log;
$ cmdlog.py "python3 test_script.py > test_file_2.txt"
the log now contains
2015-10-12#10:47:09 test_file.txt "python3 test_script.py > test_file.txt"
2015-10-12#10:47:44 test_file_2.txt "python3 test_script.py > test_file_2.txt"
Running on the original file name again changes the file order in the log, based on modification time of the files:
$ cmdlog.py "python3 test_script.py > test_file.txt"
produces
2015-10-12#10:47:44 test_file_2.txt "python3 test_script.py > test_file_2.txt"
2015-10-12#10:48:01 test_file.txt "python3 test_script.py > test_file.txt"
Full script:
#!/usr/bin/env python3
'''
A wrapper script that will write the command-line
args associated with any files generated to a log
file in the directory where the files were made.
'''
import sys
import os
from os import listdir
from os.path import isfile, join
import subprocess
import time
from datetime import datetime
def listFiles(mypath):
"""
Return relative paths of all files in mypath
"""
return [join(mypath, f) for f in listdir(mypath) if
isfile(join(mypath, f))]
def read_log(log_file):
"""
Reads a file history log and returns a dictionary
of {filename: command} entries.
Expects tab-separated lines of [time, filename, command]
"""
entries = {}
with open(log_file) as log:
for l in log:
l = l.strip()
mod, name, cmd = l.split("\t")
# cmd = cmd.lstrip("\"").rstrip("\"")
entries[name] = [cmd, mod]
return entries
def time_sort(t, fmt):
"""
Turn a strftime-formatted string into a tuple
of time info
"""
parsed = datetime.strptime(t, fmt)
return parsed
ARGS = sys.argv[1]
ARG_LIST = ARGS.split()
# Guess where logfile should be put
if (">" or ">>") in ARG_LIST:
# Get position after redirect in arg list
redirect_index = max(ARG_LIST.index(e) for e in ARG_LIST if e in ">>")
output = ARG_LIST[redirect_index + 1]
output = os.path.abspath(output)
out_dir = os.path.dirname(output)
elif ("cp" or "mv") in ARG_LIST:
output = ARG_LIST[-1]
out_dir = os.path.dirname(output)
else:
out_dir = os.getcwd()
# Set logfile location within the inferred output directory
LOGFILE = out_dir + "/cmdlog_history.log"
# Get file list state prior to running
all_files = listFiles(out_dir)
pre_stats = [os.path.getmtime(f) for f in all_files]
# Run the desired external commands
subprocess.call(ARGS, shell=True)
# Get done time of external commands
TIME_FMT = "%Y-%m-%d#%H:%M:%S"
log_time = time.strftime(TIME_FMT)
# Get existing entries from logfile, if present
if LOGFILE in all_files:
logged = read_log(LOGFILE)
else:
logged = {}
# Get file list state after run is complete
post_stats = [os.path.getmtime(f) for f in all_files]
post_files = listFiles(out_dir)
# Find files whose states have changed since the external command
changed = [e[0] for e in zip(all_files, pre_stats, post_stats) if e[1] != e[2]]
new = [e for e in post_files if e not in all_files]
all_modded = list(set(changed + new))
if not all_modded: # exit early, no need to log
sys.exit(0)
# Replace files that have changed, add those that are new
for f in all_modded:
name = os.path.basename(f)
logged[name] = [ARGS, log_time]
# Write changed files to logfile
with open(LOGFILE, 'w') as log:
for name, info in sorted(logged.items(), key=lambda x: time_sort(x[1][1], TIME_FMT)):
cmd, mod_time = info
if not cmd.startswith("\""):
cmd = "\"{}\"".format(cmd)
log.write("\t".join([mod_time, name, cmd]) + "\n")
sys.exit(0)

You can use the tee command, which stores its standard input to a file and outputs it on standard output. Pipe the command line into tee, and pipe tee's output into a new invocation of your shell:
echo '<command line to be logged and executed>' | \
tee --append /path/to/your/logfile | \
$SHELL
i.e., for your example of other_script other_arg1 other_arg2 > file,
echo 'other_script other_arg1 other_arg2 > file' | \
tee --append /tmp/mylog.log | \
$SHELL
If your command line needs single quotes, they need to be escaped properly.

OK, so you don't mention Python in your question, but it is tagged Python, so I figured I would see what I could do. I came up with this script:
import sys
from os.path import expanduser, join
from subprocess import Popen, PIPE
def issue_command(command):
process = Popen(command, stdout=PIPE, stderr=PIPE, shell=True)
return process.communicate()
home = expanduser("~")
log_file = join(home, "command_log")
command = sys.argv[1:]
with open(log_file, "a") as fout:
fout.write("{}\n".format(" ".join(command)))
out, err = issue_command(command)
which you can call like (if you name it log_this and make it executable):
$ log_this echo hello world
and it will put "echo hello world" in a file ~/command_log, note though that if you want to use pipes or redirection you have to quote your command (this may be a real downfall for your use case or it may not be, but I haven't figured out how to do this just yet without the quotes) like this:
$ log_this "echo hello world | grep h >> /tmp/hello_world"
but since it's not perfect, I thought I would add a little something extra.
The following script allows you to specify a different file to log your commands to as well as record the execution time of the command:
#!/usr/bin/env python
from subprocess import Popen, PIPE
import argparse
from os.path import expanduser, join
from time import time
def issue_command(command):
process = Popen(command, stdout=PIPE, stderr=PIPE, shell=True)
return process.communicate()
home = expanduser("~")
default_file = join(home, "command_log")
parser = argparse.ArgumentParser()
parser.add_argument("-f", "--file", type=argparse.FileType("a"), default=default_file)
parser.add_argument("-p", "--profile", action="store_true")
parser.add_argument("command", nargs=argparse.REMAINDER)
args = parser.parse_args()
if args.profile:
start = time()
out, err = issue_command(args.command)
runtime = time() - start
entry = "{}\t{}\n".format(" ".join(args.command), runtime)
args.file.write(entry)
else:
out, err = issue_command(args.command)
entry = "{}\n".format(" ".join(args.command))
args.file.write(entry)
args.file.close()
You would use this the same way as the other script, but if you wanted to specify a different file to log to just pass -f <FILENAME> before your actual command and your log will go there, and if you wanted to record the execution time just provide the -p (for profile) before your actual command like so:
$ log_this -p -f ~/new_log "echo hello world | grep h >> /tmp/hello_world"
I will try to make this better, but if you can think of anything else this could do for you, I am making a github project for this where you can submit bug reports and feature requests.

OSError: [Errno 8] Exec format error

I am having hard time parsing the arguments to subprocess.Popen. I am trying to execute a script on my Unix server. The script syntax when running on shell prompt is as follows:
/usr/local/bin/script hostname = <hostname> -p LONGLIST. No matter how I try, the script is not running inside subprocess.Popen
The space before and after "=" is mandatory.
import subprocess
Out = subprocess.Popen(['/usr/local/bin/script', 'hostname = ', 'actual server name', '-p', 'LONGLIST'],shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
The above does not work.
And when I use shell=False, I get OSError: [Errno 8] Exec format error

OSError: [Errno 8] Exec format error can happen if there is no shebang line at the top of the shell script and you are trying to execute the script directly. Here's an example that reproduces the issue:
>>> with open('a','w') as f: f.write('exit 0') # create the script
...
>>> import os
>>> os.chmod('a', 0b111101101) # rwxr-xr-x make it executable
>>> os.execl('./a', './a') # execute it
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/os.py", line 312, in execl
execv(file, args)
OSError: [Errno 8] Exec format error
To fix it, just add the shebang e.g., if it is a shell script; prepend #!/bin/sh at the top of your script:
>>> with open('a','w') as f: f.write('#!/bin/sh\nexit 0')
...
>>> os.execl('./a', './a')
It executes exit 0 without any errors.
On POSIX systems, shell parses the command line i.e., your script won't see spaces around = e.g., if script is:
#!/usr/bin/env python
import sys
print(sys.argv)
then running it in the shell:
$ /usr/local/bin/script hostname = '<hostname>' -p LONGLIST
produces:
['/usr/local/bin/script', 'hostname', '=', '<hostname>', '-p', 'LONGLIST']
Note: no spaces around '='. I've added quotes around <hostname> to escape the redirection metacharacters <>.
To emulate the shell command in Python, run:
from subprocess import check_call
cmd = ['/usr/local/bin/script', 'hostname', '=', '<hostname>', '-p', 'LONGLIST']
check_call(cmd)
Note: no shell=True. And you don't need to escape <> because no shell is run.
"Exec format error" might indicate that your script has invalid format, run:
$ file /usr/local/bin/script
to find out what it is. Compare the architecture with the output of:
$ uname -m

I will hijack this thread to point out that this error may also happen when target of Popen is not executable. Learnt it hard way when by accident I have had override a perfectly executable binary file with zip file.

Have you tried this?
Out = subprocess.Popen('/usr/local/bin/script hostname = actual_server_name -p LONGLIST'.split(), shell=False,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
Edited per the apt comment from #J.F.Sebastian

It wouldn't be wrong to mention that Pexpect does throw a similar error
#python -c "import pexpect; p=pexpect.spawn('/usr/local/ssl/bin/openssl_1.1.0f version'); p.interact()"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/lib/python2.7/site-packages/pexpect.py", line 430, in __init__
self._spawn (command, args)
File "/usr/lib/python2.7/site-packages/pexpect.py", line 560, in _spawn
os.execv(self.command, self.args)
OSError: [Errno 8] Exec format error
Over here, the openssl_1.1.0f file at the specified path has exec command specified in it and is running the actual openssl binary when called.
Usually, I wouldn't mention this unless I have the root cause, but this problem was not there earlier. Unable to find the similar problem, the closest explanation to make it work is the same as the one provided by #jfs above.
what worked for me is both
adding /bin/bash at the beginning of the command or file you are
facing the problem with, or
adding shebang #!/bin/sh as the first line.
for ex.
#python -c "import pexpect; p=pexpect.spawn('/bin/bash /usr/local/ssl/bin/openssl_1.1.0f version'); p.interact()"
OpenSSL 1.1.0f 25 May 2017

If you think the space before and after "=" is mandatory, try it as separate item in the list.
Out = subprocess.Popen(['/usr/local/bin/script', 'hostname', '=', 'actual server name', '-p', 'LONGLIST'],shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)

Python subprocess.call doesn't wait for command to execute

I'm new to python, which I need to use for an assignment in a course. I developed the solution (an optimization algorithm) in Freemat / octave / matlab .m file and wanted to call it from Python (the python code will be called by a grading python script).
The .m file reads a file called tmp.data and writes the output to output.txt. The python script should then read from that output and convert it to the result that the grading script expects.
All runs fine, except I haven't been able to make Python wait for the call to Matlab to complete and therefore generates an error on the following lines.
Here's the code:
#!/usr/bin/python
# -*- coding: utf-8 -*-
from collections import namedtuple
Item = namedtuple("Item", ['index', 'value', 'weight'])
import subprocess
import os
from subprocess import Popen, PIPE
def solve_it(input_data):
# Modify this code to run your optimization algorithm
# Write the inputData to a temporay file
tmp_file_name = 'tmp.data'
tmp_file = open(tmp_file_name, 'w')
tmp_file.write(input_data)
tmp_file.close()
# call matlab (or any other solver)
# subprocess.call('matlab -r gp(\'tmp.data\')', shell=1)
# run=os.system
# a=run('matlab -r gp(\'tmp.data\')')
# process = Popen('matlab -r gp(\'tmp.data\')', stdout=PIPE)
# Popen.wait()
# (stdout, stderr) = process.communicate()
subprocess.call('matlab -r gp(\'tmp.data\')',shell=0)
# Read result from file
with open('output.txt') as f:
result = f.read()
# remove the temporay file
os.remove(tmp_file_name)
os.remove('output.txt')
return result
# return stdout.strip()
# prepare the solution in the specified output format
# output_data = str(value) + ' ' + str(0) + '\n'
# output_data += ' '.join(map(str, taken))
# return output_data
import sys
if __name__ == '__main__':
if len(sys.argv) > 1:
file_location = sys.argv[1].strip()
input_data_file = open(file_location, 'r')
input_data = ''.join(input_data_file.readlines())
input_data_file.close()
print solve_it(input_data)
else:
print 'This test requires an input file. Please select one from the data directory. (i.e. python solver.py ./data/ks_4_0)'
As you see, I've tried with subprocess.call, popen, os.system... to no avail. All of them give me similar errors:
C:\Users\gp\Documents\Documents\personal\educacion\Discrete Optimization\knapsack>python2 solver.py data/ks_19_0
Traceback (most recent call last):
File "solver.py", line 60, in <module>
print solve_it(input_data)
File "solver.py", line 30, in solve_it
with open('output.txt') as f:
IOError: [Errno 2] No such file or directory: 'output.txt'
Of course! The error comes while matlab is still in the process of opening. It thus is trying to access a file that hasn't been created yet.
What should I do to get Python to wait for Matlab to complete??
I appreciate your kind help, thanks.

[for the record]
As Daniel pointed out, it was solved by introducing a couple options into the matlab call:
subprocess.call('matlab -nosplash -wait -r "gp(\'tmp.data\')"',shell=0)
After that, it ran beautifully.
Thanks

Your code seems to irgnore the fact that matlab uses a launcher (matlab_root/bin/matlab.exe) and a main application (matlab_root/bin/xxx/matlab.exe). To keep the launcher open until the main application closes, you have to use the -wait option.