I have one piece of Cocoa code I wrote that takes in an XML file containing bounding boxes that are then drawn on top of a video (each box has an associated frame). The Cocoa program is meant to be run from the command line (and takes in all its parameters as command line arguments)
I can run program just fine with any XML document. However, I run into problems when I try to run the program from within a Python script. For example:
with file("test.xml") as temp:
temp.write(doc.toprettyxml())
# cval is my cocoa program to call, the other arguments are given to the Python script and parsed with optparser
command = ["./cval", "-o", options.output, "-i", str(options.interval), "-s", "%dx%d" % (options.width, options.height), "-f", str(options.frames), "-x", temp.name]
subprocess.call(command)
Sometimes this will cause my 'cval' to fail, other times not (changing one number in the XML document can change its behavior). I can also verify it's breaking when trying to read an XML element that isn't there. Only, I can open up 'test.xml', and verify the element does in fact exist.
However, if I then run 'cval' myself (outside of the Python script) with 'test.xml', it works fine. This leads me to believe that there is something strange happening when I do 'subprocess.call', but I'm not sure what it could be. I have other Cocoa/Python mixes that do completely different tasks (i.e. not using XML) that also arbitrarily exhibit weird behavior, but are more complex in nature.
I was hoping someone might have run into this problem as well, or might know the next step in debugging this weirdness.
Because the code originally used temporary files, I couldn't close the file before passing it to the subprocess. However, what I should have done instead is to flush the file before subprocess.call was invoked. The inconsistent behavior likely resulted from the size of input causing automatic flushing at different thresholds.
The code should read:
with file("test.xml") as temp:
temp.write(doc.toprettyxml())
temp.flush()
command = ["./cval", "-o", options.output, "-i", str(options.interval), "-s", "%dx%d" % (options.width, options.height), "-f", str(options.frames), "-x", temp.name]
subprocess.call(command)
Perhaps try placing a "print command" statement in there, when the return code of subprocess.call indicates an error. On failure, see if there's any difference between what's being executed by subprocess and what you might run from the command line. Also, try calling subprocess.call(command, shell=True), so your command is being executed as it would in the shell (with string formatting, etc).
Related
I'm struggling to get some python script to start a subprocess, wait until it completes and then retrieve the required data. I'm quite new to Python.
The command I wish to run as a subprocess is
./bin.testing/Eva -t --suite="temp0"
Running that command by hand in the Linux terminal produces:
in terminal mode
Evaluation error = 16.7934
I want to run the command as a python sub-process, and receive the output back. However, everything I try seems to skip the second line (ultimately, it's the second line that I want.) At the moment, I have this:
def job(self,fen_file):
from subprocess import Popen, PIPE
from sys import exit
try:
eva=Popen('{0}/Eva -t --suite"{0}"'.format(self.exedir,fen_file),shell=True,stdout=PIPE,stderr=PIPE)
stdout,stderr=eva.communicate()
except:
print ('Error running test suite '+fen_file)
exit("Stopping")
print(stdout)
.
.
.
return 0
All this seems to produce is
in terminal mode
0
with the important line missing. The print statement is just so I can see what I am getting back from the sub-process -- the intention is that it will be replaced with code that processes the number from the second line and returns the output (here I'm just returning 0 just so I can get this particular bit to work first. The caller of this function prints the result, which is why there is a zero at the end of the output.) exedir is just the directory of the executable for the sub-process, and fen-file is just an ascii file that the sub-process needs. I have tried removing the 'in terminal mode' from the source code of the sub-process and re compiling it, but that doesn't work -- it still doesn't return the important second line.
Thanks in advance; I expect what I am doing wrong is really very simple.
Edit: I ought to add that the subprocess Eva can take a second or two to complete.
Since the 2nd line is an error message, it's probably stored in your stderr variable!
To know for sure you can print your stderr in your code, or you can run the program on the command line and see if the output is split into stdout and stderr. One easy way is to do ./bin.testing/Eva -t --suite="temp0" > /dev/null. Any messages you get are stderr since stdout is redirected to /dev/null.
Also, typically with Popen the shell=True option is discouraged unless really needed. Instead pass a list:
[os.path.join(self.exedir, 'Eva'), '-t', '--suite=' + fen_file], shell=False, ...
This can avoid problems down the line if one of your arguments would normally be interpreted by the shell. (Note, I removed the ""'s, because the shell would normally eat those for you!)
Try using subprocess check_output.
output_lines = subprocess.check_output(['./bin.testing/Eva', '-t', '--suite="temp0"'])
for line in output_lines.splitlines():
print(line)
I have a python 3.5 script running under Windows that is calling an external problem (tblastn from the BLAST+ suite to be precise) on a number of files. With most of these files it runs fine but on some it fails with return code 0xC0000005. If I take the exact same command line call and run it from the console in the same current working directory it executes fine.
I am currently running the command with subprocess.Popen, like this:
childProcess = subprocess.Popen(blast_cmd, stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
universal_newlines=True,shell=True)
and then calling subprocess.poll() until it completes. I am multi-threading this by running four processes simultaneously but it still happens if I force it to run one at a time. The same thing happens with os.system, subprocess.run(), subprocess.call() and subprocess.check_call() and it happens whether I set shell to True or False.
Which file(s) it fails on is/are the same each time I run the code but the same file will work if put into a different list of files to process. Changing the calling method sometimes changes which files fail so using os.system can cause different files to fail compared to subprocess.Popen. Thus it doesn't appear to be down to which file I am invoking tblastn on.
Does anyone have any idea what might be causing this behaviour?
Or if anyone knows what could be different between running in created process (the documentation says it uses CreateProcess()) as compared to running from the command line then at least I'd have somewhere to start?
the error code is likely to be "Access Denied" (although there are 4 code constructs in the windows header files, the Access Denied is the most likely:
# for hex 0xc0000005 / decimal -1073741819
FILE_LOG_INFORMATION_FAILED iasmsg.h
# Information for the %1 log could not be logged to the text
# file %2 in the path %3. Error code: %0
STATUS_ACCESS_VIOLATION ntstatus.h
# The instruction at 0x%08lx referenced memory at 0x%08lx.
# The memory could not be %s.
USBD_STATUS_DEV_NOT_RESPONDING usb.h
# as an HRESULT: Severity: FAILURE (1), FACILITY_NULL (0x0), Code 0x5
# for hex 0x5 / decimal 5
ERROR_ACCESS_DENIED winerror.h
# Access is denied.
I would start by looking at the user priviledges/credentials which are used to run the original (launching/parent) script, which the childprocess/subprocess inherits its credentials from... then compare this to the credentials which are used when you "run this on cmd prompt" as you have described.
HTH,
Edwin.
sub proc that are launched programmatically often get other memory settings (heap size etc.) than interactively launched procs. So try putting some heap/mem checking wrapper around tblastn.exe.
Your description of "failing on one file but processing in another file list works" shows, that the error is not related to the failing call itself but to some condition caused by the prior activities.
output is buffered in memory. If tblastn has a lot of output take communicate() to void (or use) the output of the sub procs.
shell=True is not needed for calling executables, it's meant to execute shell build-ins. Using this you wrap tblastn with a senseless cmd.exe.
I have been scripting with windows for many years and have only started to look at python as an alternative in the last few weeks. I'm trying to write a native python script to backup a mysql database with mysqldump. I normally do this with a command line piping the output > without issue.
I see many answers with subprocess.popen and shell=True, equally I see many statements say I should avoid shell=True
So I'm trying to get the following code to redirect my stdout to a file, all without success
sys.stdout=open("mysqldump.txt",'w')
print("testing line1")
subprocess.check_output(["mysqldump", "-u", "usernmae", "-ppassword", "-h", "dbserver_name", database_name])
If I comment out the sys.sdout line I see the sqldump outputting to my screen so I know I have the syntax correct for this part. I added the print statement and can see this gets written to the file mysqldump.txt. But when run in full there is no dump to the screen or the file
Any ideas? I'm trying to avoid using shell solution
What you tried to do doesn't work because modifying sys.stdout only affects Python-level statements such as print, not lower-level writes from C, and particularly not those performed by an external program. You want to tell subprocess to create a pipe, like you did with the > redirection, which goes like this:
with open("mysqldump.txt",'w') as out:
subprocess.check_call(["mysqldump", "-u", "usernmae", "-ppassword",
"-h", "dbserver_name", database_name],
stdout=out)
Could you use the --result-file=file argument for mysqldump?
might have to change check_output to subprocess.call for this to complete.
or
subprocess.call(["mysqldump", "-u", "usernmae", "-ppassword", "-h", "dbserver_name", database_name],stdout=open('myfile.txt','w'))
edit: myfile.txt will close after subprocess is done.
I am attempting to wrap a program that is routinely used at work. When called with an insufficient number of arguments, or with a misspelled argument, the program issues a prompt to the user, asking for the needed input. As a consequence, when calling the routine with subprocess.Popen, the routine never sends any information to stdout or stderr when wrong parameters are passed. subprocess.Popen.communicate() and subprocess.Popen.read(1) both wait for a newline character before any information becomes available.
Is there any way to retrieve information from subprocess.Popen.stdout before the newline character is issued? If not, is there any method that can be used to determine whether the subprocess is waiting for input?
First thing to try: use the bufsize argument to Popen, and set it to 0:
subprocess.Popen(args, bufsize=0, ...)
Unfortunately, whether or not this works also depends upon how the subprocess flushes its output, and I presume you don't have much control over that.
On some platforms, when data written to stdout is flushed will actually change depending on whether the underlying I/O library detects an interactive terminal or a pipe. So while you might think the data is there waiting to be read — because that's how it works in a terminal window — it might actually be line buffered when you're running the same program as a subprocess from another within Python.
Added: I just realised that bufsize=0 is the default anyway. Nuts.
After asking around quite a bit, someone pointed me to the solution. Use pexpect.spawn and pexpect.expect. For example:
Bash "script" in a file titled prompt.sh to emulate the problem - read cannot be called directly from pexpect.spawn.
#!/bin/bash
read -p "This is a prompt: "
This will hang when called by subprocess.Popen. It can be handled by pexpect.spawn, though:
import pexpect
child = pexpect.spawn('./prompt.sh')
child.expect(search)
>>> 0
print child.after #Prints the matched text
>>> 'This is a prompt: '
A list, compiled regex, or list of compiled regex can also be used in place of the string in pexpect.expect to deal with differing prompts.
I am running a sub-program using subprocess.popen. When I start my Python program from the command window (cmd.exe), the program writes some info and dates in the window as the program evolves.
When I run my Python code not in a command window, it opens a new command window for this sub-program's output, and I want to avoid that. When I used the following code, it doesn't show the cmd window, but it also doesn't print the status:
p = subprocess.Popen("c:/flow/flow.exe", shell=True, stdout=subprocess.PIPE)
print p.stdout.read()
How can I show the sub-program's output in my program's output as it occurs?
Use this:
cmd = subprocess.Popen(["c:/flow/flow.exe"], stdout=subprocess.PIPE)
for line in cmd.stdout:
print line.rstrip("\n")
cmd.wait() # you may already be handling this in your current code
Note that you will still have to wait for the sub-program to flush its stdout buffer (which is commonly buffered differently when not writing to a terminal window), so you may not see each line instantaneously as the sub-program prints it (this depends on various OS details and details of the sub-program).
Also notice how I've removed the shell=True and replaced the string argument with a list, which is generally recommended.
Looking for a recipe to process Popen data asynchronously I stumbled upon http://code.activestate.com/recipes/576759-subprocess-with-async-io-pipes-class/
This looks quite promising, however I got the impression that there might be some typos in it. Not tried it yet.
It is an old post, but a common problem with a hard to find solution. Try this: http://code.activestate.com/recipes/440554-module-to-allow-asynchronous-subprocess-use-on-win/