Related
On my local machine, I run a python script which contains this line
bashCommand = "cwm --rdf test.rdf --ntriples > test.nt"
os.system(bashCommand)
This works fine.
Then I run the same code on a server and I get the following error message
'import site' failed; use -v for traceback
Traceback (most recent call last):
File "/usr/bin/cwm", line 48, in <module>
from swap import diag
ImportError: No module named swap
So what I did then is I inserted a print bashCommand which prints me than the command in the terminal before it runs it with os.system().
Of course, I get again the error (caused by os.system(bashCommand)) but before that error it prints the command in the terminal. Then I just copied that output and did a copy paste into the terminal and hit enter and it works...
Does anyone have a clue what's going on?
Don't use os.system. It has been deprecated in favor of subprocess. From the docs: "This module intends to replace several older modules and functions: os.system, os.spawn".
Like in your case:
import subprocess
bashCommand = "cwm --rdf test.rdf --ntriples > test.nt"
process = subprocess.Popen(bashCommand.split(), stdout=subprocess.PIPE)
output, error = process.communicate()
To somewhat expand on the earlier answers here, there are a number of details which are commonly overlooked.
Prefer subprocess.run() over subprocess.check_call() and friends over subprocess.call() over subprocess.Popen() over os.system() over os.popen()
Understand and probably use text=True, aka universal_newlines=True.
Understand the meaning of shell=True or shell=False and how it changes quoting and the availability of shell conveniences.
Understand differences between sh and Bash
Understand how a subprocess is separate from its parent, and generally cannot change the parent.
Avoid running the Python interpreter as a subprocess of Python.
These topics are covered in some more detail below.
Prefer subprocess.run() or subprocess.check_call()
The subprocess.Popen() function is a low-level workhorse but it is tricky to use correctly and you end up copy/pasting multiple lines of code ... which conveniently already exist in the standard library as a set of higher-level wrapper functions for various purposes, which are presented in more detail in the following.
Here's a paragraph from the documentation:
The recommended approach to invoking subprocesses is to use the run() function for all use cases it can handle. For more advanced use cases, the underlying Popen interface can be used directly.
Unfortunately, the availability of these wrapper functions differs between Python versions.
subprocess.run() was officially introduced in Python 3.5. It is meant to replace all of the following.
subprocess.check_output() was introduced in Python 2.7 / 3.1. It is basically equivalent to subprocess.run(..., check=True, stdout=subprocess.PIPE).stdout
subprocess.check_call() was introduced in Python 2.5. It is basically equivalent to subprocess.run(..., check=True)
subprocess.call() was introduced in Python 2.4 in the original subprocess module (PEP-324). It is basically equivalent to subprocess.run(...).returncode
High-level API vs subprocess.Popen()
The refactored and extended subprocess.run() is more logical and more versatile than the older legacy functions it replaces. It returns a CompletedProcess object which has various methods which allow you to retrieve the exit status, the standard output, and a few other results and status indicators from the finished subprocess.
subprocess.run() is the way to go if you simply need a program to run and return control to Python. For more involved scenarios (background processes, perhaps with interactive I/O with the Python parent program) you still need to use subprocess.Popen() and take care of all the plumbing yourself. This requires a fairly intricate understanding of all the moving parts and should not be undertaken lightly. The simpler Popen object represents the (possibly still-running) process which needs to be managed from your code for the remainder of the lifetime of the subprocess.
It should perhaps be emphasized that just subprocess.Popen() merely creates a process. If you leave it at that, you have a subprocess running concurrently alongside with Python, so a "background" process. If it doesn't need to do input or output or otherwise coordinate with you, it can do useful work in parallel with your Python program.
Avoid os.system() and os.popen()
Since time eternal (well, since Python 2.5) the os module documentation has contained the recommendation to prefer subprocess over os.system():
The subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function.
The problems with system() are that it's obviously system-dependent and doesn't offer ways to interact with the subprocess. It simply runs, with standard output and standard error outside of Python's reach. The only information Python receives back is the exit status of the command (zero means success, though the meaning of non-zero values is also somewhat system-dependent).
PEP-324 (which was already mentioned above) contains a more detailed rationale for why os.system is problematic and how subprocess attempts to solve those issues.
os.popen() used to be even more strongly discouraged:
Deprecated since version 2.6: This function is obsolete. Use the subprocess module.
However, since sometime in Python 3, it has been reimplemented to simply use subprocess, and redirects to the subprocess.Popen() documentation for details.
Understand and usually use check=True
You'll also notice that subprocess.call() has many of the same limitations as os.system(). In regular use, you should generally check whether the process finished successfully, which subprocess.check_call() and subprocess.check_output() do (where the latter also returns the standard output of the finished subprocess). Similarly, you should usually use check=True with subprocess.run() unless you specifically need to allow the subprocess to return an error status.
In practice, with check=True or subprocess.check_*, Python will throw a CalledProcessError exception if the subprocess returns a nonzero exit status.
A common error with subprocess.run() is to omit check=True and be surprised when downstream code fails if the subprocess failed.
On the other hand, a common problem with check_call() and check_output() was that users who blindly used these functions were surprised when the exception was raised e.g. when grep did not find a match. (You should probably replace grep with native Python code anyway, as outlined below.)
All things counted, you need to understand how shell commands return an exit code, and under what conditions they will return a non-zero (error) exit code, and make a conscious decision how exactly it should be handled.
Understand and probably use text=True aka universal_newlines=True
Since Python 3, strings internal to Python are Unicode strings. But there is no guarantee that a subprocess generates Unicode output, or strings at all.
(If the differences are not immediately obvious, Ned Batchelder's Pragmatic Unicode is recommended, if not outright obligatory, reading. There is a 36-minute video presentation behind the link if you prefer, though reading the page yourself will probably take significantly less time.)
Deep down, Python has to fetch a bytes buffer and interpret it somehow. If it contains a blob of binary data, it shouldn't be decoded into a Unicode string, because that's error-prone and bug-inducing behavior - precisely the sort of pesky behavior which riddled many Python 2 scripts, before there was a way to properly distinguish between encoded text and binary data.
With text=True, you tell Python that you, in fact, expect back textual data in the system's default encoding, and that it should be decoded into a Python (Unicode) string to the best of Python's ability (usually UTF-8 on any moderately up to date system, except perhaps Windows?)
If that's not what you request back, Python will just give you bytes strings in the stdout and stderr strings. Maybe at some later point you do know that they were text strings after all, and you know their encoding. Then, you can decode them.
normal = subprocess.run([external, arg],
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
check=True,
text=True)
print(normal.stdout)
convoluted = subprocess.run([external, arg],
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
check=True)
# You have to know (or guess) the encoding
print(convoluted.stdout.decode('utf-8'))
Python 3.7 introduced the shorter and more descriptive and understandable alias text for the keyword argument which was previously somewhat misleadingly called universal_newlines.
Understand shell=True vs shell=False
With shell=True you pass a single string to your shell, and the shell takes it from there.
With shell=False you pass a list of arguments to the OS, bypassing the shell.
When you don't have a shell, you save a process and get rid of a fairly substantial amount of hidden complexity, which may or may not harbor bugs or even security problems.
On the other hand, when you don't have a shell, you don't have redirection, wildcard expansion, job control, and a large number of other shell features.
A common mistake is to use shell=True and then still pass Python a list of tokens, or vice versa. This happens to work in some cases, but is really ill-defined and could break in interesting ways.
# XXX AVOID THIS BUG
buggy = subprocess.run('dig +short stackoverflow.com')
# XXX AVOID THIS BUG TOO
broken = subprocess.run(['dig', '+short', 'stackoverflow.com'],
shell=True)
# XXX DEFINITELY AVOID THIS
pathological = subprocess.run(['dig +short stackoverflow.com'],
shell=True)
correct = subprocess.run(['dig', '+short', 'stackoverflow.com'],
# Probably don't forget these, too
check=True, text=True)
# XXX Probably better avoid shell=True
# but this is nominally correct
fixed_but_fugly = subprocess.run('dig +short stackoverflow.com',
shell=True,
# Probably don't forget these, too
check=True, text=True)
The common retort "but it works for me" is not a useful rebuttal unless you understand exactly under what circumstances it could stop working.
To briefly recap, correct usage looks like
subprocess.run("string for 'the shell' to parse", shell=True)
# or
subprocess.run(["list", "of", "tokenized strings"]) # shell=False
If you want to avoid the shell but are too lazy or unsure of how to parse a string into a list of tokens, notice that shlex.split() can do this for you.
subprocess.run(shlex.split("no string for 'the shell' to parse")) # shell=False
# equivalent to
# subprocess.run(["no", "string", "for", "the shell", "to", "parse"])
The regular split() will not work here, because it doesn't preserve quoting. In the example above, notice how "the shell" is a single string.
Refactoring Example
Very often, the features of the shell can be replaced with native Python code. Simple Awk or sed scripts should probably just be translated to Python instead.
To partially illustrate this, here is a typical but slightly silly example which involves many shell features.
cmd = '''while read -r x;
do ping -c 3 "$x" | grep 'min/avg/max'
done <hosts.txt'''
# Trivial but horrible
results = subprocess.run(
cmd, shell=True, universal_newlines=True, check=True)
print(results.stdout)
# Reimplement with shell=False
with open('hosts.txt') as hosts:
for host in hosts:
host = host.rstrip('\n') # drop newline
ping = subprocess.run(
['ping', '-c', '3', host],
text=True,
stdout=subprocess.PIPE,
check=True)
for line in ping.stdout.split('\n'):
if 'min/avg/max' in line:
print('{}: {}'.format(host, line))
Some things to note here:
With shell=False you don't need the quoting that the shell requires around strings. Putting quotes anyway is probably an error.
It often makes sense to run as little code as possible in a subprocess. This gives you more control over execution from within your Python code.
Having said that, complex shell pipelines are tedious and sometimes challenging to reimplement in Python.
The refactored code also illustrates just how much the shell really does for you with a very terse syntax -- for better or for worse. Python says explicit is better than implicit but the Python code is rather verbose and arguably looks more complex than this really is. On the other hand, it offers a number of points where you can grab control in the middle of something else, as trivially exemplified by the enhancement that we can easily include the host name along with the shell command output. (This is by no means challenging to do in the shell, either, but at the expense of yet another diversion and perhaps another process.)
Common Shell Constructs
For completeness, here are brief explanations of some of these shell features, and some notes on how they can perhaps be replaced with native Python facilities.
Globbing aka wildcard expansion can be replaced with glob.glob() or very often with simple Python string comparisons like for file in os.listdir('.'): if not file.endswith('.png'): continue. Bash has various other expansion facilities like .{png,jpg} brace expansion and {1..100} as well as tilde expansion (~ expands to your home directory, and more generally ~account to the home directory of another user)
Shell variables like $SHELL or $my_exported_var can sometimes simply be replaced with Python variables. Exported shell variables are available as e.g. os.environ['SHELL'] (the meaning of export is to make the variable available to subprocesses -- a variable which is not available to subprocesses will obviously not be available to Python running as a subprocess of the shell, or vice versa. The env= keyword argument to subprocess methods allows you to define the environment of the subprocess as a dictionary, so that's one way to make a Python variable visible to a subprocess). With shell=False you will need to understand how to remove any quotes; for example, cd "$HOME" is equivalent to os.chdir(os.environ['HOME']) without quotes around the directory name. (Very often cd is not useful or necessary anyway, and many beginners omit the double quotes around the variable and get away with it until one day ...)
Redirection allows you to read from a file as your standard input, and write your standard output to a file. grep 'foo' <inputfile >outputfile opens outputfile for writing and inputfile for reading, and passes its contents as standard input to grep, whose standard output then lands in outputfile. This is not generally hard to replace with native Python code.
Pipelines are a form of redirection. echo foo | nl runs two subprocesses, where the standard output of echo is the standard input of nl (on the OS level, in Unix-like systems, this is a single file handle). If you cannot replace one or both ends of the pipeline with native Python code, perhaps think about using a shell after all, especially if the pipeline has more than two or three processes (though look at the pipes module in the Python standard library or a number of more modern and versatile third-party competitors).
Job control lets you interrupt jobs, run them in the background, return them to the foreground, etc. The basic Unix signals to stop and continue a process are of course available from Python, too. But jobs are a higher-level abstraction in the shell which involve process groups etc which you have to understand if you want to do something like this from Python.
Quoting in the shell is potentially confusing until you understand that everything is basically a string. So ls -l / is equivalent to 'ls' '-l' '/' but the quoting around literals is completely optional. Unquoted strings which contain shell metacharacters undergo parameter expansion, whitespace tokenization and wildcard expansion; double quotes prevent whitespace tokenization and wildcard expansion but allow parameter expansions (variable substitution, command substitution, and backslash processing). This is simple in theory but can get bewildering, especially when there are several layers of interpretation (a remote shell command, for example).
Understand differences between sh and Bash
subprocess runs your shell commands with /bin/sh unless you specifically request otherwise (except of course on Windows, where it uses the value of the COMSPEC variable). This means that various Bash-only features like arrays, [[ etc are not available.
If you need to use Bash-only syntax, you can
pass in the path to the shell as executable='/bin/bash' (where of course if your Bash is installed somewhere else, you need to adjust the path).
subprocess.run('''
# This for loop syntax is Bash only
for((i=1;i<=$#;i++)); do
# Arrays are Bash-only
array[i]+=123
done''',
shell=True, check=True,
executable='/bin/bash')
A subprocess is separate from its parent, and cannot change it
A somewhat common mistake is doing something like
subprocess.run('cd /tmp', shell=True)
subprocess.run('pwd', shell=True) # Oops, doesn't print /tmp
The same thing will happen if the first subprocess tries to set an environment variable, which of course will have disappeared when you run another subprocess, etc.
A child process runs completely separate from Python, and when it finishes, Python has no idea what it did (apart from the vague indicators that it can infer from the exit status and output from the child process). A child generally cannot change the parent's environment; it cannot set a variable, change the working directory, or, in so many words, communicate with its parent without cooperation from the parent.
The immediate fix in this particular case is to run both commands in a single subprocess;
subprocess.run('cd /tmp; pwd', shell=True)
though obviously this particular use case isn't very useful; instead, use the cwd keyword argument, or simply os.chdir() before running the subprocess. Similarly, for setting a variable, you can manipulate the environment of the current process (and thus also its children) via
os.environ['foo'] = 'bar'
or pass an environment setting to a child process with
subprocess.run('echo "$foo"', shell=True, env={'foo': 'bar'})
(not to mention the obvious refactoring subprocess.run(['echo', 'bar']); but echo is a poor example of something to run in a subprocess in the first place, of course).
Don't run Python from Python
This is slightly dubious advice; there are certainly situations where it does make sense or is even an absolute requirement to run the Python interpreter as a subprocess from a Python script. But very frequently, the correct approach is simply to import the other Python module into your calling script and call its functions directly.
If the other Python script is under your control, and it isn't a module, consider turning it into one. (This answer is too long already so I will not delve into details here.)
If you need parallelism, you can run Python functions in subprocesses with the multiprocessing module. There is also threading which runs multiple tasks in a single process (which is more lightweight and gives you more control, but also more constrained in that threads within a process are tightly coupled, and bound to a single GIL.)
Call it with subprocess
import subprocess
subprocess.Popen("cwm --rdf test.rdf --ntriples > test.nt")
The error you are getting seems to be because there is no swap module on the server, you should install swap on the server then run the script again
It is possible you use the bash program, with the parameter -c for execute the commands:
bashCommand = "cwm --rdf test.rdf --ntriples > test.nt"
output = subprocess.check_output(['bash','-c', bashCommand])
You can use subprocess, but I always felt that it was not a 'Pythonic' way of doing it. So I created Sultan (shameless plug) that makes it easy to run command line functions.
https://github.com/aeroxis/sultan
Also you can use 'os.popen'.
Example:
import os
command = os.popen('ls -al')
print(command.read())
print(command.close())
Output:
total 16
drwxr-xr-x 2 root root 4096 ago 13 21:53 .
drwxr-xr-x 4 root root 4096 ago 13 01:50 ..
-rw-r--r-- 1 root root 1278 ago 13 21:12 bot.py
-rw-r--r-- 1 root root 77 ago 13 21:53 test.py
None
According to the error you are missing a package named swap on the server. This /usr/bin/cwm requires it. If you're on Ubuntu/Debian, install python-swap using aptitude.
To run the command without a shell, pass the command as a list and implement the redirection in Python using [subprocess]:
#!/usr/bin/env python
import subprocess
with open('test.nt', 'wb', 0) as file:
subprocess.check_call("cwm --rdf test.rdf --ntriples".split(),
stdout=file)
Note: no > test.nt at the end. stdout=file implements the redirection.
To run the command using the shell in Python, pass the command as a string and enable shell=True:
#!/usr/bin/env python
import subprocess
subprocess.check_call("cwm --rdf test.rdf --ntriples > test.nt",
shell=True)
Here's the shell is responsible for the output redirection (> test.nt is in the command).
To run a bash command that uses bashisms, specify the bash executable explicitly e.g., to emulate bash process substitution:
#!/usr/bin/env python
import subprocess
subprocess.check_call('program <(command) <(another-command)',
shell=True, executable='/bin/bash')
copy paste this:
def run_bash_command(cmd: str) -> Any:
import subprocess
process = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE)
output, error = process.communicate()
if error:
raise Exception(error)
else:
return output
subprocess.Popen() is prefered over os.system() as it offers more control and visibility. However, If you find subprocess.Popen() too verbose or complex, peasyshell is a small wrapper I wrote above it, which makes it easy to interact with bash from Python.
https://github.com/davidohana/peasyshell
The pythonic way of doing this is using subprocess.Popen
subprocess.Popen takes a list where the first element is the command to be run followed by any command line arguments.
As an example:
import subprocess
args = ['echo', 'Hello!']
subprocess.Popen(args) // same as running `echo Hello!` on cmd line
args2 = ['echo', '-v', '"Hello Again"']
subprocess.Popen(args2) // same as running 'echo -v "Hello Again!"` on cmd line
I'm apprenticing into system administration without schooling, so sometimes I'm missing what is elementary information to many others.
I'm attempting to give my stdout line another argument before printing, but I'm not sure which process I should use, and I'm a bit fuzzy on the commands for subprocess if that's what I should be using.
My current code is:
f = open('filelist', 'r')
searchterm = f.readline()
f.close()|
#takes line from a separate file and gives it definition so that it may be callable.
import commands
commands.getoutput('print man searchterm')
This is running, but not giving me an ouput to the shell. My more important question is though, am I using the right command to get my preferred process? Should I be using one of the subprocess commands instead? I tried playing around with popen, but I don't understand it fully enough to use it correctly.
Ie, I was running
subprocess.Popen('print man searchterm')
but I know without a doubt that's not how you're supposed to run it. Popen requires more arguments than I have given it, like file location and where to run it (Stdout or stderr). But I was having trouble making these commands work. Would it be something like:
subprocess.Popen(pipe=stdout 'man' 'searchterm')
#am unsure how to give the program my arguments here.
I've been researching everywhere, but it is such a widely used process I seem to be suffering from a surplus of information rather than not enough. Any help would be appreciated, I'm quite new.
Preemptive thanks for any help.
The cannonical way to get data from a separate process is to use subprocess (commands is deprecated)
import subprocess
p = subprocess.Popen(['print','man','searchitem'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdoutdata, stderrdata = p.communicate()
Note that some convenience functions exist for splitting strings into lists of arguments. Most notably is shlex.split which will take a string and split it into a list the same way a shell does. (If nothing is quoted in the string, str.split() works just as well).
commands is deprecated in Python 2.6 and later, and has been removed in Python 3. There's probably no situation where it's preferable in new code, even if you are stuck with Python 2.5 or earlier.
From the docs:
Deprecated since version 2.6: The commands module has been removed in
Python 3. Use the subprocess module instead.
To run man searchterm in a separate process and display the result in the terminal, you could do this:
import subprocess
proc = subprocess.Popen('man searchterm'.split())
proc.communicate()
I'm writing a little python IDE, and I want to add simple debugging. I don't need all the features of winpdb.
How do I launch a python program (by file name) with a breakpoint set at a line number so that it runs until that line number and halts?
Note that I don't want to do this from the command-line, and I don't want to edit the source (by inserting set_trace, for example). And I don't want it to stop at the first line so I have to run the debugger from there. I've tried all the obvious ways with pdb and bdb, but I must be missing something.
Pretty much the only viable way to do it (as far as I know) is to run Python as a subprocess from within your IDE. This avoids "pollution" from the current Python interpreter, which makes it fairly likely that the program will run in the same way as if you had started it independently. (If you have issues with this, check the subprocess environment.) In this manner, you can run a script in "debug mode" using
p = subprocess.Popen(args=[sys.executable, '-m', 'pdb', 'scriptname.py', 'arg1'],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
This will start up Python at the debugger prompt. You'll need to run some debugger commands to set breakpoints, which you can do like so:
o,e = p.communicate('break scriptname.py:lineno')
If this works, o should be the normal output of the Python interpreter after it sets a breakpoint, and e should be empty. I'd suggest you play around with this and add some checks in your code to ensure whether the breakpoints were properly set.
After that, you can start the program running with
p.communicate('continue')
At this point you'd probably want to hook the input, output, and error streams up to the console that you're embedding in your IDE. You would probably need to do this with an event loop, roughly like so:
while p.returncode is None:
o,e = p.communicate(console.read())
console.write(o)
console.write(e)
You should consider that snippet to be effectively pseudocode, since depending on how exactly your console works, it'll probably take some tinkering to get it right.
If this seems excessively messy, you can probably simplify the process a bit using the features of Python's pdb and bdb modules (I'm guessing "Python debugger" and basic debugger" respectively). The best reference on how to do this is the source code of the pdb module itself. Basically, the way the responsibilities of the modules are split is that bdb handles "under the hood" debugger functionality, like setting breakpoints, or stopping and restarting execution; pdb is a wrapper around this that handles user interaction, i.e. reading commands and displaying output.
For your IDE-integrated debugger, it would make sense to adjust the behavior of the pdb module in two ways that I can think of:
have it automatically set breakpoints during initialization, without you having to explicity send the textual commands to do so
make it take input from and send output to your IDE's console
Just these two changes should be easy to implement by subclassing pdb.Pdb. You can create a subclass whose initializer takes a list of breakpoints as an additional argument:
class MyPDB(pdb.Pdb):
def __init__(self, breakpoints, completekey='tab',
stdin=None, stdout=None, skip=None):
pdb.Pdb.__init__(self, completekey, stdin, stdout, skip)
self._breakpoints = breakpoints
The logical place to actually set up the breakpoints is just after the debugger reads its .pdbrc file, which occurs in the pdb.Pdb.setup method. To perform the actual setup, use the set_break method inherited from bdb.Bdb:
def setInitialBreakpoints(self):
_breakpoints = self._breakpoints
self._breakpoints = None # to avoid setting breaks twice
for bp in _breakpoints:
self.set_break(filename=bp.filename, line=bp.line,
temporary=bp.temporary, conditional=bp.conditional,
funcname=bp.funcname)
def setup(self, f, t):
pdb.Pdb.setup(self, f, t)
self.setInitialBreakpoints()
This piece of code would work for each breakpoint being passed as e.g. a named tuple. You could also experiment with just constructing bdb.Breakpoint instances directly, but I'm not sure if that would work properly, since bdb.Bdb maintains its own information about breakpoints.
Next, you'll need to create a new main method for your module which runs it the same way pdb runs. To some extent, you can copy the main method from pdb (and the if __name__ == '__main__' statement of course), but you'll need to augment it with some way to pass in the information about your additional breakpoints. What I'd suggest is writing the breakpoints to a temporary file from your IDE, and passing the name of that file as a second argument:
tmpfilename = ...
# write breakpoint info
p = subprocess.Popen(args=[sys.executable, '-m', 'mypdb', tmpfilename, ...], ...)
# delete the temporary file
Then in mypdb.main(), you would add something like this:
def main():
# code excerpted from pdb.main()
...
del sys.argv[0]
# add this
bpfilename = sys.argv[0]
with open(bpfilename) as f:
# read breakpoint info
breakpoints = ...
del sys.argv[0]
# back to excerpt from pdb.main()
sys.path[0] = os.path.dirname(mainpyfile)
pdb = Pdb(breakpoints) # modified
Now you can use your new debugger module just like you would use pdb, except that you don't have to explicitly send break commands before the process starts. This has the advantage that you can directly hook the standard input and output of the Python subprocess to your console, if it allows you to do that.
I just started working on Python, and I have been trying to run an outside executable from Python.
I have an executable for a program written in Fortran. Let’s say the name for the executable is flow.exe. And my executable is located in C:\Documents and Settings\flow_model. I tried both os.system and popen commands, but so far I couldn't make it work. The following code seems like it opens the command window, but it wouldn't execute the model.
# Import system modules
import sys, string, os, arcgisscripting
os.system("C:/Documents and Settings/flow_model/flow.exe")
How can I fix this?
If using Python 2.7 or higher (especially prior to Python 3.5) you can use the following:
import subprocess
subprocess.call(args, *, stdin=None, stdout=None, stderr=None, shell=False)
Runs the command described by args. Waits for command to complete, then returns the returncode attribute.
subprocess.check_call(args, *, stdin=None, stdout=None, stderr=None, shell=False)
Runs command with arguments. Waits for command to complete. If the return code was zero then returns, otherwise raises CalledProcessError. The CalledProcessError object will have the return code in the returncode attribute
Example: subprocess.check_call([r"C:\pathToYourProgram\yourProgram.exe", "your", "arguments", "comma", "separated"])
In regular Python strings, the \U character combination signals a
extended Unicode code point escape.
Here is the link to the documentation: http://docs.python.org/3.2/library/subprocess.html
For Python 3.5+ you can now use run() in many cases: https://docs.python.org/3.5/library/subprocess.html#subprocess.run
Those whitespaces can really be a bother. Try os.chdir('C:/Documents\ and\ Settings/') followed by relative paths for os.system, subprocess methods, or whatever...
If best-effort attempts to bypass the whitespaces-in-path hurdle keep failing, then my next best suggestion is to avoid having blanks in your crucial paths. Couldn't you make a blanks-less directory, copy the crucial .exe file there, and try that? Are those havoc-wrecking space absolutely essential to your well-being...?
The simplest way is:
import os
os.startfile("C:\Documents and Settings\flow_model\flow.exe")
It works; I tried it.
I'd try inserting an 'r' in front of your path if I were you, to indicate that it's a raw string - and then you won't have to use forward slashes. For example:
os.system(r"C:\Documents and Settings\flow_model\flow.exe")
Your usage is correct. I bet that your external program, flow.exe, needs to be executed in its directory, because it accesses some external files stored there.
So you might try:
import sys, string, os, arcgisscripting
os.chdir('c:\\documents and settings\\flow_model')
os.system('"C:\\Documents and Settings\\flow_model\\flow.exe"')
(Beware of the double quotes inside the single quotes...)
Use subprocess, it is a smaller module so it runs the .exe quicker.
import subprocess
subprocess.Popen([r"U:\Year 8\kerbal space program\KSP.exe"])
By using os.system:
import os
os.system(r'"C:/Documents and Settings/flow_model/flow.exe"')
Try
import subprocess
subprocess.call(["C:/Documents and Settings/flow_model/flow.exe"])
If it were me, I'd put the EXE file in the root directory (C:) and see if it works like that. If so, it's probably the (already mentioned) spaces in the directory name. If not, it may be some environment variables.
Also, try to check you stderr (using an earlier answer by int3):
import subprocess
process = subprocess.Popen(["C:/Documents and Settings/flow_model/flow.exe"], \
stderr = subprocess.PIPE)
if process.stderr:
print process.stderr.readlines()
The code might not be entirely correct as I usually don't use Popen or Windows, but should give the idea. It might well be that the error message is on the error stream.
in python 2.6 use string enclosed inside quotation " and apostrophe ' marks. Also a change single / to double //.
Your working example will look like this:
import os
os.system("'C://Documents and Settings//flow_model//flow.exe'")
Also You can use any parameters if Your program ingest them.
os.system('C://"Program Files (x86)"//Maxima-gcl-5.37.3//gnuplot//bin//gnuplot -e "plot [-10:10] sin(x),atan(x),cos(atan(x)); pause mouse"')
finally You can use string variable, as an example is plotting using gnuplot directly from python:
this_program='C://"Program Files (x86)"//Maxima-gcl-5.37.3//gnuplot//bin//gnuplot'
this_par='-e "set polar; plot [-2*pi:2*pi] [-3:3] [-3:3] t*sin(t); pause -1"'
os.system(this_program+" "+this_par)
import os
path = "C:/Documents and Settings/flow_model/"
os.chdir(path)
os.system("flow.exe")
Note added by barlop
A commenter asked why this works. Here is why.
The OP's problem is os.system("...") doesn't work properly when there is a space in the path. (Note os.system can work with ('"...."') but anyhow)
Had the OP tried their program from a cmd prompt they'd have seen the error clearly.
C:\carp>type blah.py
import os
os.system(R"C:\Program Files (x86)\Google\Chrome\Application\chrome.exe")
C:\carp>python blah.py
'C:\Program' is not recognized as an internal or external command,
operable program or batch file.
C:\carp>
So it's fine for os.system("calc.exe") (there calc.exe is in the path environment variable). Or for os.system(R"c:\windows\system32\calc.exe"). There's no space in that path.
C:\>md "aa bb cc"
C:\>copy c:\windows\system32\calc.exe "c:\aa bb cc\cccalc.exe"
1 file(s) copied.
This works (Given file "c:\aa bb cc\cccalc.exe" )
import os
os.chdir(R"c:\aa bb cc")
os.system("cccalc.exe")
Other options are subprocess.run and subprocess.popen.
Is that trying to execute C:\Documents with arguments of "and", "Settings/flow_model/flow.exe"?
Also, you might consider subprocess.call().
There are loads of different solutions, and the results will strongly depend on:
the OS you are using: Windows, Cygwin, Linux, MacOS
the python version you are using: Python2 or Python3x
As I have discovered some things that are claimed to work only in Windows, doesn't, probably because I happen to use Cygwin which is outsmarting the OS way to deal with Windows paths. Other things only work in pure *nix based OS's or in Python2 or 3.
Here are my findings:
Generally speaking, os.system() is the most forgiving method.
os.startfile() is the least forgiving. (Windows only && if you're lucky)
subprocess.Popen([...]) not recommended
subprocess.run(winView, shell=True) the recommended way!
Remembering that using subprocess for anything may pose a security risk.
Try these:
import os, subprocess
...
winView = '/cygdrive/c/Windows/explorer.exe %s' % somefile
...
# chose one of these:
os.system(winView)
subprocess.Popen(['/cygdrive/c/Windows/explorer.exe', 'somefile.png'])
subprocess.run(winView, shell=True)
Q: Why would you want to use explorer in Windows?
A: Because if you just want to look at the results of some new file, explorer will automatically open the file with whatever default windows program you have set for that file type. So no need to re-specify the default program to use.
That's the correct usage, but perhaps the spaces in the path name are messing things up for some reason.
You may want to run the program under cmd.exe as well so you can see any output from flow.exe that might be indicating an error.
for the above question this solution works.
just change the path to where your executable file is located.
import sys, string, os
os.chdir('C:\\Downloads\\xpdf-tools-win-4.00\\xpdf-tools-win-4.00\\bin64')
os.system("C:\\Downloads\\xpdf-tools-win-4.00\\xpdf-tools-win-4.00\bin64\\flowwork.exe")
'''import sys, string, os
os.chdir('C:\\Downloads\\xpdf-tools-win-4.00\\xpdf-tools-win-4.00\\bin64')
os.system(r"C:\\Downloads\\xpdf-tools-win-4.00\\xpdf-tools-win-4.00\bin64\\pdftopng.exe test1.pdf rootimage")'''
Here test1.pdf rootimage is for my code .
If I have a program where stdout is redirected, my pdb prompts all go to the redirection, because the library was written to write to stdout.
Oftentimes this problem is subtle, causing me to think a program is hanging when it's really waiting for input.
How do people work around this? (Unfortunately, using other debuggers like winpdb is not an option).
This answer is just to supplement Ned's, as a way of wrapping the pdb.py main() function in a manner which doesn't require copying 40 lines just to change one of them:
# sane_pdb.py: launch Pdb with stdout on original
import sys, pdb
def fixed_pdb(Pdb=pdb.Pdb):
'''make Pdb() tied to original stdout'''
return Pdb(stdout=sys.__stdout__)
if __name__ == '__main__':
pdb.Pdb = fixed_pdb
pdb.main()
I don't know if it actually works for the questioner's problem, but it does what Ned described...
The problem here is that PDB uses Cmd class where by default:
use_rawinput = 1
It means that Cmd will use raw_input() method by default instead of sys.stdout.readline() to read from the console . This is done because raw_input() supports history (only if readline module is loaded) and other useful bits . The only issue is that raw_input() does not support redirection, so if you have a script:
#!/usr/bin/python
name=raw_input("Enter your name: ")
and run it
> python test.py
Enter your name: Alex
but, if you run it with output redirection it will stuck
> python test.py | tee log
this is exactly what PDB uses and why it's stuck as well. As I mentioned sys.stdin.readline() supports redirection and if you rewrite the above script using readline() it should work.
Back to the original issue all you need to do is to tell Cmd to not use raw_input():
Cmd.use_rawinput = 0
or
pdb = pdb.Pdb()
pdb.use_rawinput=0
pdb.set_trace()
If you are invoking pdb in code, you can pass your own stdout into the constructor. sys.__stdout__ might be a good choice.
If you are invoking pdb from the command line, you could copy the main() function from pdb.py into your own sane_pdb.py. Then change the Pdb() initialization to:
pdb = Pdb(stdout=sys.__stdout__)
Then you can invoke sane_pdb.py instead of pdb.py. It's not awesome that you'd have to copy 40 lines into your own file just to change one of them, but it's an option.