This question already has answers here:
Using greater than operator with subprocess.call
(2 answers)
Closed 7 years ago.
I want to redirect the output of python script to the file using greater than operator. I have below code which is not working properly. Can someone please help me on this?
proc= subprocess.Popen(['python', 'countmapper.py',file],cwd="C:\pythonPrograms\\",stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
countReducer= subprocess.Popen(['python', 'countreducer.py'],cwd="C:\pythonPrograms\\",stdout=subprocess.PIPE,stdin=proc.stdout, stderr=subprocess.STDOUT)
countpostprocesser= subprocess.Popen(['python','countpostprocesser.py','>','output.json'],cwd="C:\pythonPrograms\\",stdout=subprocess.PIPE,stdin=countReducer.stdout,stderr=subprocess.STDOUT)
'file' is name of the log file that I want to process. Last line (starting with countpostprocesser...) is failing.
Your call is failing because the redirection operator is being passed to your script as an argument, not being acted on by a shell to redirect your output to a file. See the Popen documentation.
This answer to another SO question shows a good example of opening a file, then redirecting the subprocess' output to the file.
Also, as shx2 mentioned in another answer, passing the shell=True argument to your Popen constructor should accomplish what you're looking for as well. It will cause the process to be opened in it's own shell, allowing the shell program to interpret the arguments you pass. Note an important line in the Popen documentation though: "If shell is True, it is recommended to pass args as a string rather than as a sequence."
Use the shell=True flag of Popen.
Also, as I mentioned in the comments, your task can be done simply and elegantly using plumbum.
Related
This is certainly answered as part of a long discussion about subprocess elsewhere. But the answer is so simple it should be broken out.
How do I pass a string "foo" to a program expecting it on stdin if I use Python 3's subprocess.run()?
Pass input="whatever string you want" and text=True to subprocess.run:
import subprocess
subprocess.run("cat", input="foo\n", text=True)
Per the docs for subprocess.run:
The input argument is passed to Popen.communicate() and thus to the subprocess’s stdin. If used it must be a byte sequence, or a string if encoding or errors is specified or text is true. When used, the internal Popen object is automatically created with stdin=PIPE, and the stdin argument may not be used as well.
To also get the output of the command as a string, add capture_output=True:
subprocess.run("cat", input="foo\n", capture_output=True, text=True)
Simplest possible example, send foo to cat and let it print to the screen.
import subprocess
subprocess.run(['cat'],input=b'foo\n')
Notice that you send binary data and the carriage return.
This question already has answers here:
How do I execute a program or call a system command?
(65 answers)
Closed 4 years ago.
I have a Python program, it generates a string, say fileName. In fileName I hold something like "video1.avi", or "sound1.wav". Then I make an os call to start a program ! program arg1 arg2, where my fileName is arg2. How can I achieve that on the fly, without making the whole program return a single string (the fileName) and then pass it to the shell line? How can I make that during execution. The script executes in Jupyter.
P.S. I am looping and changing the file name and I have to run that script at every loop.
If you want your script to run some outside program, passing in an argument, the way to do that is the subprocess module.
Exactly which function to call depends on what exactly do you want to do. Just start it in the background and ignore the result? Wait for it to finish, ignore any output, but check that it returned success? Collect output so you can log it? I'm going to pick one of the many options arbitrarily, but read the linked docs to see how to do whichever one you actually want.
for thingy in your_loop:
fileName = your_filename_creating_logic(thingy)
try:
subprocess.run(['program', 'arg1', fileName],
check=True)
print(f'program ran on {filename} successfully')
except subprocess.CalledProcessError as e:
print(f'program failed on {filename} with #{e.returncode}')
Notice that I'm passing a list of arguments, with the program name (or full path) as the first one. You can throw in hardcoded strings like arg1 or --breakfast=spam, and variables like fileName. Because it's a list of strings, not one big string, and because it's not going through the shell at all (at least on Mac and Linux; things are a bit more complicated on Windows, but mostly it "just works" anyway), I don't have to worry about quoting filename in case it has spaces or other funky characters.
If you're using Python 3.4 or 2.7, you won't have that run function; just change it to check_call (and without that check=True argument).
This question already has answers here:
How to set environment variables in Python?
(19 answers)
Closed 6 years ago.
The following code allows me to dynamically identify and load the custom module if it is not located in any of the directory of sys.path variable
import sys
sys.path.append("/lib")
But, this gives me OSError
import subprocess
x = subprocess.Popen(["export", "PYTHONPATH=/lib"], stdout=subprocess.PIPE)
Not just this, even simple Linux/Unix variable declaration setting fails in subprocess.Popen()
import subprocess
x = subprocess.Popen("x=y", stdout=subprocess.PIPE)
I wanted to check subprocess as I tried setting PYTHONPATH via os.system(), os.popen() etc., and the variable did not set (may be it is set in the child process shell)
Try this:
>>> subprocess.call(["export foo=bar && echo foo=$foo"], shell=True)
foo=bar
0
>>>
There are several things that are going on here and are probably confusing you a little. One thing is, that whatever instructions given to Popen will be executed in the child process and will not affect your main process. You can merely Pipe or retrieve results from it.
First to comment on your second use case, where you use string as an argument. From the docs you can read:
class subprocess.Popen(args, bufsize=-1, executable=None, stdin=None,
stdout=None, stderr=None, preexec_fn=None, close_fds=True,
shell=False, cwd=None, env=None, universal_newlines=False,
startupinfo=None, creationflags=0, restore_signals=True,
start_new_session=False, pass_fds=())
...
args should be a sequence of program arguments or else a single
string. By default, the program to execute is the first item in args
if args is a sequence. If args is a string, the interpretation is
platform-dependent and described below. See the shell and executable
arguments for additional differences from the default behavior. Unless
otherwise stated, it is recommended to pass args as a sequence.
On POSIX, if args is a string, the string is interpreted as the name
or path of the program to execute. However, this can only be done if
not passing arguments to the program.
So in your second case, you are trying to execute a file or program x=y which doesn't go.
Even if you use list, like in your first use case, you must be aware, that this isn't equivalent to passing code to the bash shell. If you want this, you can use shell=True as an keyword argument, but this has other issues as indicated by the docs. But your code will execute with shell=True.
If your sole purpose is to set environmental variable, then you should consider the option to use os.environ variable that maps your environmental variables to values (as indicated by #cdarke first).
I would like to call an external script from within a function, for instance:
import subprocess
def call_script(script):
subprocess.call(script)
return answer #retrieving the result is the part I'm struggling with
call_script("/user/bin/calc_delta.py")
The script calc_delta.py simply prints the result when it is done. How can I assign the result of that script to a variable? Thank you.
Instead of using subprocess.call you should use Popen and call communicate on it.
That will allow you to read stdout and stderr. You can also input data with stdin.
Example from the docs http://docs.python.org/library/subprocess.html#replacing-bin-sh-shell-backquote:
output = Popen(["mycmd", "myarg"], stdout=PIPE).communicate()[0]
You should look at subprocess.Popen (popen)
For instance (from the docs):
pipe = Popen("cmd", shell=True, bufsize=bufsize, stdout=PIPE).stdout
subprocess.call is only going to give you the response code for your call, not the output.
In Python ≥2.7 / ≥3.1, you could use subprocess.check_output to convert the stdout into a str.
answer = subprocess.check_output(['/usr/bin/calc_delta.py'])
There are also subprocess.getstatusoutput and subprocess.getoutput on Python ≥3.1 if what you need is a shell command, but it is supported on *NIX only.
This question already has answers here:
Store output of subprocess.Popen call in a string [duplicate]
(15 answers)
Closed 8 years ago.
I need to capture the stdout of a process I execute via subprocess into a string to then put it inside a TextCtrl of a wx application I'm creating. How do I do that?
EDIT: I'd also like to know how to determine when a process terminates
From the subprocess documentation:
from subprocess import *
output = Popen(["mycmd", "myarg"], stdout=PIPE).communicate()[0]
Take a look at the subprocess module.
http://docs.python.org/library/subprocess.html
It allows you to do a lot of the same input and output redirection that you can do in the shell.
If you're trying to redirect the stdout of the currently executing script, that's just a matter of getting a hold of the correct file handle. Off the top of my head, stdin is 0, stdout is 1, and stderr is 2, but double check. I could be wrong on that point.