I am trying the following:
#!/usr/bin/python
import os, subprocess
func = 'print("Hello World")'
x = subprocess.Popen(['mongo', '--eval', func], stdout=subprocess.PIPE,
stderr=subprocess.PIPE, stdin=subprocess.PIPE)
print x.stdout.read()
print x.stderr.read()
But all I am getting is:
MongoDB shell version: 2.2.3
followed by two new lines. How do I capture the output of function execution?
Reading the pipes gets whatever is currently inside said pipe. Your mongo is waiting to connect to the localhost. Since it doesn't return quickly enough, your read command is not getting the results. This may be because you don't have mongo running locally, but you will run into this problem repeatedly if you don't wait for the subprocess to complete.
Also, keep in mind that subprocess.Popen, to my knowledge, doesn't block. You would probably need to make a x.wait() call if you want the function to complete before trying to grab the output.
Related
On the terminal, I have two programs to run using subprocess
First, I will call ./matrix-odas & so the first program will run in the background and I can then type the second command. The first command will return some messages.
The second command ~/odas/bin/odaslive -vc ~/odas/config/odaslive/matrix_creator.cfg will open the second program and it will keep running and keep printing out text. I'd like to use subprocess to open these programs and capture both outputs.
I have never used subprocess before and following tutorials, I am writing the script on Jupyter notebook (python 3.7) in order to see the output easily.
from subprocess import Popen, PIPE
p = Popen(["./matrix-odas", "&"], stdout=PIPE, stderr=PIPE, cwd=wd, universal_newlines=True)
stdout, stderr = p.communicate()
print(stdout)
This is the code that i tried to open the first program. But Jupyter notebook always gets stuck at p.communicate() and I can't see the messages. Without running the first program in the background, I won't be able to get the command prompt after the messages are printed.
I would like to know what subprocess function should I use to solve this issue and which platform is better to test subprocess code. Any suggestions will be appreciated. Thank you so much!
From this example at the end of this section of the docs
with Popen(["ifconfig"], stdout=PIPE) as proc:
log.write(proc.stdout.read())
it looks like you can access stdout (and I would assume stderr) from the object directly. I am not sure whether you need to use Popen as a context manager to access that property or not.
I have a python script that launches subprocesses using subprocess.Popen. The subprocess then launches an external command (in my case, it plays an mp3). The python script needs to be able to interrupt the subprocesses, so I used the method described here which gives the subprocess its own session ID. Unfortunately, when I close the python script now, the subprocess will continue to run.
How can I make sure a subprocess launched from a script, but given a different session ID still closes when the python script stops?
Is there any way to kill a Thread in Python?
and make sure you use it as thread
import threading
from subprocess import call
def thread_second():
call(["python", "secondscript.py"])
processThread = threading.Thread(target=thread_second) # <- note extra ','
processThread.start()
print 'the file is run in the background'
TL;DR Change the Popen params: Split up the Popen cmd (ex. "list -l" -> ["list", "-l"]) and use Shell=False
~~~
The best solution I've seen so far was just not to use shell=True as an argument for Popen, this worked because I didn't really need shell=True, I was simply using it because Popen wouldn't recognize my cmd string and I was too lazy too split it into a list of args. This caused me a lot of other problems (ex. using .terminate() becomes a lot more complicated while using shell and needs to have its session id, see here)
Simply splitting the cmd from a string to a list of args lets me use Popen.terminate() without having to give it its own session id, by not having a separate session id the process will be closed when the python script is stopped
I have a python build script for a Xamarin application that I need to compile into different ipa's and apk's based on locale.
The script manipulates the necessary values in info.plist and the Android manifest and then builds each of the versions using subprocess.popen to call xbuild. Or at least that's how it's suppose to be.
The problem is that when I in anyway interact with the subprocess (basically i need to wait until it's done before I start changing values for the next version)
This works:
build_path = os.path.dirname(os.path.realpath(__file__))
ipa_path = "/path/to/my.ipa"
cmd = '/Library/Frameworks/Mono.framework/Versions/4.6.2/Commands/xbuild /p:Configuration="Release" /p:Platform="iPhone" /p:IpaPackageDir="%s" /t:Build %s/MyApp/iOS/MyApp.iOS.csproj' % (ipa_path, build_path)
subprocess.Popen(cmd, env=os.environ, shell=True)
However it will result in the python script continuing in parallel with the build.
If I do this:
subprocess.Popen(cmd, env=os.environ, shell=True).wait()
Xbuild fail with the following error message:
Build FAILED.
Errors:
/Users/sune/dev/MyApp/iOS/MyApp.iOS.csproj: error :
/Users/sune/dev/MyApp/iOS/MyApp.iOS.csproj: There is an unclosed literal string.
Line 2434, position 56.
It fails within milliseconds of being called, whereas normally the build process takes several minutes
Any other shorthand methods of subprocess.popen such as .call, .check_call, and the underlying operations of subprocess.poll and subprocess.communicate causes the same error to happen.
What's really strange is that even calling time.sleep can provoke the same error:
subprocess.Popen(cmd, env=os.environ, shell=True)
time.sleep(2)
Which I don't get because as I understand it I should also be able to do something like this:
shell = subprocess.Popen(cmd, env=os.environ, shell=True)
while shell.poll() is None:
time.sleep(2)
print "done"
To essentially achieve the same as calling shell.wait()
Edit: Using command list instead of string
If I use a command list and shell=False like this
cmd = [
'/Library/Frameworks/Mono.framework/Versions/4.6.2/Commands/xbuild',
'/p:Configuration="Release"',
'/p:Platform="iPhone"',
'/p:IpaPackageDir="%s' % ipa_path,
'/t:Build %s/MyApp/iOS/MyApp.iOS.csproj' % build_path
]
subprocess.Popen(cmd, env=os.environ, shell=False)
Then this is the result:
MSBUILD: error MSBUILD0003: Please specify the project or solution file to build, as none was found in the current directory.
Any input is much appreciated. I'm banging my head against the wall here.
I firmly believe that this is not possible. It must be a shortcoming of the way the subprocess module is implemented.
xbuild spawns multiple subprocesses during the build and if polled for status the subprocess in python will discover that one of these had a non-zero return status and stop the execution of one or more of the xbuild subprocesses causing the build to fail as described.
I ended up using a bash script to do the compiling and use python to manipulate xml files etc.
I'm trying to run a shell program through python. I need to run a command, then while it's still running and waiting for input to continue, I need to take the output received by the program, and process that data as a string. Then I need to parse some data into that program, and simulate an enter pressing.
What would be the best way to achieve this?
subprocess.Popen will work for this, but to read and then write and then read again you can't use communicate (because this will cause the process to end).
Instead, you'll need to work with the process's output pipe (process.stdout below). This is tricky to get right, because reading on the process's stdout is blocking, so you sort of need to know when to stop trying to read (or know how much output the process is going to produce).
In this example, the subprocess is a shell script that writes a line of output, and then echoes whatever you give it until it reads EOF.
import subprocess
COMMAND_LINE = 'echo "Hello World!" ; cat'
process = subprocess.Popen(COMMAND_LINE, shell=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE)
s = process.stdout.readline().strip()
print(s)
s2 = process.communicate(s)[0]
print(s2)
Gives:
Hello World!
Hello World!
For more complicated cases, you might think about looking at something like pexpect.
Use subprocess.Popen to run your shell application and use communicate to interact with it.
I have a script that reads from an mssql database and passes the read data to a subprocess of some.exe.
The data fetching works, fine but as soon as it is supposed to start proc = subprocess.(["C:\\absolute\\path\\some.exe ", fetched_data]) proc.wait() it seems to skip it and goes on for the next "fetched_data".. I also tried to use subprocess.call(["C:\\absolute\\path\\some.exe ", fetched_data])
If I start python in the console (windows cmd) and do the exact same thing it works.
Why does calling the subprocess in the script not work and if issued manually in the console it does?
edit: The problem was that the subprocess started in the script again used another.exe, which couldn't be found by the subprocess (as the it used the python path). When started from directory where some.exe and another.exe are, the script runs fine.
fetched_data is an additional argument therefore:
proc = subprocess.call(["C:\\absolute\\path\\some.exe ", fetched_data])
It's an argument LIST not a string, what subprocess expects.