We Keep Coding

Python: subprocess.call and variants fail for a particular application from executed .py but not from python in CLI

I have a strange issue here - I have an application that I'm attempting to launch from python, but all attempts to launch it from within a .py script fail without any discernable output. Testing from within VSCode debugger. Here's some additional oddities:
When I swap in notepad.exe into the .py instead of my target applications path, notepad launches ok.
When I run the script line by line from the CLI (start by launching python, then type out the next 4-5 lines of Python), the script works as expected.
Examples:
#This works in the .py, and from the CLI
import subprocess
cmd = ['C:\\Windows\\system32\\notepad.exe', 'C:\\temp\\myfiles\\test_24.xml']
pipe = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
pipe.wait()
print(pipe)
#This fails in the .py, but works ok when pasted in line by line from the CLI
import subprocess
cmd = ['C:\\temp\\temp_app\\target_application.exe', 'C:\\temp\\myfiles\\test_24.xml']
pipe = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
pipe.wait()
print(pipe)
The result is no output when running the .py
I've tried several other variants, including the following:
import subprocess
tup = 'C:\\temp\\temp_app\\target_application.exe C:\temp\test\test_24.xml'
proc = subprocess.Popen(tup)
proc.wait()
(stdout, stderr) = proc.communicate()
print(stdout)
if proc.returncode != 0:
print("The error is: " + str(stderr))
else:
print("Executed: " + str(tup))
Result:
None
The error is: None
1.082381010055542
Now this method indicates there is an error because we are returning something other than 0 and printing "The error is: None", and this is because stderror is "None". So - is it throwing an error without giving an error?
stdout is also reporting "None".
So, lets try check_call and see what happens:
print("Trying check_call")
try:
subprocess.check_call('C:\\temp\\temp_app\\target_application.exe C:\\temp\\test\\test_24.xml', shell=True)
except subprocess.CalledProcessError as error:
print(error)
Results:
Trying check_call
Command 'C:\temp\temp_app\target_application.exe C:\temp\test\test_24.xml' returned non-zero exit status 1.
I've additionally tried subprocess.run, although it is missing the wait procedure I was hoping to use.
import subprocess
tup = 'C:\\temp\\temp_app\\target_application.exe C:\temp\test\test_24.xml'
proc = subprocess.run(tup, check=True)
proc.wait()
(stdout, stderr) = proc.communicate()
print(stdout)
if proc.returncode != 0:
print("The error is: " + str(stderr))
else:
print("Executed: " + str(tup))
What reasons might be worth chasing, or what other ways of trying to catch an error might work here? I don't know how to interpret "`" as an error result.

How to run minecraft server from python?

I cant figure out how to start the server using a python command.
s = subprocess.Popen('"D:\MC SERVER 2k19\server.jar" -jar server.jar java', stdin=subprocess.PIPE, stdout=subprocess.PIPE, shell=True)
This code runs without error but doesn't start the server in cmd.
Thanks.

It's got to do with how you're passing your arguments.
subprocess.Popen(['java', '-jar', 'server.jar'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, shell=True, cwd='D:\MC SERVER 2k19')

If you need to start a Java program using CMD process from Python and show the window, you can use subprocess to call open another CMD terminal and run the command.
In Windows you will need to CMD-escape spaces in the path you passing to the secondary CMD process. This is done with the carrot ^
proc = subprocess.Popen(
['start', 'cmd', '/k', "D:\\MC^ SERVER^ 2k19\\server.jar",
'-jar', 'server.jar', 'java'],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
shell=True
)
Keep in mind you will NOT be able to retrieve any output from the secondary CMD process from Python.
I.e. the process will return nothing.
proc.communicate()
# returns:
(b'', b'')

python subprocess.call output is not interleaved

I have a python (v3.3) script that runs other shell scripts. My python script also prints message like "About to run script X" and "Done running script X".
When I run my script I'm getting all the output of the shell scripts separate from my print statements. I see something like this:
All of script X's output
All of script Y's output
All of script Z's output
About to run script X
Done running script X
About to run script Y
Done running script Y
About to run script Z
Done running script Z
My code that runs the shell scripts looks like this:
print( "running command: " + cmnd )
ret_code = subprocess.call( cmnd, shell=True )
print( "done running command")
I wrote a basic test script and do *not* see this behaviour. This code does what I would expect:
print("calling")
ret_code = subprocess.call("/bin/ls -la", shell=True )
print("back")
Any idea on why the output is not interleaved?

Thanks. This works but has one limitation - you can't see any output until after the command completes. I found an answer from another question (here) that uses popen but also lets me see the output in real time. Here's what I ended up with this:
import subprocess
import sys
cmd = ['/media/sf_git/test-automation/src/SalesVision/mswm/shell_test.sh', '4', '2']
print('running command: "{0}"'.format(cmd)) # output the command.
# Here, we join the STDERR of the application with the STDOUT of the application.
process = subprocess.Popen(cmd, bufsize=1, universal_newlines=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in iter(process.stdout.readline, ''):
line = line.replace('\n', '')
print(line)
sys.stdout.flush()
process.wait() # Wait for the underlying process to complete.
errcode = process.returncode # Harvest its returncode, if needed.
print( 'Script ended with return code of: ' + str(errcode) )
This uses Popen and allows me to see the progress of the called script.

It has to do with STDOUT and STDERR buffering. You should be using subprocess.Popen to redirect STDOUT and STDERR from your child process into your application. Then, as needed, output them. Example:
import subprocess
cmd = ['ls', '-la']
print('running command: "{0}"'.format(cmd)) # output the command.
# Here, we join the STDERR of the application with the STDOUT of the application.
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
process.wait() # Wait for the underlying process to complete.
out, err = process.communicate() # Capture what it outputted on STDOUT and STDERR
errcode = process.returncode # Harvest its returncode, if needed.
print(out)
print('done running command')
Additionally, I wouldn't use shell = True unless it's really required. It forces subprocess to fire up a whole shell environment just to run a command. It's usually better to inject directly into the env parameter of Popen.

prevent subprocess.Popen from displaying output in python

So I am trying to store the output of a command into a variable. I do not want it to display output while running the command though...
The code I have right now is as follows...
def getoutput(*args):
myargs=args
listargs=[l.split(' ',1) for l in myargs]
import subprocess
output=subprocess.Popen(listargs[0], shell=False ,stdout=subprocess.PIPE)
out, error = output.communicate()
return(out,error)
def main():
a,b=getoutput("httpd -S")
if __name__ == '__main__':
main()
If I put this in a file and execute it on the command line. I get the following output even though I do not have a print statement in the code. How can I prevent this, while still storing the output?
#python ./apache.py
httpd: Could not reliably determine the server's fully qualified domain name, using xxx.xxx.xxx.xx for ServerName
Syntax OK

What you are seeing is standard-error output, not standard-output output. Stderr redirection is controlled by the stderr constructor argument. It defaults to None, which means no redirection occurs, which is why you see this output.
Usually it's a good idea to keep stderr output since it aids debugging and doesn't affect normal redirection (e.g. | and > shell redirection won't capture stderr by default). However you can redirect it somewhere else the same way you do stdout:
sp = subprocess.Popen(listargs[0], shell=False,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output, error = sp.communicate()
Or you can just drop stderr:
devnull = open(os.devnull, 'wb') #python >= 2.4
sp = subprocess.Popen(listargs[0], shell=False,
stdout=subprocess.PIPE, stderr=devnull)
#python 3.x:
sp = subprocess.Popen(listargs[0], shell=False
stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)

You're catching stdout, but you're not catching stderr(standard error) which I think is where that message is coming from.
output=subprocess.Popen(listargs[0], shell=False ,stdout=subprocess.PIPE, stderr=STDOUT)
That will put anything from stderr into the same place as stdout.

How to control a command window opened from a .cmd file using Python

There's a file named startup.cmd that sets some environment variables, runs some preparation commands, then does:
start "startup" cmd /k
Which opens a command shell named startup. The manual process I'm trying to automate is to then enter the following command into this shell: get startup.xml. I thought the correct way to do this in Python would be something like this:
import subprocess
p = subprocess.Popen('startup.cmd', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
getcommand = 'get startup.xml'
servercommand = 'startserver'
p.stdin.write(getcommand)
p.stdin.write(startserver)
(stdoutdata, stderrdata) = p.communicate()
print stdoutdata
print stderrdata
But those commands don't seem to be executing in the shell. What am I missing? Also, the command shell appears regardless of whether shell is set to True or False.

I found this warning in subprocess's document,
Warning Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.
So my suggestion is to use communicate to send your command.
import subprocess
p = subprocess.Popen('startup.cmd', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
command = 'get startup.xml\n'
command += 'startserver\n'
(stdoutdata, stderrdata) = p.communicate(command)
print stdoutdata
print stderrdata

This is a new process, so one cannot communicate directly with Popen.