We Keep Coding

Python: subprocess.call and variants fail for a particular application from executed .py but not from python in CLI

I have a strange issue here - I have an application that I'm attempting to launch from python, but all attempts to launch it from within a .py script fail without any discernable output. Testing from within VSCode debugger. Here's some additional oddities:
When I swap in notepad.exe into the .py instead of my target applications path, notepad launches ok.
When I run the script line by line from the CLI (start by launching python, then type out the next 4-5 lines of Python), the script works as expected.
Examples:
#This works in the .py, and from the CLI
import subprocess
cmd = ['C:\\Windows\\system32\\notepad.exe', 'C:\\temp\\myfiles\\test_24.xml']
pipe = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
pipe.wait()
print(pipe)
#This fails in the .py, but works ok when pasted in line by line from the CLI
import subprocess
cmd = ['C:\\temp\\temp_app\\target_application.exe', 'C:\\temp\\myfiles\\test_24.xml']
pipe = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
pipe.wait()
print(pipe)
The result is no output when running the .py
I've tried several other variants, including the following:
import subprocess
tup = 'C:\\temp\\temp_app\\target_application.exe C:\temp\test\test_24.xml'
proc = subprocess.Popen(tup)
proc.wait()
(stdout, stderr) = proc.communicate()
print(stdout)
if proc.returncode != 0:
print("The error is: " + str(stderr))
else:
print("Executed: " + str(tup))
Result:
None
The error is: None
1.082381010055542
Now this method indicates there is an error because we are returning something other than 0 and printing "The error is: None", and this is because stderror is "None". So - is it throwing an error without giving an error?
stdout is also reporting "None".
So, lets try check_call and see what happens:
print("Trying check_call")
try:
subprocess.check_call('C:\\temp\\temp_app\\target_application.exe C:\\temp\\test\\test_24.xml', shell=True)
except subprocess.CalledProcessError as error:
print(error)
Results:
Trying check_call
Command 'C:\temp\temp_app\target_application.exe C:\temp\test\test_24.xml' returned non-zero exit status 1.
I've additionally tried subprocess.run, although it is missing the wait procedure I was hoping to use.
import subprocess
tup = 'C:\\temp\\temp_app\\target_application.exe C:\temp\test\test_24.xml'
proc = subprocess.run(tup, check=True)
proc.wait()
(stdout, stderr) = proc.communicate()
print(stdout)
if proc.returncode != 0:
print("The error is: " + str(stderr))
else:
print("Executed: " + str(tup))
What reasons might be worth chasing, or what other ways of trying to catch an error might work here? I don't know how to interpret "`" as an error result.

how can i close a subprocess started by python program without closing the program

I have a the following python program which open a command as subprocess and displays its output. This command though waits for user input after displaying information. I on the other hand want to get that information and close this subprocess and get back to what program is doing
Program is this
#!/usr/bin/python
from commands import *
import subprocess, os, sys, time
cmd = '/usr/local/bin/astrisctl reset'
p = subprocess.Popen(cmd, shell=True, stderr=subprocess.PIPE)
time.sleep( 5 )
p.kill()
out = p.stderr
if out == '' and p.poll() != None:
print "No output from the commnand"
sys.exit()
if out != '':
sys.stdout.write(out)
sys.stdout.flush()
print "i am back"
i want something like
something something
0) xyz-0008E8
1) xyz-000AB9
2) xyz-000D18
3) xyz-000D2B
4) xyz-000E9D
pick a choice
i am back
but comes like
dta-rack-04:~ admin$ python Desktop/getBurnInLogs.py
something something
0) xyz-0008E8
1) xyz-000AB9
2) xyz-000D18
3) xyz-000D2B
4) xyz-000E9D
Choose a device: Traceback (most recent call last):
File "Desktop/getBurnInLogs.py", line 16, in <module>
sys.stdout.write(out)
but i am hoping
something something
0) xyz-0008E8
1) xyz-000AB9
2) xyz-000D18
3) xyz-000D2B
4) xyz-000E9D
pick a choice
i am back
Main purpose is to not wait for that choice.
Also if you could suggest a good way to get this options in list looking like {'xyz-000E9D', 'xyz-000E9D'..... etc etc}
Thanks a lot for your help and time

If you want to send it SIGKILL then use
p.kill()
This should kill the process immediately
If you want to wait for it to finish (which it doesn't sound like you do), then use
p.wait()
UPDATE: in the case that the output is coming from stdout and not stderr, you'll need to capture that. Since you're doing some weird stuff, you'll want to make it unbuffered
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, bufsize=0)
and then read from p.stdout, handle with care.
You can also get rid of shell=True pretty easily
cmd = ['/usr/local/bin/astrisctl', 'reset']
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, bufsize=0)

How to implement retry mechanism if the shell script execution got failed?

I am trying to execute shell script in Python code. And so far everything is looking good.
Below is my Python script which will execute a shell script. Now for an example sake, here it is a simple Hello World shell script.
jsonStr = '{"script":"#!/bin/bash\\necho Hello world 1\\n"}'
j = json.loads(jsonStr)
shell_script = j['script']
print "start"
proc = subprocess.Popen(shell_script, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(stdout, stderr) = proc.communicate()
if stderr:
print "Shell script gave some error"
print stderr
else:
print stdout
print "end" # Shell script ran fine.
Now what I am looking for is, suppose for whatever reason whenever I am executing my shell script from Python code and it got failed for whatever reason. Then that means stderr won't be empty. So now I want to retry executing the shell script again, let's say after sleeping for couple of milliseconds?
Meaning is there any possibility of implementing of retry mechanism if the shell script execution got failed? Can I retry for 5 or 6 times? Meaning is it possible to configure this number as well?

from time import sleep
MAX_TRIES = 6
# ... your other code ...
for i in xrange(MAX_TRIES):
proc = subprocess.Popen(shell_script, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(stdout, stderr) = proc.communicate()
if stderr:
print "Shell script gave some error..."
print stderr
sleep(0.05) # delay for 50 ms
else:
print stdout
print "end" # Shell script ran fine.
break

Something like this maybe:
maxRetries = 6
retries = 0
while (retries < maxRetries):
doSomething ()
if errorCondition:
retries += 1
continue
break

How about using a decorator? Seems like a very clear way.
You can read about them here https://wiki.python.org/moin/PythonDecoratorLibrary. (Retry decorator)

Catching runtime error for process created by python subprocess

I am writing a script which can take a file name as input, compile it and run it.
I am taking the name of a file as input(input_file_name). I first compile the file from within python:
self.process = subprocess.Popen(['gcc', input_file_name, '-o', 'auto_gen'], stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.STDOUT, shell=False)
Next, I'm executing the executable using the same(Popen) call:
subprocess.Popen('./auto_gen', stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.STDOUT, shell=False)
In both cases, I'm catching the stdout(and stderr) contents using
(output, _) = self.process.communicate()
Now, if there is an error during compilation, I am able to catch the error because the returncode is 1 and I can get the details of the error because gcc sends them on stderr.
However, the program itself can return a random value even on executing successfully(because there might not be a "return 0" at the end). So I can't catch runtime errors using the returncode. Moreover, the executable does not send the error details on stderr. So I can't use the trick I used for catching compile-time errors.
What is the best way to catch a runtime error OR to print the details of the error? That is, if ./auto_gen throws a segmentation fault, I should be able to print either one of:
'Runtime error'
'Segmentation Fault'
'Program threw a SIGSEGV'

Try this. The code runs a subprocess which fails and prints to stderr. The except block captures the specific error exit code and stdout/stderr, and displays it.
#!/usr/bin/env python
import subprocess
try:
out = subprocess.check_output(
"ls non_existent_file",
stderr=subprocess.STDOUT,
shell=True)
print 'okay:',out
except subprocess.CalledProcessError as exc:
print 'error: code={}, out="{}"'.format(
exc.returncode, exc.output,
)
Example output:
$ python ./subproc.py
error: code=2, out="ls: cannot access non_existent_file: No such file or directory
"

If ./autogen is killed by a signal then self.process.returncode (after .wait() or .communicate()) is less than zero and its absolute value reports the signal e.g., returncode == -11 for SIGSERV.

please check following link for runtime errors or output of subprocess
https://www.endpoint.com/blog/2015/01/28/getting-realtime-output-using-python
def run_command(command):
process = subprocess.Popen(shlex.split(command),
stdout=subprocess.PIPE)
while True:
output = process.stdout.readline()
if output == '' and process.poll() is not None:
break
if output:
print output.strip()
rc = process.poll()
return rc

How can I call a python script from a python script

I have a python script 'b.py' which prints out time ever 5 sec.
while (1):
print "Start : %s" % time.ctime()
time.sleep( 5 )
print "End : %s" % time.ctime()
time.sleep( 5 )
And in my a.py, I call b.py by:
def run_b():
print "Calling run b"
try:
cmd = ["./b.py"]
p = subprocess.Popen(cmd,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
for line in iter(p.stdout.readline, b''):
print (">>>" + line.rstrip())
except OSError as e:
print >>sys.stderr, "fcs Execution failed:", e
return None
and later on, I kill 'b.py' by:
PS_PATH = "/usr/bin/ps -efW"
def kill_b(program):
try:
cmd = shlex.split(PS_PATH)
retval = subprocess.check_output(cmd).rstrip()
for line in retval.splitlines():
if program in line:
print "line =" + line
pid = line.split(None)[1]
os.kill(int(pid), signal.SIGKILL)
except OSError as e:
print >>sys.stderr, "kill_all Execution failed:", e
except subprocess.CalledProcessError as e:
print >>sys.stderr, "kill_all Execution failed:", e
run_b()
time.sleep(600)
kill_b("b.py")
I have 2 questions.
1. why I don't see any prints out from 'b.py' and when I do 'ps -efW' I don't see a process named 'b.py'?
2. Why when I kill a process like above, I see 'permission declined'?
I am running above script on cygwin under windows.
Thank you.

Why I don't see any prints out from 'b.py' and when I do 'ps -efW' I don't see a process named 'b.py'?
Change run_b() lines:
p = subprocess.Popen(cmd,
stdout=sys.stdout,
stderr=sys.stderr)
You will not see a process named "b.py" but something like "python b.py" which is little different. You should use pid instead of name to find it (in your code "p.pid" has the pid).
Why when I kill a process like above, I see 'permission declined'?
os.kill is supported under Windows only 2.7+ and acts a little bit different than posix version. However you can use "p.pid". Best way to kill a process in a cross platform way is:
if platform.system() == "Windows":
subprocess.Popen("taskkill /F /T /PID %i" % p.pid, shell=True)
else:
os.killpg(p.pid, signal.SIGKILL)
killpg works also on OS X and other Unixy operating systems.