We Keep Coding

Python - evaluate shell command before executing it

I have a Python function that I made with the subprocess package:
def run_sh(command):
"""Print output of bash command"""
try:
process = Popen(shlex.split(command), stdout=PIPE)
for line in TextIOWrapper(process.stdout, newline=""):
print(line)
except CalledProcessError as e:
raise RuntimeError(
"command '{}' return with error (code {}): {}".format(
e.cmd, e.returncode, e.output
)
)
Let's say I want to run the following from within my Python script:
run_sh(newman run MY_COLLECTION.json "--env-var 'current_branch'=`git branch --show-current`")
Currently, it does not evaluate it as git branch --show-current but just treats it like test - how do I get it to evaluate it from my shell, and then run it?
Thanks!

Here's a code snippet that might help
import subprocess
def run_command(cmd):
try:
proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = proc.communicate()
if out:
print(out.decode())
if err:
print(err.decode())
except Exception as e:
print(e)
if __name__ == '__main__':
run_command('ls -l')
run_command('ls -l /tmp')
run_command('ls -l /tmp1')

Print realtime output of subprocess in jupyter notebook

In my file 'wrapper.py' I call a subprocess and print its output to stdout at realtime . This works just fine if I call the python script from the console. However, when calling it from a jupyter notebook the code hangs at the line proc.stdout.readline(). All previous print statements work fine..
proc = subprocess.Popen(["calc", "input.txt"], cwd=__dir, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
while True:
out = proc.stdout.readline().decode("utf-8")
err = proc.stderr.readline().decode("utf-8")
if out == '' and err == '' and proc.poll() is not None:
break
if out:
print("::::%s"%(out), end='')
if err:
print("::::%s"%(err), end='', file=sys.stderr)
rc = proc.poll()
print("---> returns %s"%(rc))
Does anyone have an idea on how to fix this?

I use this custom function to execute a command in Notebooks.
import subprocess
def run_cmd(cmd: str, stderr=subprocess.STDOUT) -> None:
"""Run a command in terminal
Args:
cmd (str): command to run in terminal
stderr (subprocess, optional): Where the error has to go. Defaults to subprocess.STDOUT.
Raises:
e: Excetion of the CalledProcessError
"""
out = None
try:
out = subprocess.check_output(
[cmd],
shell=True,
stderr=stderr,
universal_newlines=True,
)
except subprocess.CalledProcessError as e:
print(f'ERROR {e.returncode}: {cmd}\n\t{e.output}',
flush=True, file=sys.stderr)
raise e
print(out)
Usecase:
run_cmd("pip install emoji")

python subprocess wildcard is selection only first file

I want to delete all the below files:
20200922_051424_00011_v4wzh_db508ed0-b8b9-488b-a796-773d1fb4045c_08
20200922_051424_00011_v4wzh_db508ed0-b8b9-488b-a796-773d1fb4045c_04 20200922_051424_00011_v4wzh_db508ed0-b8b9-488b-a796-773d1fb4045c_09
20200922_051424_00011_v4wzh_db508ed0-b8b9-488b-a796-773d1fb4045c_05 20200922_051424_00011_v4wzh_db508ed0-b8b9-488b-a796-773d1fb4045c_10
In Linux I simply do:
rm 20200922_051424_00011_v4wzh_db508ed0-b8b9-488b-a796-773d1fb4045c_*
But when I am doing the same using python script. It is just deleting first file matching the pattern but not all of them:
temp = subprocess.Popen('rm 20200922_051424_00011_v4wzh_db508ed0-b8b9-488b-a796-773d1fb4045c_*', shell=True, stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
Can anyone tell me the reason why its not working and also what should I do?
Complete python function is:
def remove(filename):
try:
cmd = 'rm ' + filename
print(cmd)
temp = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
stdout, stderr = temp.communicate()
if stderr:
print('Error while running rm command.')
print("Result of running rm command: ", stdout)
except CalledProcessError as e:
pass

Since you're in python, why not remove them directly from python rather than calling a shell command?
for filename in glob.glob(pattern):
os.remove(filename)
Documentation:
os.remove()
glob.glob()

python calling an external cmd and redirect stdout to file

Error message:
Failed to open output_file_path/**.txt
Code:
cmd = 'showTxt "%s" > "%s"' % (file_path, output_file_path)
LoggerInstance.log('[cmd] '+cmd)
#os.system(cmd)
splited_cmd=shlex.split(cmd)
p = subprocess.Popen(splited_cmd, stderr=subprocess.PIPE)
#p.wait()
output = p.stderr.read()
print output
LoggerInstance.log('[console std error]'+ output)
How to redirect stdout to a file in a cmd?

You can provide a file-handler as stdout parameter to Popen, i.e:
p = subprocess.Popen(splited_cmd,
stderr=subprocess.PIPE,
stdout=open(output_file_path, "w"))
Of course, be ready to catch the exception that it can throw.

Python: get output from a command line which exits with nonzero exit code

I am Using Python 2.7.1 on a Windows Server 2008 R2 x64 box.
I'm trying to get the output of a command line process which gives a nonzero exit status after outputting the information I need.
I was initially using subprocess.check_output, and catching the CalledProcessError which occurs with nonzero exit status, but while the returncode was stored in the error, no output revealed this.
Running this against cases which give output but have an exit status of 0 works properly and I can get the output using subprocess.check_output.
My assumption was that the output was being written to STDOUT but the exception pulls its 'output' from STDERR. I've tried to re implement the functionality of check_output, but I still get nothing on the output when I believe I should be seeing output to STDOUT and STDERR. My current code is below (where 'command' is the full text, including parameters, of command I am running:
process = subprocess.Popen(command, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, universal_newlines=True)
output = process.communicate()
retcode = process.poll()
if retcode:
raise subprocess.CalledProcessError(retcode, image_check, output=output)
return output
This gives me the following in the variable output: [('', None)]
Is my subprocess.Popen code correct?

You code works fine. Turns out that the process that you are calling is probably outputing to CON. See the following example
import subprocess
def check_output(command):
process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True)
output = process.communicate()
retcode = process.poll()
if retcode:
raise subprocess.CalledProcessError(retcode, command, output=output[0])
return output
command = "echo this>CON"
print "subprocess -> " + subprocess.check_output(command, shell=True)
print "native -> " + str(check_output(command))
try:
subprocess.check_output("python output.py", shell=True)
except subprocess.CalledProcessError, e:
print "subproces CalledProcessError.output = " + e.output
try:
check_output("python output.py")
except subprocess.CalledProcessError, e:
print "native CalledProcessError.output = " + e.output
Output
subprocess ->
native -> ('', None)
stderr subproces CalledProcessError.output = stdout
native CalledProcessError.output = stderr stdout
Sadly, I do not know how to resolve the issue. Notice that subprocess.check_output results contains only the output from stdout. Your check_output replacement would output both stderr and stdout.
After inspecting subprocess.check_output, it does indeed generate a CalledProcessError with the output containing only stdout.

Have you tried stderr=subprocess.STDOUT as mentioned in the python doc page:
To also capture standard error in the result, use
stderr=subprocess.STDOUT:
Here is a test code:
import subprocess
try:
subprocess.check_output('>&2 echo "errrrr"; exit 1', shell=True)
except subprocess.CalledProcessError as e:
print 'e.output: ', e.output
try:
subprocess.check_output('>&2 echo "errrrr"; exit 1', shell=True, stderr=subprocess.STDOUT)
except subprocess.CalledProcessError as e:
print 'e.output: ', e.output
output:
errrrr
e.output:
e.output: errrrr

There is an issue here that might be hitting you-
http://bugs.python.org/issue9905