We Keep Coding

subprocess.Popen communicate method open automaticaly the file, stops the program execution until I manually close the file

I have one problem with the python subprocess module.
import os, subprocess
BLEU_SCRIPT_PATH = os.path.join(os.path.abspath(os.path.dirname(__file__)), 'multi-bleu.perl')
command = BLEU_SCRIPT_PATH + ' %s < %s'
ref = "ref.en-fr.test.txt"
hyp = "hyp100.en-fr.test.txt"
p = subprocess.Popen(command % (ref, hyp), stdout=subprocess.PIPE, shell=True)
result = p.communicate()[0].decode("utf-8")
# ...
# ...
The multi-bleu.perl file does the evaluation and returns a real number or an error if any; but that's not my concern.
The last line of code automatically opens the multi-bleu.perl file with my default text editor, stops the program execution until I manually close the file.
How can I disable this behavior?

I don't think subprocess.Popen interprets any shebang in the file. You need to specify the executable in the command to be executed. Also, Popen requires a list as first argument, so you need to "lift up" your string formatting into the command-list.
command = [
'/path/to/perl',
BLEU_SCRIPT_PATH + ' %s < %s' % (ref, hyp)
]
p = subprocess.Popen(command, stdout=subprocess.PIPE, shell=True)
You might also want to have a look at subprocess.check_output which will make the code a bit easier.

Unix Popen.communicate not able to gzip large file

I need to gzip files of size more than 10 GB using python on top of shell commands and hence decided to use subprocess Popen.
Here is my code:
outputdir = '/mnt/json/output/'
inp_cmd='gzip -r ' + outputdir
pipe = Popen(["bash"], stdout =PIPE,stdin=PIPE,stderr=PIPE)
cmd = bytes(inp_cmd.encode('utf8'))
stdout_data,stderr_data = pipe.communicate(input=cmd)
It is not gzip-ing the files within output directory.
Any way out?

The best way is to use subprocess.call() instead of subprocess.communicate().
call() waits till the command is executed completely while in Popen(), one has to extrinsically use wait() method for the execution to finish.

Have you tried it like this:
output_dir = "/mnt/json/output/"
cmd = "gzip -r {}".format(output_dir)
proc = subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
stdin=subprocess.PIPE,
shell=True,
)
out, err = proc.communicate()

multiprocessing.Process subprocess.Popen completed?

I have a server that launches command line apps. They receive a local file path, load a file, export something, then close.
It's working, but I would like to be able to keep track of which tasks are active and which completed.
So with this line:
p = mp.Process(target=subprocess.Popen(mayapy + ' -u ' + job.pyFile), group=None)
I have tried 'is_alive', and it always returns False.
The subprocess closes, I see it closed in task manager, but the process and pid still seem queryable.

Your use of mp.Process is wrong. The target should be a function, not the return value of subprocess.Popen(...).
In any case, if you define:
proc = subprocess.Popen(mayapy + ' -u ' + job.pyFile)
Then proc.poll() will be None while the process is working, and will equal a return value (not None) when the process has terminated.
For example, (the output is in the comments)
import subprocess
import shlex
import time
PIPE = subprocess.PIPE
proc = subprocess.Popen(shlex.split('ls -lR /'), stdout=PIPE)
time.sleep(1)
print(proc.poll())
# None
proc.terminate()
time.sleep(1)
print(proc.poll())
# -15

Assign output of os.system to a variable and prevent it from being displayed on the screen [duplicate]

This question already has answers here:
Running shell command and capturing the output
(21 answers)
Closed 2 years ago.
I want to assign the output of a command I run using os.system to a variable and prevent it from being output to the screen. But, in the below code ,the output is sent to the screen and the value printed for var is 0, which I guess signifies whether the command ran successfully or not. Is there any way to assign the command output to the variable and also stop it from being displayed on the screen?
var = os.system("cat /etc/services")
print var #Prints 0

From this question which I asked a long time ago, what you may want to use is popen:
os.popen('cat /etc/services').read()
From the docs for Python 3.6,
This is implemented using subprocess.Popen; see that class’s
documentation for more powerful ways to manage and communicate with
subprocesses.
Here's the corresponding code for subprocess:
import subprocess
proc = subprocess.Popen(["cat", "/etc/services"], stdout=subprocess.PIPE, shell=True)
(out, err) = proc.communicate()
print("program output:", out)

You might also want to look at the subprocess module, which was built to replace the whole family of Python popen-type calls.
import subprocess
output = subprocess.check_output("cat /etc/services", shell=True)
The advantage it has is that there is a ton of flexibility with how you invoke commands, where the standard in/out/error streams are connected, etc.

The commands module is a reasonably high-level way to do this:
import commands
status, output = commands.getstatusoutput("cat /etc/services")
status is 0, output is the contents of /etc/services.

For python 3.5+ it is recommended that you use the run function from the subprocess module. This returns a CompletedProcess object, from which you can easily obtain the output as well as return code. Since you are only interested in the output, you can write a utility wrapper like this.
from subprocess import PIPE, run
def out(command):
result = run(command, stdout=PIPE, stderr=PIPE, universal_newlines=True, shell=True)
return result.stdout
my_output = out("echo hello world")
# Or
my_output = out(["echo", "hello world"])

I know this has already been answered, but I wanted to share a potentially better looking way to call Popen via the use of from x import x and functions:
from subprocess import PIPE, Popen
def cmdline(command):
process = Popen(
args=command,
stdout=PIPE,
shell=True
)
return process.communicate()[0]
print cmdline("cat /etc/services")
print cmdline('ls')
print cmdline('rpm -qa | grep "php"')
print cmdline('nslookup google.com')

I do it with os.system temp file:
import tempfile, os
def readcmd(cmd):
ftmp = tempfile.NamedTemporaryFile(suffix='.out', prefix='tmp', delete=False)
fpath = ftmp.name
if os.name=="nt":
fpath = fpath.replace("/","\\") # forwin
ftmp.close()
os.system(cmd + " > " + fpath)
data = ""
with open(fpath, 'r') as file:
data = file.read()
file.close()
os.remove(fpath)
return data

Python 2.6 and 3 specifically say to avoid using PIPE for stdout and stderr.
The correct way is
import subprocess
# must create a file object to store the output. Here we are getting
# the ssid we are connected to
outfile = open('/tmp/ssid', 'w');
status = subprocess.Popen(["iwgetid"], bufsize=0, stdout=outfile)
outfile.close()
# now operate on the file

from os import system, remove
from uuid import uuid4
def bash_(shell_command: str) -> tuple:
"""
:param shell_command: your shell command
:return: ( 1 | 0, stdout)
"""
logfile: str = '/tmp/%s' % uuid4().hex
err: int = system('%s &> %s' % (shell_command, logfile))
out: str = open(logfile, 'r').read()
remove(logfile)
return err, out
# Example:
print(bash_('cat /usr/bin/vi | wc -l'))
>>> (0, '3296\n')```

What is the multiplatform alternative to subprocess.getstatusoutput (older commands.setstatusoutput() from Python?

The code below is outdated in Python 3.0 by being replaced by subprocess.getstatusoutput().
import commands
(ret, out) = commands.getstatusoutput('some command')
print ret
print out
The real question is what's the multiplatform alternative to this command from Python because the above code does fail ugly under Windows because getstatusoutput is supported only under Unix and Python does not tell you this, instead you get something like:
>test.py
1
'{' is not recognized as an internal or external command,
operable program or batch file.

This would be the multiplatform implementation for getstatusoutput():
def getstatusoutput(cmd):
"""Return (status, output) of executing cmd in a shell."""
"""This new implementation should work on all platforms."""
import subprocess
pipe = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True, universal_newlines=True)
output = "".join(pipe.stdout.readlines())
sts = pipe.returncode
if sts is None: sts = 0
return sts, output

I wouldn't really consider this multiplatform, but you can use subprocess.Popen:
import subprocess
pipe = subprocess.Popen('dir', stdout=subprocess.PIPE, shell=True, universal_newlines=True)
output = pipe.stdout.readlines()
sts = pipe.wait()
print sts
print output
Here's a drop-in replacement for getstatusoutput:
def getstatusoutput(cmd):
"""Return (status, output) of executing cmd in a shell."""
"""This new implementation should work on all platforms."""
import subprocess
pipe = subprocess.Popen(cmd, shell=True, universal_newlines=True,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
output = str.join("", pipe.stdout.readlines())
sts = pipe.wait()
if sts is None:
sts = 0
return sts, output
This snippet was proposed by the original poster. I made some changes since getstatusoutput duplicates stderr onto stdout.
The problem is that dir isn't really a multiplatform call but subprocess.Popen allows you to execute shell commands on any platform. I would steer clear of using shell commands unless you absolutely need to. Investigate the contents of the os, os.path, and shutil packages instead.
import os
import os.path
for rel_name in os.listdir(os.curdir):
abs_name = os.path.join(os.curdir, rel_name)
if os.path.isdir(abs_name):
print('DIR: ' + rel_name)
elif os.path.isfile(abs_name):
print('FILE: ' + rel_name)
else:
print('UNK? ' + rel_name)

getstatusoutput docs say it runs the command like so:
{ cmd } 2>&1
Which obviously doesn't work with cmd.exe (the 2>&1 works fine if you need it though).
You can use Popen as above, but also include the parameter 'stderr=subprocess.STDOUT' to get the same behaviour as getstatusoutput.
My tests on Windows had returncode set to None though, which is not ideal if you're counting on the return value.