We Keep Coding

Do I have to close stdout after use subprocess.run()

I am trying to create subprocess to remove python a package and return stdout and stderr. I do following but I wonder is it safe to use? Today I faced /bin/bash: resource temporarily unavailable error while I am using. And when I do ps ux I saw lots of bin/bash process.
I think that function cause lots of bash terminal in the background. How should I safely close the subprocess after I got stdout and stderr? Documentation says run method is the recommended way.
def run_subprocess_command(process_command):
response = {"stdout": "", "stderr": "", "exception": ""}
try:
plugin_install_feedback.send(
sender="", message="Package install starting..")
p = subprocess.run(process_command,
universal_newlines=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
timeout=180)
response["stdout"] = p.stdout
response["stderr"] = p.stderr
return response
except Exception as err:
response["exception"] = err
return response

How to prevent Python's subprocess from printing the standard out when calling the Linux passwd utility?

When I use subprocess I can normally capture the stdout and display it however I like. E.g,
import subprocess
proc = subprocess.Popen(['./foo.py'], stdin=subprocess.Pipe, stdout=subprocess.Pipe)
# the standard out is not displayed unless I do something with the stdout var
stdout, stderr = proc.communicate()
However, if I use subprocess to call the Linux passwd utility, the standard out is displayed as soon as proc.communicate() is called:
import subprocess
proc = subprocess.Popen(['passwd', 'foo'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
# standard out is displayed immediately
stdout, stderr = proc.communicate('password\npassword\n')
BAD PASSWORD: it is based on a dictionary word
Retype new password:
How come this happens only with passwd? For example, it doesn't happen with ls. Is there anything I can do to prevent the standard out from being printed when calling passwd from subprocess?
Note that I want to actually capture the standard out and do something with it later, so I would not want to set stdout to a devnull pipe.

It only happens with passwd because passwd directly communicates with the TTY, not via stdin or stdout. This is a security measure, and accepted best practice for prompting for a password directly from a user.
If you really must bypass this security measure, consider using the unbuffer utility (shipped with expect) to create a fake TTY:
p = subprocess.Popen(['unbuffer', 'passwd', 'foo'],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
(stdout, stderr) = p.communicate('password\npassword\n')

prevent subprocess.Popen from displaying output in python

So I am trying to store the output of a command into a variable. I do not want it to display output while running the command though...
The code I have right now is as follows...
def getoutput(*args):
myargs=args
listargs=[l.split(' ',1) for l in myargs]
import subprocess
output=subprocess.Popen(listargs[0], shell=False ,stdout=subprocess.PIPE)
out, error = output.communicate()
return(out,error)
def main():
a,b=getoutput("httpd -S")
if __name__ == '__main__':
main()
If I put this in a file and execute it on the command line. I get the following output even though I do not have a print statement in the code. How can I prevent this, while still storing the output?
#python ./apache.py
httpd: Could not reliably determine the server's fully qualified domain name, using xxx.xxx.xxx.xx for ServerName
Syntax OK

What you are seeing is standard-error output, not standard-output output. Stderr redirection is controlled by the stderr constructor argument. It defaults to None, which means no redirection occurs, which is why you see this output.
Usually it's a good idea to keep stderr output since it aids debugging and doesn't affect normal redirection (e.g. | and > shell redirection won't capture stderr by default). However you can redirect it somewhere else the same way you do stdout:
sp = subprocess.Popen(listargs[0], shell=False,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output, error = sp.communicate()
Or you can just drop stderr:
devnull = open(os.devnull, 'wb') #python >= 2.4
sp = subprocess.Popen(listargs[0], shell=False,
stdout=subprocess.PIPE, stderr=devnull)
#python 3.x:
sp = subprocess.Popen(listargs[0], shell=False
stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)

You're catching stdout, but you're not catching stderr(standard error) which I think is where that message is coming from.
output=subprocess.Popen(listargs[0], shell=False ,stdout=subprocess.PIPE, stderr=STDOUT)
That will put anything from stderr into the same place as stdout.

How to control a command window opened from a .cmd file using Python

There's a file named startup.cmd that sets some environment variables, runs some preparation commands, then does:
start "startup" cmd /k
Which opens a command shell named startup. The manual process I'm trying to automate is to then enter the following command into this shell: get startup.xml. I thought the correct way to do this in Python would be something like this:
import subprocess
p = subprocess.Popen('startup.cmd', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
getcommand = 'get startup.xml'
servercommand = 'startserver'
p.stdin.write(getcommand)
p.stdin.write(startserver)
(stdoutdata, stderrdata) = p.communicate()
print stdoutdata
print stderrdata
But those commands don't seem to be executing in the shell. What am I missing? Also, the command shell appears regardless of whether shell is set to True or False.

I found this warning in subprocess's document,
Warning Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.
So my suggestion is to use communicate to send your command.
import subprocess
p = subprocess.Popen('startup.cmd', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
command = 'get startup.xml\n'
command += 'startserver\n'
(stdoutdata, stderrdata) = p.communicate(command)
print stdoutdata
print stderrdata

This is a new process, so one cannot communicate directly with Popen.

Spawn subprocess that expects console input without blocking?

I am trying to do a CVS login from Python by calling the cvs.exe process.
When calling cvs.exe by hand, it prints a message to the console and then waits for the user to input the password.
When calling it with subprocess.Popen, I've noticed that the call blocks. The code is
subprocess.Popen(cvscmd, shell = True, stdin = subprocess.PIPE, stdout = subprocess.PIPE,
stderr = subprocess.PIPE)
I assume that it blocks because it's waiting for input, but my expectation was that calling Popen would return immediately and then I could call subprocess.communicate() to input the actual password. How can I achieve this behaviour and avoid blocking on Popen?
OS: Windows XP
Python: 2.6
cvs.exe: 1.11

Remove the shell=True part. Your shell has nothing to do with it. Using shell=True is a common cause of trouble.
Use a list of parameters for cmd.
Example:
cmd = ['cvs',
'-d:pserver:anonymous#bayonne.cvs.sourceforge.net:/cvsroot/bayonne',
'login']
p = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
This won't block on my system (my script continues executing).
However since cvs reads the password directly from the terminal (not from standard input or output) you can't just write the password to the subprocess' stdin.
What you could do is pass the password as part of the CVSROOT specification instead, like this:
:pserver:<user>[:<passwd>]#<server>:/<path>
I.e. a function to login to a sourceforge project:
import subprocess
def login_to_sourceforge_cvs(project, username='anonymous', password=''):
host = '%s.cvs.sourceforge.net' % project
path = '/cvsroot/%s' % project
cmd = ['cvs',
'-d:pserver:%s:%s#%s:%s' % (username, password, host, path),
'login']
p = subprocess.Popen(cmd, stdin=subprocess.PIPE,
stdout=subprocess.PIPE
stderr=subprocess.STDOUT)
return p
This works for me. Calling
login_to_sourceforge_cvs('bayonne')
Will log in anonymously to the bayonne project's cvs.

If you are automating external programs that need input - like password - your best bet would probably be to use pexpect.