I am running spyder on windows 10 and when I attempt to run a command similar to the following:
cmd = 'python /path/to/program.py arg1 arg2'
subprocess.run(cmd,shell=True)
The script is being run as expected but I would like to see what is being printed to screen by the executed command in the spyder ipython console. I know the program is printing things to screen as expected by other methods (running the program from a shell) so there is not an error in the script I am running.
How do I go about enabling printing for the subprocess?
The output comes in a stream called stdout. To capture it, you need to redirect it to a pipe, which then is terminated in the calling process. subprocess.run(...) has builtin support for handling this:
import subprocess
cmd = 'python /path/to/program.py arg1 arg2'.split()
proc = subprocess.run(cmd, stdout=subprocess.PIPE, universal_newlines=True)
print(proc.stdout)
As can be seen, the output is caught in the CompletedProcess object (proc) and then accessed as member data.Also, to make the output into text (a string) rather than a bytearray, I have passed the parameter universal_newlines=True.
A caveat, though, is that subprocess.run(...) runs to completion before it returns control. Therefore, this does not allow for capturing the output "live" but rather after the whole process has finsihed. If you want live capture, you must instead use subprocess.Popen(...) and then use .communicate() or some other means of communication to catch the output from the subprocess.
Another comment I like to make, is that using shell=True is not recommended. Specifically not when handling unknown or not trusted input. It leaves the interpretation of cmd to the shell which can lead to all kind of security breaches and bad behavior. Instead, split cmd into a list (e.g. as I have done) and then pass that list to subprocess.run(...) and leave out shell=True.
Related
I'm trying to include sequence alignment using muscle or mafft, depending of the user in a pipeline.
To do so, i'm using the subprocess package, but sometimes, the subprocess never terminates and my script doesn't continue. Here is how I call the subprocess:
child = subprocess.Popen(str(muscle_cline), stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
child.wait()
The command muscle_cline looks like this:
./tools/muscle/muscle5.1.win64.exe -align C:\Users\alexis\Desktop\git-repo\MitoSplitter\results\genes-fasta\12S_tmp.fasta -output C:\Users\alexis\Desktop\git-repo\MitoSplitter\results\alignement\12S_tmp_muscle_align.fasta
I'm calling this line in a function that just creates the command line and calls the subprocess, and converts the output.
I'm then calling this function in a for loop
for file in getFastaFile(my_dir):
alignSequenceWithMuscle(file)
The issue is that sometimes, for unknown reasons, the subprocess never finishes and get locked...
I tried to check the returncode of the child, or print stuff to see where it gets locked, and it's getting locked when I'm calling the subprocess.
Any ideas?
You generally want to avoid bare Popen, especially if you don't have a good understanding of its requirements. This is precisely why Python offers you subprocess.check_output and other higher-level functions which take care of the nitty-gritty of managing a subprocess.
output = subprocess.check_output(
["./tools/muscle/muscle5.1.win64.exe",
"-align", r"C:\Users\alexis\Desktop\git-repo\MitoSplitter\results\genes-fasta\12S_tmp.fasta",
"-output", r"C:\Users\alexis\Desktop\git-repo\MitoSplitter\results\alignement\12S_tmp_muscle_align.fasta"],
text=True)
Notice also the raw strings r"..." to avoid having to double the backslashes, and the text=True keyword argument to instruct Python to implicitly decode the bytes you receive from the subprocess.
I am using popen to run the following command on a windows vm
'tf changeset ...'
however when I run it using
commandLine = 'tf changeset /noprompt /latest /loginType:OAuth /login:.,***'
process = Popen(commandLine, shell=True, stdout=PIPE, stderr=PIPE)
I see the following being executed in the logs
'C:\Azure\Agent-1/externals/tf/tf changeset ...'
Meaning that 'C:\Azure\Agent-1/externals/tf/' has been prepended to my command. I was just expecting to see
'tf changeset ...'
Unfortunately adding the path to the execution breaks the command, is there any way to stop python from doing this?
Try passing the commandLine to Popen as a list of arguments:
commandLine = ["tf", "changeset", "/noprompt", "/latest", "/loginType:OAuth", "/login:.,***'"]
process = Popen(commandLine, stdout=PIPE, stderr=PIPE)
Python by itself does no such thing. Perhaps the shell=True is doing more than you hoped or bargained for? But we would need access to your shell's configuration to get beyond mere speculation around this.
Calling Popen on the result from Popen is obviously not well-defined; but perhaps this is just an error in your transcription of your real code?
Removing the first process =Popen( would fix this with minimal changes. As per the above, I would also remove shell=True as at least superfluous and at worst directly harmful.
commandLine = 'tf changeset /noprompt /latest /loginType:OAuth /login:.,***'
process = Popen(commandLine, stdout=PIPE, stderr=PIPE)
Like the subprocess documentation tells you, shell=True is only useful on Windows when your command is a cmd built-in.
For proper portability, you should break the command into tokens, manually or by way of shlex.split() if you are lazy or need the user to pass in a string to execute.
commandLine = ['tf ', 'changeset', '/noprompt', '/latest', '/loginType:OAuth', '/login:.,***']
process = Popen(commandLine, stdout=PIPE, stderr=PIPE)
This avoids the other dangers of shell=True and will be portable to non-Windows platforms (assuming of course that the command you are trying to run is available on the target platform).
I am trying to create a program to easily handle IT requests, and I have created a program to test if a PC on my network is active from a list.
To do this, I wrote the following code:
self.btn_Ping.clicked.connect(self.ping)
def ping(self):
hostname = self.listWidget.currentItem().text()
if hostname:
os.system("ping " + hostname + " -t")
When I run it my main program freezes and I can't do anything until I close the ping command window. What can I do about this? Is there any other command I can use to try to ping a machine without making my main program freeze?
The docs state that os.system() returns the value returned by the command you called, therefore blocking your program until it exits.
They also state that you should use the subprocess module instead.
From ping documentation:
ping /?
Options:
-t Ping the specified host until stopped.
To see statistics and continue - type Control-Break;
To stop - type Control-C.
So, by using -t you are waiting until that machine has stopped, and in case that machine is not stopping, your Python script will run forever.
As mentioned by HyperTrashPanda, use another parameter for launching ping, so that it stops after one or some attempts.
As mentioned in Tim Pietzcker's answer, the use of subprocess is highly recommended over os.system (and others).
To separate the new process from your script, use subprocess.Popen. You should get the output printed normally into sys.stdout. If you want something more complex (e.g. for only printing something if something changes), you can set the stdout (and stderr and stdin) arguments:
Valid values are PIPE, DEVNULL, an existing file descriptor (a positive integer), an existing file object, and None. PIPE indicates that a new pipe to the child should be created. DEVNULL indicates that the special file os.devnull will be used. With the default settings of None, no redirection will occur; the child’s file handles will be inherited from the parent.
-- docs on subproces.Popen, if you scroll down
If you want to get the exit code, use myPopenProcess.poll().
I have a python script that launches subprocesses using subprocess.Popen. The subprocess then launches an external command (in my case, it plays an mp3). The python script needs to be able to interrupt the subprocesses, so I used the method described here which gives the subprocess its own session ID. Unfortunately, when I close the python script now, the subprocess will continue to run.
How can I make sure a subprocess launched from a script, but given a different session ID still closes when the python script stops?
Is there any way to kill a Thread in Python?
and make sure you use it as thread
import threading
from subprocess import call
def thread_second():
call(["python", "secondscript.py"])
processThread = threading.Thread(target=thread_second) # <- note extra ','
processThread.start()
print 'the file is run in the background'
TL;DR Change the Popen params: Split up the Popen cmd (ex. "list -l" -> ["list", "-l"]) and use Shell=False
~~~
The best solution I've seen so far was just not to use shell=True as an argument for Popen, this worked because I didn't really need shell=True, I was simply using it because Popen wouldn't recognize my cmd string and I was too lazy too split it into a list of args. This caused me a lot of other problems (ex. using .terminate() becomes a lot more complicated while using shell and needs to have its session id, see here)
Simply splitting the cmd from a string to a list of args lets me use Popen.terminate() without having to give it its own session id, by not having a separate session id the process will be closed when the python script is stopped
I am trying to compile a set of lines and execute them and append the output to text file. Instead of writing the same thing, I used a python script to compile and execute in background.
import subprocess
subprocess.call(["ifort","-openmp","mod1.f90","mod2.f90","pgm.f90","-o","op.o"])
subprocess.call(["nohup","./op.o",">","myout.txt","&"])
The program pgm.f90 is getting compliled using the ifort compiler, but the ouput is not getting appended to myout.txt. Instead it is appending output to nohup.out and the program is not running in the background even after specifying "&" in the python script.
What obvious error have I made here?
Thanks in advance
You can call a subprocess as if you were in the shell by using Popen() with the argument shell=True:
subprocess.Popen("nohup ./op.o > myout.txt &", shell=True)
This issue is that when you supply arguments as a list of elements, the subprocess library bypasses the shell and uses the exec syscall to directly run your program (in your case, "nohup"). Thus, rather than the ">" and "&" operators being interpreted by the shell to redirect your output and run in the background, they are being passed as literal arguments to the nohup command.
You can tell subprocess to execute your command via the shell, but this starts a whole extra instance of shell and can be wasteful. For a workaround, use the built-in redirection functionality in subprocess instead of using the shell primitives:
p = subprocess.Popen(['nohup', "./op.o"],
stdout=open('myout.txt', 'w'))
# process is now running in the background.
# if you want to wait for it to finish, use:
p.wait()
# or investigate p.poll() if you want to check to see if
# your process is still running.
For more information: http://docs.python.org/2/library/subprocess.html