We Keep Coding

Why does the command and its arguments have to be in a list for subprocess.Popen?

I tried doing
import subprocess
p = subprocess.Popen("ls -la /etc", stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p.stdout.read().decode()
Which gives me
FileNotFoundError: [Errno 2] No such file or directory: 'ls -la /etc': 'ls -la /etc'
Following
Python subprocess.Popen with var/args
I did
import subprocess
p = subprocess.Popen(["ls", "-la", "/etc"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p.stdout.read().decode()
Which did work.
Why is that? Why do I have to split my command and its arguments? What's the rationale behind this design?
Python version:
3.7.3 (default, Mar 27 2019, 22:11:17)
[GCC 7.3.0]

That's how all process invocations work on UNIX.
Under the hood, running a program on UNIX is traditionally done with the following steps:
fork() off a child process.
In that child process, open new copies of stdin, stdout, stderr, etc if redirections are requested, using the dup2() call to assign the newly-opened files over the file descriptors that are redirection targets.
In that child process, use the execve() syscall to replace the current process with the desired child process. This syscall takes an array of arguments, not a single string.
wait() for the child to exit, if the call is meant to be blocking.
So, subprocess.Popen exposes the array interface, because the array interface is what the operating system actually does under the hood.
When you run ls /tmp at a shell, that shell transforms the string into an array and then does the above steps itself -- but it gives you more control (and avoids serious bugs -- if someone creates a file named /tmp/$(rm -rf ~), you don't want trying to cat /tmp/$(rm -rf ~) to delete your home directory) when you do the transformations yourself.

According to the docs, it's dependent on the shell= keyword argument on how a string will work vs a list (Bold indicates what is likely causing your experienced behavior):
args should be a sequence of program arguments or else a single string or path-like object. By default, the program to execute is the first item in args if args is a sequence. If args is a string, the interpretation is platform-dependent and described below. See the shell and executable arguments for additional differences from the default behavior. Unless otherwise stated, it is recommended to pass args as a sequence.
On POSIX, if args is a string, the string is interpreted as the name or path of the program to execute. However, this can only be done if not passing arguments to the program.
Further down...
On POSIX with shell=True, the shell defaults to /bin/sh. If args is a string, the string specifies the command to execute through the shell. This means that the string must be formatted exactly as it would be when typed at the shell prompt. This includes, for example, quoting or backslash escaping filenames with spaces in them. If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional arguments to the shell itself. That is to say, Popen does the equivalent of:
Popen(['/bin/sh', '-c', args[0], args[1], ...])
On Windows with shell=True, the COMSPEC environment variable specifies the default shell. The only time you need to specify shell=True on Windows is when the command you wish to execute is built into the shell (e.g. dir or copy). You do not need shell=True to run a batch file or console-based executable.

Enable printing of subprocess.run in spyder ipython

I am running spyder on windows 10 and when I attempt to run a command similar to the following:
cmd = 'python /path/to/program.py arg1 arg2'
subprocess.run(cmd,shell=True)
The script is being run as expected but I would like to see what is being printed to screen by the executed command in the spyder ipython console. I know the program is printing things to screen as expected by other methods (running the program from a shell) so there is not an error in the script I am running.
How do I go about enabling printing for the subprocess?

The output comes in a stream called stdout. To capture it, you need to redirect it to a pipe, which then is terminated in the calling process. subprocess.run(...) has builtin support for handling this:
import subprocess
cmd = 'python /path/to/program.py arg1 arg2'.split()
proc = subprocess.run(cmd, stdout=subprocess.PIPE, universal_newlines=True)
print(proc.stdout)
As can be seen, the output is caught in the CompletedProcess object (proc) and then accessed as member data.Also, to make the output into text (a string) rather than a bytearray, I have passed the parameter universal_newlines=True.
A caveat, though, is that subprocess.run(...) runs to completion before it returns control. Therefore, this does not allow for capturing the output "live" but rather after the whole process has finsihed. If you want live capture, you must instead use subprocess.Popen(...) and then use .communicate() or some other means of communication to catch the output from the subprocess.
Another comment I like to make, is that using shell=True is not recommended. Specifically not when handling unknown or not trusted input. It leaves the interpretation of cmd to the shell which can lead to all kind of security breaches and bad behavior. Instead, split cmd into a list (e.g. as I have done) and then pass that list to subprocess.run(...) and leave out shell=True.

How to viewthe actual command executed by "subprocess.check_call()"

I have a command as given below
subprocess.check_call(["C:\\Program Files\\operation.exe", "execute", "-af", "createrecord.xml", " -stuName", student,"-gender" ,gender], shell=True)
When I run this command manually it works fine. I believe that the subprocess.check_call() has not parsed the command properly (probably my mistake). How to view the output of subprocess.check_call().
I'd like to see the acutal command with arguments as called by subprocess.check_call()
NOTE - I don't want to see the return value of the command's execution. I only what to see how the command has been formatted by subprocess.check_call()

You should use the list form of the subprocess stuff only with (possibly implicit) shell=False and the string form only with shell=True.
The shell can only work with a command line which it then parses on its own. Without the shell, exec*() functions are usedm, which take the command line arguments in a separated way.
For Windows, this only holds up to a certain level, but it remains valid.

Executing shell command from python

I am trying to compile a set of lines and execute them and append the output to text file. Instead of writing the same thing, I used a python script to compile and execute in background.
import subprocess
subprocess.call(["ifort","-openmp","mod1.f90","mod2.f90","pgm.f90","-o","op.o"])
subprocess.call(["nohup","./op.o",">","myout.txt","&"])
The program pgm.f90 is getting compliled using the ifort compiler, but the ouput is not getting appended to myout.txt. Instead it is appending output to nohup.out and the program is not running in the background even after specifying "&" in the python script.
What obvious error have I made here?
Thanks in advance

You can call a subprocess as if you were in the shell by using Popen() with the argument shell=True:
subprocess.Popen("nohup ./op.o > myout.txt &", shell=True)

This issue is that when you supply arguments as a list of elements, the subprocess library bypasses the shell and uses the exec syscall to directly run your program (in your case, "nohup"). Thus, rather than the ">" and "&" operators being interpreted by the shell to redirect your output and run in the background, they are being passed as literal arguments to the nohup command.
You can tell subprocess to execute your command via the shell, but this starts a whole extra instance of shell and can be wasteful. For a workaround, use the built-in redirection functionality in subprocess instead of using the shell primitives:
p = subprocess.Popen(['nohup', "./op.o"],
stdout=open('myout.txt', 'w'))
# process is now running in the background.
# if you want to wait for it to finish, use:
p.wait()
# or investigate p.poll() if you want to check to see if
# your process is still running.
For more information: http://docs.python.org/2/library/subprocess.html

Launch gnu screen from python?

I tried executing a server daemon with gnu screen from subprocess call but it didn't even start
subprocess.call(["screen", "-dmS test ./server"])
I was told that running screen requires terminal, hence the reason why I can't simply execute it with call. Can you show me some piece of codes to do this?

Try
subprocess.call( ["screen", "-d", "-m", "-S", "test", "./server"] )
You need to break the argument string into separate arguments, one per string.
Here's the relevant quote from the subprocess docs:
On UNIX, with shell=False (default): In this case, the Popen class
uses os.execvp() to execute the child program. args should normally
be a sequence. A string will be treated as a sequence with the string
as the only item (the program to execute).
On UNIX, with shell=True: If args is a string, it specifies the
command string to execute through the shell. If args is a sequence,
the first item specifies the command string, and any additional items
will be treated as additional shell arguments.
So by default, the arguments are used exactly as you give them; it doesn't try to parse a string into multiple arguments. If you set shell to true, you could try the following:
subprocess.call("screen -dmS test ./server", shell=True)
and the string would be parsed exactly like a command line.