We Keep Coding

When should you use system(3) to run an external process?

C supplies the standard function system to run a subprocess using the shell, and many languages provide similar functions, like AWK, Perl (with a single argument), and PHP. Sometimes those functions are criticized as being unsuitable for general use, either on security grounds or because the shell is not portable or is not the one used interactively.
Some other languages seem to agree: they provide only a means of running a process without the shell, like Java (which tokenizes any single string argument itself) and Tcl. Python provides both a direct wrapper and a sophisticated replacement that can avoid using the shell and explicitly recommends the latter (as does the user community).
Certainly the shell is unnecessary complexity for many applications; running an external process at all can bring in issues of deadlock, orphan processes, ambiguous exit statuses, and file descriptor sharing and is unnecessary in cases like running mkdir or echo $VAR. However, assuming that system exists for a reason, when is it the right tool to use?

Even assuming a use case for which it's appropriate to run an external process and in particular to run one via the shell (without being able to filter output as with popen), for C and Python (that uses the actual C system(3)) there are additional caveats. POSIX specifies additional behavior for system: it ignores SIGINT and SIGQUIT and blocks SIGCHLD during its execution. The rationale is that the user (who can send SIGINT and SIGQUIT from the terminal) is interacting with the subprocess, not the parent, during its execution, and that system must handle the SIGCHLD for its child process without the application's interference.
This directly implies the answer to the question: it is appropriate to use system only when
The user has directly asked for a particular shell command to be executed (e.g., with ! in less), and
The application need not react to any other child process exiting during this time (e.g, it should not be multithreaded).
If #1 is not satisfied, the user is likely to send a terminal signal expecting it to kill the whole process and have it kill only the (unexpected if not invisible) child. The Linux man pages caution particularly about using it in a loop that the user cannot then interrupt. It is possible to notice that a child has exited with a signal and reraise it, but this is unreliable because some programs (e.g., Python) exit upon receiving certain signals rather than reraising it to indicate why they exited—and because the shell (mandated by system!) conflates exit statuses with signal-kill statuses.
In Python the error-handling problems are compounded by the fact that os.system follows the C exit-status (read: error code) convention instead of reporting failure as an exception, inviting the user to ignore the exit status of the child.

The answer is simple (in theory), because it's the same answer that applies to many other programming questions: it's appropriate to use system() when it makes the programmer's life easier, and makes the user's life no harder.
Spotting when this is true, however, requires considerable judgement, and probably we won't always get it right. But, again, that's true of many judgement calls in programming.
Since most shells are written in C, there's no reason in principle why anything done using system() can't be done without it. However, sometimes it requires a whole heap of coding to do what can be done in one line by invoking a shell. The same applies to popen() which, I guess, raises exactly the same kinds of questions.
Using system() raises portability, thread safety, and signal-management concerns.
My experience, unfortunately, is that the situations where system() gives the most benefit (to the programmer) are precisely the ones where it will be least portable.
Sometimes concerns like this will suggest a different approach, and sometimes they won't matter -- it depends on the application.

Interprocess communication with a modified python interpreter

TL;DR: How can I spawn a different python interpreter (from within python) and create a communication channel between the parent and child when stdin/stdout are unavailable?
I would like my python script to execute a modified python interpreter and through some kind of IPC such as multiprocessing.Pipe communicate with the script that interpreter runs.
Lets say I've got something similar to the following:
subprocess.Popen(args=["/my_modified_python_interpreter.exe",
"--my_additional_flag",
"my_python_script.py"])
Which works fine and well, executes my python script and all.
I would now like to set up some kind of interprocess communication with that modified python interpreter.
Ideally, I would like to share something similar to one of the returned values from multiprocessing.Pipe(), however I will need to share that object with the modified python process (and I suspect multiprocessing.Pipe won't handle that well even if I do that).
Although sending text and binary will be sufficient (I don't need to share python objects or anything), I do need this to be functional on all major OSes (windows, Linux, Mac).
Some more use-case/business explanation
More specifically, the modified interpreter is the IDAPython interpreter that is shipped with IDA to allow scripting within the IDA tool.
Unfortunately, since stdio is already heavily used for the existing user interface functionalities (provided by IDA), I cannot use stdin/stdout for the communication.
I'm searching for possibilities that are better than the one's I thought of:
Use two (rx and tx channels) hard-disk files and pass paths to both as the arguments.
Use a local socket and pass a path as an argument.
Use a memory mapped file and the tagname on windows and some other sync method on other OSes.

After some tinkering with the multiprocessing.Pipe function and the multiprocesing.Connection objects it returns, I realized that serialization of Connection objects is far simpler that I originally thought.
A Connection object has three descripting properties:
fileno - A handle. An arbitrary file descriptor on Unix and a socket on windows.
readable - A boolean controlling whether Connection object can be read.
writable - A boolean controlling whether Connection object can be written.
All three properties are accessible as object attributes and are controllable through the Connection class constructor.
It appears that if:
The process calling Pipe spawns a child process and shares the connection.fileno() number.
The child process creates a Connection object using that file descriptor as the handle.
Both interpreters implement the Connection object roughly the same (And this is the risky part, I guess).
It is possible to Connection.send and Connection.recv between those two processes although they do not share the same interpreter build and the multiprocessing module was not actually used to instantiate the child process.
EDIT:
Please note the Connection class is available as multiprocessing.connection.Connection in python3 and as _multiprocessing.Connection in python2 (which might suggest it's usage is discouraged. YMMV)

Going with the other answer of mine turned out to be a mistake. Because of how handles are inherited in python2 on Windows I couldn't get the same solution to work on Windows machines. I ended up using the far superior Listener and Client interfaces also found in the multiprocessing module.
This question of mine discusses that mistake.

What's the difference between Python's subprocess.call and subprocess.run

I've been trying to understand for a while now what's the difference between subprocess.call and subprocess.run. I know the last one is new on Python 3.5 and both are based on subprocess.Popen, but I'm not able to understand the difference yet.

The definition of subprocess.call() clearly mentions:
It is equivalent to:
run(...).returncode
(except that the input and check parameters are not supported)
As the Python 3.5's subprocess document says:
Prior to Python 3.5, these three functions (i.e. .call(), .check_call(), .check_output()) comprised the high level API to subprocess. You can now use run() in many cases, but lots of existing code calls these functions.
It is a common practice that when some functions are replaced, they are not instantly deprecated but there is a support window for them for some versions. This helps in preventing the breakage of older code when the language version is upgraded. I do not know whether .call() is going to be replaced in the future or not. But based on the document, what I know is that they are pretty much same.

To make it clear for anyone wanting to know which to use:
subprocess.run() is the recommended approach for all use cases it can handle. The suprocess documentation states:
The recommended approach to invoking subprocesses is to use the run() function for all use cases it can handle. For more advanced use cases, the underlying Popen interface can be used directly.
subprocess.call() is part of the Older high-level API (Prior to Python 3.5).

I'm not sure I agree with the other answers.
I just had a very frustrating time with a bash script which starts a daemon process (Elasticsearch). The command merely supplies the path to the executable Bash script.
But subprocess.run(...) does not return from this, whereas subprocess.call(...) does.
From my experience, if you then stop the process (e.g. the Terminal if running from a Terminal) using subprocess.run(...) this kills off the daemon process started in it. But this is not the case with subprocess.call(...): the daemon carries on happily.
In both cases I set the kwarg shell=True.
I also tried subprocess.run ẁith shell=False (i.e. default if you omit shell): no change.
I can't see any other possible options in subprocess.run which might overcome this, so it appears, as far as I can tell that subprocess.call is fundamentally different, despite what the docs appear to say. At the time of writing the docs say "You can now use run() in many cases, but lots of existing code calls these functions." (i.e. the older functions, including call).
What is particularly strange, and frustrating, is that (obviously) when you run a script which starts a daemon, such as:
./bin/elasticsearch -d -p pid
... it just returns and you can close the Terminal quite happily. So there appears something quite odd about subprocess.run, which some super-expert might care to explain.

I am not fully clear on the differences either.
I can say that you use subprocess.call() when you want the program to wait for the process to complete before moving onto the next process. In the case of subprocess.run(), the program will attempt to run all the processes at once, inevitably causing the program to crash.

Resume thread of process created suspended, having lost the thread handle

On Windows, Python (2)'s standard library routine subprocess.Popen allows you to specify arbitrary flags to CreateProcess, and you can access the process handle for the newly-created process from the object that Popen returns. However, the thread handle for the newly-created process's initial thread is closed by the library before Popen returns.
Now, I need to create a process suspended (CREATE_SUSPENDED in creation flags) so that I can manipulate it (specifically, attach it to a job object) before it has a chance to execute any code. However, that means I need the thread handle in order to release the process from suspension (using ResumeThread). The only way I can find, to recover the thread handle, is to use the "tool help" library to walk over all threads on the entire system (e.g. see this question and answer). This works, but I do not like it. Specifically, I am concerned that taking a snapshot of all the threads on the system every time I need to create a process will be too expensive. (The larger application is a test suite, using processes for isolation; it creates and destroys processes at a rate of tens to hundreds a second.)
So, the question is: is there a more efficient way to resume execution of a process that was suspended by CREATE_SUSPENDED, if all you have is the process handle, and the facilities of the Python 2 standard library (including ctypes, but not the winapi add-on)? Vista-and-higher techniques are acceptable, but XP compatibility is preferred.

I have found a faster approach; unfortunately it relies on an undocumented API, NtResumeProcess. This does exactly what it sounds like - takes a process handle and applies the equivalent of ResumeThread to every thread in the process. Python/ctypes code to use it looks something like
import ctypes
from ctypes.wintypes import HANDLE, LONG, ULONG
ntdll = ctypes.WinDLL("ntdll.dll")
RtlNtStatusToDosError = ntdll.RtlNtStatusToDosError
NtResumeProcess = ntdll.NtResumeProcess
def errcheck_ntstatus(status, *etc):
if status < 0: raise ctypes.WinError(RtlNtStatusToDosError(status))
return status
RtlNtStatusToDosError.argtypes = (LONG,)
RtlNtStatusToDosError.restype = ULONG
# RtlNtStatusToDosError cannot fail
NtResumeProcess.argtypes = (HANDLE,)
NtResumeProcess.restype = LONG
NtResumeProcess.errcheck = errcheck_ntstatus
def resume_subprocess(proc):
NtResumeProcess(int(proc._handle))
I measured approximately 20% less process setup overhead using this technique than using Toolhelp, on an otherwise-idle Windows 7 virtual machine. As expected given how Toolhelp works, the performance delta gets bigger the more threads exist on the system -- whether or not they have anything to do with the program in question.
Given the obvious general utility of NtResumeProcess and its counterpart NtSuspendProcess, I am left wondering why they have never been documented and given kernel32 wrappers. They are used by a handful of core system DLLs and EXEs all of which, AFAICT, are part of the Windows Error Reporting mechanism (faultrep.dll, werui.dll, werfault.exe, dwwin.exe, etc) and don't appear to re-expose the functionality under documented names. It seems unlikely that these functions would change their semantics without also changing their names, but a defensively-coded program should probably be prepared for them to disappear (falling back to toolhelp, I suppose).

I'm posting this here, because I found something that addresses this question. I'm looking into this myself and I believe that I've found the solution with this.
I can't give you an excerpt or a summary, because it's just too much and I found it just two hours ago. I'm posting this here for all the others who, like me, seek a way to "easily" spawn a proper child process in windows, but want to execute a cuckoo instead. ;)
The whole second chapter is of importance, but the specifics start at page 12.
http://lsd-pl.net/winasm.pdf
I hope that it helps others as much as it hopefully going to help me.
Edit:
I guess I can add more to it. From what I've gathered, does this document explain how to spawn a sleeping process which never gets executed. This way we have a properly set-up windows process running. Then it explains that by using the win32api functions VirtualAllocEx and WriteProcessMemory, we can easily allocate executable pages and inject machine code into the other process.
Then - the best part in my opinion - it's possible to change the registers of the process, allowing the programmer to change the instruction pointer to point at the cuckoo!
Amazing!

How to write a system agnostic Python daemon/service? [duplicate]

I would like to have my Python program run in the background as a daemon, on either Windows or Unix. I see that the python-daemon package is for Unix only; is there an alternative for cross platform? If possible, I would like to keep the code as simple as I can.

In Windows it's called a "service" and you could implement it pretty easily e.g. with the win32serviceutil module, part of pywin32. Unfortunately the two "mental models" -- service vs daemon -- are very different in detail, even though they serve similar purposes, and I know of no Python facade that tries to unify them into a single framework.

This question is 6 years old, but I had the same problem, and the existing answers weren't cross-platform enough for my use case. Though Windows services are often used in similar ways as Unix daemons, at the end of the day they differ substantially, and "the devil's in the details". Long story short, I set out to try and find something that allows me to run the exact same application code on both Unix and Windows, while fulfilling the expectations for a well-behaved Unix daemon (which is better explained elsewhere) as best as possible on both platforms:
Close open file descriptors (typically all of them, but some applications may need to protect some descriptors from closure)
Change the working directory for the process to a suitable location to prevent "Directory Busy" errors
Change the file access creation mask (os.umask in the Python world)
Move the application into the background and make it dissociate itself from the initiating process
Completely divorce from the terminal, including redirecting STDIN, STDOUT, and STDERR to different streams (often DEVNULL), and prevent reacquisition of a controlling terminal
Handle signals, in particular, SIGTERM.
The fundamental problem with cross-platform daemonization is that Windows, as an operating system, really doesn't support the notion of a daemon: applications that start from a terminal (or in any other interactive context, including launching from Explorer, etc) will continue to run with a visible window, unless the controlling application (in this example, Python) has included a windowless GUI. Furthermore, Windows signal handling is woefully inadequate, and attempts to send signals to an independent Python process (as opposed to a subprocess, which would not survive terminal closure) will almost always result in the immediate exit of that Python process without any cleanup (no finally:, no atexit, no __del__, etc).
Windows services (though a viable alternative in many cases) were basically out of the question for me: they aren't cross-platform, and they're going to require code modification. pythonw.exe (a windowless version of Python that ships with all recent Windows Python binaries) is closer, but it still doesn't quite make the cut: in particular, it fails to improve the situation for signal handling, and you still cannot easily launch a pythonw.exe application from the terminal and interact with it during startup (for example, to deliver dynamic startup arguments to your script, say, perhaps, a password, file path, etc), before "daemonizing".
In the end, I settled on using subprocess.Popen with the creationflags=subprocess.CREATE_NEW_PROCESS_GROUP keyword to create an independent, windowless process:
import subprocess
independent_process = subprocess.Popen(
'/path/to/pythonw.exe /path/to/file.py',
creationflags=subprocess.CREATE_NEW_PROCESS_GROUP
)
However, that still left me with the added challenge of startup communications and signal handling. Without going into a ton of detail, for the former, my strategy was:
pickle the important parts of the launching process' namespace
Store that in a tempfile
Add the path to that file in the daughter process' environment before launching
Extract and return the namespace from the "daemonization" function
For signal handling I had to get a bit more creative. Within the "daemonized" process:
Ignore signals in the daemon process, since, as mentioned, they all terminate the process immediately and without cleanup
Create a new thread to manage signal handling
That thread launches daughter signal-handling processes and waits for them to complete
External applications send signals to the daughter signal-handling process, causing it to terminate and complete
Those processes then use the signal number as their return code
The signal handling thread reads the return code, and then calls either a user-defined signal handler, or uses a cytpes API to raise an appropriate exception within the Python main thread
Rinse and repeat for new signals
That all being said, for anyone encountering this problem in the future, I've rolled a library called daemoniker that wraps both proper Unix daemonization and the above Windows strategy into a unified facade. The cross-platform API looks like this:
from daemoniker import Daemonizer
with Daemonizer() as (is_setup, daemonizer):
if is_setup:
# This code is run before daemonization.
do_things_here()
# We need to explicitly pass resources to the daemon; other variables
# may not be correct
is_parent, my_arg1, my_arg2 = daemonizer(
path_to_pid_file,
my_arg1,
my_arg2
)
if is_parent:
# Run code in the parent after daemonization
parent_only_code()
# We are now daemonized, and the parent just exited.
code_continues_here()

Two options come to mind:
Port your program into a windows service. You can probably share much of your code between the two implementations.
Does your program really use any daemon functionality? If not, you rewrite it as a simple server that runs in the background, manages communications through sockets, and perform its tasks. It will probably consume more system resources than a daemon would, but it would be quote platform independent.

In general the concept of a daemon is Unix specific, in particular expected behaviour with respect to file creation masks, process hierarchy, and signal handling.
You may find PEP 3143 useful wherein a proposed continuation of python-daemon is considered for Python 3.2, and many related daemonizing modules and implementations are discussed.

The reason it's unix only is that daemons are a Unix specific concept i.e a background process initiated by the os and usually running as a child of the root PID .
Windows has no direct equivalent of a unix daemon, the closest I can think of is a Windows Service.
There's a program called pythonservice.exe for windows . Not sure if it's supported on all versions of python though