We Keep Coding

Python & subprocess - Open terminal session as user and execute one/two commands

As much as I hate regurgitating questions, it's a necessary evil to achieve a result to the next issue I'll present.
Using python3, tkinter and the subprocess package, my goal is to write a control panel to start and stop different terminal windows with a specific set of commands to run applications/sessions of the ROS application stack, including the core.
As such, the code would look like this per executable I wish to control:
class TestProc(object):
def __init__(self):
pass
def start(self):
self.process = subprocess.Popen(["gnome-terminal", "-c", "'cd /path/to/executable/script.sh; ./script.sh'"])
print("Process started.")
def stop(self):
self.process.terminate()
print("Process terminated.")
Currently, it is possible to start a terminal window and the assigned commands/processes, yet two issues persist:
gnome-terminal is set to launch a terminal window, then relieve control to the processes inside; as such, I have no further control once it has started. A possible solution for this is to use xterm yet that poses a slew of other issues. I am required to have variables from the user's .bashrc and/or export
Certain "global commands" eg. cd or roslaunch would be unavailable to the terminal sessions, perhaps due to the order of execution (eg. the commands are run before the bash profile is loaded) preventing any usable terminal at all
Thus, the question rings: How would I be able to start and stop a new terminal window that would run up to two commands/processes in the user environment?

There are a couple approaches you can take, the most flexible here is also the most complicated, so you'd want to consider whether you need to do it.
If you only need to show the output of the script, you can simply pipe the output to a file or to a named pipe. You can then capture that output by reading/tailing the file. This is simplest, as long as the script don't actually need to have any user interaction.
If you really only need to spawn a script that runs in the background, and you need to simulate user interaction but you don't actually need to accept actual user input, you can use expect approach (using the pexpect library).
If you need to actually allow the real user to interact with the program, then you have two approaches. First is that you can embed the VTE widget into your application, this is the most seamless integration as it'll make the terminal look seamless with your application, however it's also the most heavy.
Another approach is to start gnome-terminal as you've done here, this necessarily spawns a new window.
If you need to both script some interaction while also allowing some user input, you can do this by spawning your script in a tmux session. Using tmux send-keys command to automate the moon interactive part, and then spawn a terminal emulator for users to interact with tmux attach. If you need to go back and forth between automated part and interactive part, you can combine this approach with expect.

Python launch external process in its own window on any platform?

I'm hoping this isn't a dupe. I've searched and searched and tried suggestions from many posts here, but so far I can't find what I'm looking for.
I have a Python 2.7 app which communicates via a second app via sockets. I would like to make it so that when the first Python app is launched, it automatically starts the second one.
The challenge is I'd really like the second app to pop up in its own console window with its own console logging. The catch is my app is used by people on different platforms, so I'd like something that's platform-independent, and so far all I have found that do this use platform-specific things. So I wonder, is this something that's even possible?
The closest thing I've found is to use subprocess.Popen, like this:
subprocess.Popen(['python', 'myscript.py'],
stdout=None,
stderr=None,
stdin=None)
(I know that I could use sys.executable instead of 'python', but in some cases the second app that needs to run won't be a Python app.)
Anyway the above works except that the output of the second app shows up co-mingled in the same window as the first app. I tried to shell=True to the args for Popen (and then I switched the args to a string as recommended in the Popen documentation), like this:
subprocess.Popen(['python myscript.py'],
stdout=None,
stderr=None,
stdin=None,
shell=True
)
The result of that (at least on my Mac) is the same as before. The second Python app starts, but its output is co-mingled into the existing console window instead of launching a new one. (And on Windows I can't get that to work at all. I just get an error that "python myscript.py" is not a valid command. Same result if I change the string to "python.exe myscript.py")
I don't need any communication between the apps since I already have everything I need via the sockets. (My app is a game controller engine that connects to an external media controller which could be Unity, Unreal, Python, etc.)
I tried various incarnations of the "old" ways, including os.spawn, os.system and a bunch of other things, but none of them give the behavior I'm hoping for. I also tried to use the subprocess convenience functions, but as far as I can tell those all block until the subprocess terminates.
Frankly at this point the best I can think of is just using a batch file (though that would have to be OS-specific as far as I know)?
So I wonder if anyone knows whether want I want to do is possible?
Thanks!
Brian

How to start daemon process from python on windows?

Can my python script spawn a process that will run indefinitely?
I'm not too familiar with python, nor with spawning deamons, so I cam up with this:
si = subprocess.STARTUPINFO()
si.dwFlags = subprocess.CREATE_NEW_PROCESS_GROUP | subprocess.CREATE_NEW_CONSOLE
subprocess.Popen(executable, close_fds = True, startupinfo = si)
The process continues to run past python.exe, but is closed as soon as I close the cmd window.

Using the answer Janne Karila pointed out this is how you can run a process that doen't die when its parent dies, no need to use the win32process module.
DETACHED_PROCESS = 8
subprocess.Popen(executable, creationflags=DETACHED_PROCESS, close_fds=True)
DETACHED_PROCESS is a Process Creation Flag that is passed to the underlying CreateProcess function.

This question was asked 3 years ago, and though the fundamental details of the answer haven't changed, given its prevalence in "Windows Python daemon" searches, I thought it might be helpful to add some discussion for the benefit of future Google arrivees.
There are really two parts to the question:
Can a Python script spawn an independent process that will run indefinitely?
Can a Python script act like a Unix daemon on a Windows system?
The answer to the first is an unambiguous yes; as already pointed out; using subprocess.Popen with the creationflags=subprocess.CREATE_NEW_PROCESS_GROUP keyword will suffice:
import subprocess
independent_process = subprocess.Popen(
'python /path/to/file.py',
creationflags=subprocess.CREATE_NEW_PROCESS_GROUP
)
Note that, at least in my experience, CREATE_NEW_CONSOLE is not necessary here.
That being said, the behavior of this strategy isn't quite the same as what you'd expect from a Unix daemon. What constitutes a well-behaved Unix daemon is better explained elsewhere, but to summarize:
Close open file descriptors (typically all of them, but some applications may need to protect some descriptors from closure)
Change the working directory for the process to a suitable location to prevent "Directory Busy" errors
Change the file access creation mask (os.umask in the Python world)
Move the application into the background and make it dissociate itself from the initiating process
Completely divorce from the terminal, including redirecting STDIN, STDOUT, and STDERR to different streams (often DEVNULL), and prevent reacquisition of a controlling terminal
Handle signals, in particular, SIGTERM.
The reality of the situation is that Windows, as an operating system, really doesn't support the notion of a daemon: applications that start from a terminal (or in any other interactive context, including launching from Explorer, etc) will continue to run with a visible window, unless the controlling application (in this example, Python) has included a windowless GUI. Furthermore, Windows signal handling is woefully inadequate, and attempts to send signals to an independent Python process (as opposed to a subprocess, which would not survive terminal closure) will almost always result in the immediate exit of that Python process without any cleanup (no finally:, no atexit, no __del__, etc).
Rolling your application into a Windows service, though a viable alternative in many cases, also doesn't quite fit. The same is true of using pythonw.exe (a windowless version of Python that ships with all recent Windows Python binaries). In particular, they fail to improve the situation for signal handling, and they cannot easily launch an application from a terminal and interact with it during startup (for example, to deliver dynamic startup arguments to your script, say, perhaps, a password, file path, etc), before "daemonizing". Additionally, Windows services require installation, which -- though perfectly possible to do quickly at runtime when you first call up your "daemon" -- modifies the user's system (registry, etc), which would be highly unexpected if you're coming from a Unix world.
In light of that, I would argue that launching a pythonw.exe subprocess using subprocess.CREATE_NEW_PROCESS_GROUP is probably the closest Windows equivalent for a Python process to emulate a traditional Unix daemon. However, that still leaves you with the added challenge of signal handling and startup communications (not to mention making your code platform-dependent, which is always frustrating).
That all being said, for anyone encountering this problem in the future, I've rolled a library called daemoniker that wraps both proper Unix daemonization and the above strategy. It also implements signal handling (for both Unix and Windows systems), and allows you to pass objects to the "daemon" process using pickle. Best of all, it has a cross-platform API:
from daemoniker import Daemonizer
with Daemonizer() as (is_setup, daemonizer):
if is_setup:
# This code is run before daemonization.
do_things_here()
# We need to explicitly pass resources to the daemon; other variables
# may not be correct
is_parent, my_arg1, my_arg2 = daemonizer(
path_to_pid_file,
my_arg1,
my_arg2
)
if is_parent:
# Run code in the parent after daemonization
parent_only_code()
# We are now daemonized, and the parent just exited.
code_continues_here()

For that purpose you could daemonize your python process or as you are using windows environment you would like to run this as a windows service.
You know i like to hate posting only web-links:
But for more information according to your requirement:
A simple way to implement Windows Service. read all comments it will resolve any doubt
If you really want to learn more
First read this
what is daemon process or creating-a-daemon-the-python-way
update:
Subprocess is not the right way to achieve this kind of thing

Re-read environment of parent process in python

I've written a little Python (2.7.2+) module (called TWProcessing) that can be described as an improvised batch manager. The way it works is that I pass it a long list of commands that it will then run in parallel, but limiting the total number of simultaneous processes. That way, if I have 500 commands I would like to run, it will loop through all of them, but only running X of them at a time so as to not overwhelm the machine. The value of X can be easily set when declaring an instance of this batch manager (the class is called TWBatchManager) :
batch = TWProcessing.TWBatchManager(MaxJobs=X)
I then add a list of jobs to this object in a very straightforward manner :
batch.Queue.append(/CMD goes here/)
Where Queue is a list of commands that the batch manager will run. When the queue has been filled, I then call Run() which loops through all the commands, only running X at a time :
batch.Run()
So far, everything works fine. Now what I'd like to do is be able to change the value of X (i.e. the maximum number of processes running at once) dynamically i.e. while the processes are still running. My old way of doing this was rather straightforward. I had a file called MAXJOBS that the class would know to look at, and, if it existed, it would check it regularly to see if the desired value has changed. Now I'd like to try something a bit more elegant. I would like to be able to write something along the lines of export MAXJOBS=newX in the bash shell that launched the script containing the batch manager, and have the batch manager realize that this is now the value of X it should be using. Obviously os.environ['MAXJOBS'] is not what I'm looking for, because this is a dictionary that is loaded on startup. os.getenv('MAXJOBS') doesn't cut it either, because the export will only affect child processes that the shell will spawn from then on. So what I need is a way to get back to the environment of the parent process that launched my python script. I know os.ppid will give me the parent pid, but I have no idea how to get from there to the parent environment. I've poked around the interwebz to see if there was a way in which the parent shell could modify the child process environment, and I've found that people tend to insist I not try anything like that, lest I be prepared to do some of the ugliest things one can possibly do with a computer.
Any ideas on how to pull this off? Granted my "read from a standard text file" idea is not so ugly, but I'm new to Python and am therefore trying to challenge myself to do things in an elegant and clean manner to learn as much as I can. Thanks in advance for your help.

For me it looks that you are asking for inter-process communication between a bash script and a python program.
I'm not completely sure about all your requirements, but it might be a candidate for a FIFO (named pipe):
1) make the fifo:
mkfifo batch_control
2) Start the python - server, which reads from the fifo. (Note: the following is only a minimalistic example; you must adapt things:
while True:
fd = file("batch_control", "r")
for cmd in fd:
print("New command [%s]" % cmd[:-1])
fd.close()
3) From the bash script you can than 'send' things to the python server by echo-ing strings into the fifo:
$ echo "newsize 800" >batch_control
$ echo "newjob /bin/ps" >batch_control
The output of the python server is:
New command [newsize 800]
New command [newjob /bin/ps]
Hope this helps.

Python: Running Daemon Processes in Windows7

I had a program that Scraped certain data from certain Web-Pages, and when the Web-Pages changed, acted accordingly.
How would one set up the program so it continues to run in the background?
I don't need any specifics
I'm just really confused on this concept and would appreciate whatever help anybody has to offer.

start path-to-pythonw.exe your-code.py
pythonw means without console.
start means start on background.
if your python is installed system-wide, you can probably start your-code.pyw
.pyw is associated with pythonw.exe
remember you cannot use print (to stdout) in this case.

If you want to be able to just start your process and have it background itself and do a few more typical things that "daemon" processes do in Unix, look here: How do you create a daemon in Python?

There is no concept of "background" in Windows. But the UNIX shell concept of a background process can be reasonably emulated by running your Python script as a Windows service. There are a couple of suggestions in this question: Is it possible to run a Python script as a service in Windows? If possible, how?
For casual use, I suggest that you learn how to use srvany from the second answer.

You simply need to leave your program running! Please google "python daemon" and see how to implement a persistent background process in Python.
Now, you cannot know when a website changes unless you poll it. If the website is well designed, the page you are trying to poll will have a "Last-Modified" header, you can make a "HEAD" request every so often (be nice: don't poll like crazy) and act when Last-Modified is >= than the one on record. If the site is not well designed, it will not have a reliable Last-Modified or ETAG header, in that case you will have to parse manually and check for changes yourself.
Cheers.