We Keep Coding

Python & subprocess - Open terminal session as user and execute one/two commands

As much as I hate regurgitating questions, it's a necessary evil to achieve a result to the next issue I'll present.
Using python3, tkinter and the subprocess package, my goal is to write a control panel to start and stop different terminal windows with a specific set of commands to run applications/sessions of the ROS application stack, including the core.
As such, the code would look like this per executable I wish to control:
class TestProc(object):
def __init__(self):
pass
def start(self):
self.process = subprocess.Popen(["gnome-terminal", "-c", "'cd /path/to/executable/script.sh; ./script.sh'"])
print("Process started.")
def stop(self):
self.process.terminate()
print("Process terminated.")
Currently, it is possible to start a terminal window and the assigned commands/processes, yet two issues persist:
gnome-terminal is set to launch a terminal window, then relieve control to the processes inside; as such, I have no further control once it has started. A possible solution for this is to use xterm yet that poses a slew of other issues. I am required to have variables from the user's .bashrc and/or export
Certain "global commands" eg. cd or roslaunch would be unavailable to the terminal sessions, perhaps due to the order of execution (eg. the commands are run before the bash profile is loaded) preventing any usable terminal at all
Thus, the question rings: How would I be able to start and stop a new terminal window that would run up to two commands/processes in the user environment?

There are a couple approaches you can take, the most flexible here is also the most complicated, so you'd want to consider whether you need to do it.
If you only need to show the output of the script, you can simply pipe the output to a file or to a named pipe. You can then capture that output by reading/tailing the file. This is simplest, as long as the script don't actually need to have any user interaction.
If you really only need to spawn a script that runs in the background, and you need to simulate user interaction but you don't actually need to accept actual user input, you can use expect approach (using the pexpect library).
If you need to actually allow the real user to interact with the program, then you have two approaches. First is that you can embed the VTE widget into your application, this is the most seamless integration as it'll make the terminal look seamless with your application, however it's also the most heavy.
Another approach is to start gnome-terminal as you've done here, this necessarily spawns a new window.
If you need to both script some interaction while also allowing some user input, you can do this by spawning your script in a tmux session. Using tmux send-keys command to automate the moon interactive part, and then spawn a terminal emulator for users to interact with tmux attach. If you need to go back and forth between automated part and interactive part, you can combine this approach with expect.

How do i pass on control on to different terminal tab using perl?

I am trying to automate a scenario in which, I have a terminal window open with multiple tabs open in it. I am able to migrate between the tabs, but my problem is how do i pass control to another terminal tab while i run my perl script in a different tab.
Example: I have a terminal open with Tab1,Tab2,Tab3,Tab4 open in the same terminal, i run the perl script in Tab3 and i would want to pass some commands onto Tab1. Could you please tell me how can i do this ??
I use GUI tool to switch between tabs X11::GUITest and use keyboard shortcuts to switch between tabs, any alternative suggestion is welcome, my ultimate aim is to pass control on to a different tab.

The main thing to understand is that each tab has a different instance of terminal running, more importantly a different instance of shell (just thought I would mention as it didnt seem like you were clear about that from your choice of words). So "passing control" in such a scenario could most probably entail inter-process communication (IPC).
Now that opens up a range of possibilities. You could, for example, have a python/perl script running in the target shell (tab) to listen on a unix socket for commands in the form of text, which the script can then execute. In Python, you have modules subprocess (call, Popen) and os (exec*) for this. If you have to transfer control back to the calling process, then I would suggest using subprocess as you would be able to send back return codes too.
Switching between tabs is a different action and has no consequences on the calling/called processes. And you have already mentioned how you intend on doing that.

How to unhide and show a process created with subprocess.popen()?

I am trying to create a simple command-line process and show it to the user (I do NOT want the process to be hidden):
import subprocess
import win32con
kwargs = {}
info = subprocess.STARTUPINFO()
info.dwFlags |= subprocess.STARTF_USESHOWWINDOW
info.wShowWindow = win32con.SW_SHOWMAXIMIZED
ExecuteString = ('process.cmd')
kwargs['startupinfo'] = info
sp = subprocess.Popen(ExecuteString, **kwargs)
It works with e.g. notepad.exe but not with the simple process.cmd:
echo "This is a process run from python"
pause
I run out of ideas, how to achieve this. I find all kind of stuff, how to HIDE a process. But I want to achieve the opposite.
Any idea?
Thanks!

You seem to be confusing the notions of process and window. All windows are associated to a process, but a certain process may not be associated with any window.
Your simple batch script is interpreted from the cmd.exe process. If you're used to the behaviour of windows when you open batch scripts with a double-click, might believe cmd.exe is always associated with a window, but that is not true. You can check this yourself by simply running cmd.exe inside a existing command prompt - it doesn't open a new window (as running notepad.exe, for example, would).
In python, processes run "as if" they were run from a command prompt - which is why you don't get another window.
This doesn't actually answer the question, but it might be useful in understanding the problem.

For Windowed applications, you simply need to use the SW_HIDE constant instead of SW_SHOWMAXIMIZED.
If you also want to cover console applications that start up a terminal window, I'm guessing that you would want to run something like this:
start the process;
use EnumWindows();
in your EnumWindowsProc implementation, check for a top-level window (using GetParent()) that is owned by the process you just launched (use GetWindowThreadProcessId()).
This will allow you to find the top-level window(s) of the process you just launched. From there, you can call ShowWindow() to show/hide the Window.
Note that this may be subject to some timing issues (if your search runs before the child process can create its window, the search will yield no results) as well as flicker (because you'll hide the window after it displays).

Why does stdout not flush when connecting to a process that is run with supervisord?

I am using Supervisor (process controller written in python) to start and control my web server and associated services. I find the need at times to enter into pdb (or really ipdb) to debug when the server is running. I am having trouble doing this through Supervisor.
Supervisor allows the processes to be started and controlled with a daemon called supervisord, and offers access through a client called supervisorctl. This client allows you to attach to one of the foreground processes that has been started using a 'fg' command. Like this:
supervisor> fg webserver
All logging data gets sent to the terminal. But I do not get any text from the pdb debugger. It does accept my input so stdin seems to be working.
As part of my investigation I was able to confirm that neither print nor raw_input send and text out either; but in the case of raw_input the stdin is indeed working.
I was also able to confirm that this works:
sys.stdout.write('message')
sys.flush()
I though that when I issued the fg command that it would be as if I had run the process in the foreground in the standard terminal ... but it appears that supervisorctl is doing something more. Regular printing does not flush for example. Any ideas?
How can I get pdb, standard prints, etc to work properly when connecting to the foreground terminal using the fg command in supervisorctl?
(Possible helpful ref: http://supervisord.org/subprocess.html#nondaemonizing-of-subprocesses)

It turns out that python defaults to buffering its output stream. In certain cases (such as this one) - it results in output being detained.
Idioms like this exist:
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
to force the buffer to zero.
But the better alternative I think is to start the base python process in an unbuffered state using the -u flag. Within the supervisord.conf file it simply becomes:
command=python -u script.py
ref: http://docs.python.org/2/using/cmdline.html#envvar-PYTHONUNBUFFERED
Also note that this dirties up your log file - especially if you are using something like ipdb with ANSI coloring. But since it is a dev environment it is not likely that this matters.
If this is an issue - another solution is to stop the process to be debugged in supervisorctl and then run the process temporarily in another terminal for debugging. This would keep the logfiles clean if that is needed.

It could be that your webserver redirects its own stdout (internally) to a log file (i.e. it ignores supervisord's stdout redirection), and that prevents supervisord from controlling where its stdout goes.
To check if this is the case, you can tail -f the log, and see if the output you expected to see in your terminal goes there.
If that's the case, see if you can find a way to configure your webserver not to do that, or, if all else fails, try working with two terminals... (one for input, one for ouptut)

Python interpreter with Linux Screen

I was working with Python with a Linux terminal screen. When I typed:
help(somefunction)
It printed the appropriate output, but then my screen was stuck, and at the bottom of the terminal was "(end)".
How do I get unstuck? Thanks in advance.

The standard on GNU (or other Unix-like) systems is to use the environment variable PAGER for the command that should receive output for viewing one screenful ("page") at a time.
Mine is set to:
$ echo $PAGER
less
Yours might be set to more, or a different command, or not set at all in which case a system-wide default command will be used.
It sounds like yours is modelled after the more program. The program is showing you page-by-page output, and in this case telling you you're at the end.
Most of them (basically, any pager more modern than more) allow you to go forward and backward in the output by using the cursor control keys (arrows and PgUp/PgDown), and many other operations besides.
Since you can do all these things wherever you are in the output, the program needs an explicit command from you to know that you're done navigating the output. In all likelihood that command is the keypress q.
For more information on how to drive your pager, e.g. less, read its manpage with the command man less (which, of course, will show pages of output using the pager program :-)

That program uses your pager, which is by default more. You can exit just by pressing q.