Question: is there any options to hold the session using Subprocess or else module?
It runs only one instruction or command per time.
I can't find any direct solution. Why it works only so?
It's same question for Windows, Linux and Mac OS-s.
Example 1.
I need to do some work in cmd with admin rights.
from subprocess import run, Popen, call
run('net user Administrator /active: yes', shell=True)
run('pip install [some module]', shell=True) # or "powershell -command [some command]"
Example 2.
I need to use virtual env module and get in needed environment.
run(["workon", "Universal"]
run("[some changes]")
There is a CMD module, but it looks isolated from system cmd-terminal,like your own created CLI, and having a different purpose.
Please don't answer about bash scripts, or other options to get administrator rights, or about launching scripts from cmd itself like ">python main.py". There is talking about the sequential habitual execution of commands from CMD and how can Python do this, hold the cmd session, or it can't.
I suppose you want keep administrative privileges once gained.
I don't know about windows, don't use it.
But if i try something similar on Linux, i only have to type the admin password once:
$ python3
Python 3.8.2 (default, Jul 16 2020, 14:00:26)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> subprocess.run("sudo ls", shell=True)
[sudo] password for <me>:
a.txt b.txt
CompletedProcess(args='sudo ls', returncode=0)
>>> subprocess.run("sudo ls", shell=True)
a.txt b.txt
CompletedProcess(args='sudo ls', returncode=0)
>>>
But your actual question is if you can 'hold the session'.
That is, if I understand you correctly, execute some commands in a shell, keep the shell running after those commands have finished, while you do something else in your script to generate extra commands, and then send these new commands to the same old shell
Yes that is possible.
You can use subprocess.Popen() to spawn an interactive shell (i.e. the shell is the command to execute)
and write actual commands to it's stdin and read results from stdout
If that sounds complicated: yes it is!
e.g. proper error handling means parsing the text returned by the shell, compared to simply check the return code for a synchronous call. Even detecting if your command has finished requires monitoring what appears on the shell stdout/stderr
You don't want to go down the asynchronous road.
Synchronous calls to external commands (such as subprocess.run()), without any shell in between, is so much simpler.
I tried one more time, and it seems it works! Holds the session!!
I used:
Popen("workon Universal", shell=True) # 1
Popen("pip install termcolor") # 2 without shell=True
And that finally installed a module on "Universal" virtual environment. And i can endless continue the Popen(3), Popen(4).
Same works with run().
It's all about right using the "shell=True" attribute.
I guess that is the solution i looked for. And it's universal for all commands and sessions. That's hard to understand, no any example on google or youtube.
Thanks!
Related
I'm on Windows using PowerShell and WSL 'Ubuntu 20.04 LTS'. I have no native Linux Distro, and I cant use virtualisation because of nested device reasons.
My purpose is to use a Windows Python script in PowerShell to call WSL to decrypt some avd-snapshots into raw-images. I already tried os.popen, subprocess.Popen/run/call, win32com.client, multiprocessing, etc.
I can boot the WSL shell, but no further commands are getting passed to it. Does somebody know how to get the shell into focus and prepared for more instructions?
Code Example:
from multiprocessing import Process
import win32com.client
import time, os, subprocess
def wsl_shell():
shell = win32com.client.Dispatch("wscript.shell")
shell.SendKeys("Start-Process -FilePath C:\\Programme\\WindowsApps\\CanonicalGroupLimited.Ubuntu20.04onWindows_2004.2021.825.0_x64__79rhkp1fndgsc\\ubuntu2004.exe {ENTER}")
time.sleep(5)
os.popen("ls -l")
if __name__ == '__main__':
ps = Process(target = wsl_shell)
ps.start()
There are a few ways of running WSL scripts/commands from Windows Python, but a SendKeys-based approach is usually the last resort, IMHO, since it's:
Often non-deterministic
Lacks any control logic
Also, avoid the ubuntu2004.exe (or, for other users who find this, the deprecated bash.exe command). The much more capable wsl.exe command is what you are looking for. It has a lot of options for running commands that the <distroname>.exe versions lack.
With that in mind, here are a few simplified examples:
Using os.system
import os
os.system('wsl ~ -e sh -c "ls -l > filelist.txt"')
After running this code in Windows Python, go into your Ubuntu WSL instance and you should find filelist.txt in your home directory.
This works because:
os.system can be used to launch the wsl command
The ~ tells WSL to start in the user's home directory (more deterministic, while being able to avoid specifying each path in this case)
wsl -e sh runs the POSIX shell in WSL (you could also use bash for this)
Passing -c "<command(s)>" to the shell runs those commands in the WSL shell
Given that, you can pretty much run any Linux command(s) from Windows Python. For multiple commands:
Either separate them with a semicolon. E.g.:
os.system('wsl ~ -e sh -c "ls -l > filelist.txt; gzip filelist.txt')
Or better, just put them all in a script in WSL (with a shebang line), set it executable, and run the script via:
wsl -e /path/to/script.sh
That could even by a Linux Python script (assuming the correct shebang line in the script):
wsl -e /path/to/script.py
So if needed, you can even call Linux Python from Windows Python this way.
Using subprocess.run
The os.system syntax is great for "fire and forget" scripts where you don't need to process the results in Python, but often you'll want to capture the output of the WSL/Linux commands for processing in Python.
For that, use subprocess.run:
import subprocess
cp = subprocess.run(["wsl", "~", "-e", "ls", "-l"], capture_output=True)
print(cp.stdout)
As before, the -e argument can be any type of Linux script you want.
Note that subprocess.run also gives you the exit status of the command.
I'd like to give my Python scripts the ability to detect whether it was executed in a Git Bash terminal, or the Windows cmd command line interface. For example, I'm trying to write a function to clear the terminal (regardless of which terminal it is), e.g. echoes the clear command if in Git Bash, or cls if in cmd.
I've tried using sys.platform to detect this, but it returns win32 regardless of which type of terminal it was ran in.
Please try using os and psutil modules.
For example,
import os, psutil # Get the parent process name.
pprocName = psutil.Process(os.getppid()).name()
Then you can have your logic depending on the shell.
Additionally, you may want to check https://www.geeksforgeeks.org/clear-screen-python/
I don't believe what you're asking for is possible, but there are several answers here that show all the detections you can do to use the correct type of clear. Usually, it's just best to either make your own window or not clear the screen, sadly.
I was fiddling with the subprocess.Popen and got to a weird situation I just can't explain. Leaving here in the hope someone can make some sense of this.
Description
Spawning a cmd.exe subprocess from the python process, results in the prompt switching from python to cmd. And when the python prompt is shown the command is executed in cmd.exe. On the other hand when the cmd.exe prompt is shown, issuing a command executes in the python shell.
Getting to this weird back and forth prompt. An example as follows:
Microsoft Windows [Version 10.0.17134.829]
(c) 2018 Microsoft Corporation. All rights reserved.
C:\Users\Pedro Rodrigues>help
Type help() for interactive help, or help(object) for help about object.
>>>
Apparently a Windows Command Prompt. But, not only the help command is python's, afterwards we appear to be dropped in the python shell.
But, if we do it again. Its actually the other way around. This is now the Windows Command Prompt, but we are presented with the python prompt!
>>> help
For more information on a specific command, type HELP command-name
ASSOC Displays or modifies file extension associations.
... supressed ...
WMIC Displays WMI information inside interactive command shell.
For more information on tools see the command-line reference in the online help.
C:\Users\Pedro Rodrigues>
And this keeps on going, the terminal is essentially useless at this point, as far as I could tell.
Steps to Reproduce
Open a python shell, either:
Open Command Prompt, and type python.
Open a python shell.
Run the following code:
import subprocess
subprocess.Popen('cmd.exe')
Type help, press ENTER.
Repeat previous step.
What is going on?
I'm surprised by the result. With that said, I just cannot explain what is happening here.
On Linux
The behaviour is a bit different, although it also renders the terminal useless. I haven't played a lot with it on Linux, but did various times end up with a working terminal actually.
import subprocess
subprocess.Popen('/bin/bash')
But most of the times, got a working bash shell. Which stopped the python process every time.
I understand this is an esoteric scenario, yet what happens on Windows puzzles me.
I have a data processing pipeline setup that I want to debug.
The pipeline consists of a bash script that calls a python script.
I usually use iPython's embed() function for debugging. However, when calling the python script from the bash file, the embed() function is called but immediately exited, without me being able to interfere. When running the same python program directly from the command line I don't observe this kind of behavior. Is this intended behavior or am I doing something wrong?
Python 2.7.6 (default, Oct 26 2016, 20:30:19)
Type "copyright", "credits" or "license" for more information.
IPython 2.4.1 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]:
Do you really want to exit ([y]/n)?
'follow up code prints here'
I can replicate the problem like this:
# test.py
import IPython
import sys
print(sys.stdin.read())
IPython.embed()
# session
❯ echo 'foo' | python test.py
foo
Python 3.6.8 (default, Oct 7 2019, 12:59:55)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.10.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: Do you really want to exit ([y]/n)?
❯ # I didn't quit on purpose, it happened automatically
STDIN is not a TTY, so I'm thinking that IPython is worried that the inbound text (via the pipe) won't be a user typing. It doesn't want foo (from my example above) to spew into the IPython shell and do something unexpected.
You can work around this by getting your terminal id via the tty command, and redirecting stdin to the calling terminal after it has finished reading from the pipe, something like this:
with open('/dev/pts/16') as user_tty:
sys.stdin=user_tty
IPython.embed()
For more on ttys, see this post. Note also that if you put the wrong tty in there, input from some other terminal will control IPython.
I'm not sure if it's possible for IPython to know what the calling tty would have been, had it not been overwritten by bash to be the output-side of the pipe.
Edit: Here's my workaround put more simply: How do I debug a script that uses stdin with ipython?
I ran some experiments to see the behaviour. I noticed that IPython shows the console if any of the ancestor process is terminal.
Following are the files in /tmp directory:
x.py
import IPython
IPython.embed()
call.sh
/usr/bin/python /tmp/x.py
call2.sh
/tmp/call.sh
Experiment 1
Running python x.py does open the IPython shell and waits.
Experiment 2
Running bash call.sh also opens the IPython shell and waits.
Experiment 3
Running bash call2.sh also opens the IPython shell and waits.
As you can see, it does not matter how deep is your IPython.embed call is. It always starts the interactive console and waits.
Lets try if it also works when we fork a new process.
fork.sh
/usr/bin/python /tmp/x.py &
Experiment 4
In this case, IPython shell started but immediately exited. Notice the & at the end. It starts a different process. IPython was not able to access the terminal in this case and hence exited gracefully.
I've been using Cygwin on Windows recently. I want to use the Windows installation of Python, so during testing I'm using /cygdrive/c/Python26/python.exe myfile.py rather than python myfile.exe.
This is working almost perfectly, except for printing. When I run the Windows Python from Cygwin the output doesn't print until execution finishes. It works fine running in Windows Python from explorer.exe or cmd.exe, and it works in Cygwin using the Cygwin-installed Python (/bin/python.exe).
Is there a workaround for this? The important thing is to be able to run the Windows version, but I'd like to do it all from with Bash.
The real problem is that when you run a command in any of the Cygwin terminal programs like mintty, they don't act as Windows Consoles. Only Windows Console-based ones like CMD or Console2 do that. So, with Cygwin terminals the Windows python.exe doesn't think it is talking to an interactive console.
That leads to buffering output instead of flushing buffers on every line as is done in interactive sessions. That is why Amro's adding the flush() on every line fixes the symptom, but means changing the code.
One solution without changing the code is to turn off buffering in Python using the '-u' flag on the command line or setting the PYTHONUNBUFFERED environment variable.
export PYTHONUNBUFFERED=1
/cydrive/c/Python27/python.exe foo.py
or
/cydrive/c/Python27/python.exe -u foo.py
or run in interactive mode
/cydrive/c/Python27/python.exe -i foo.py
You will also not be able to run the Windows python.exe interactive mode in the Cygwin terminal. It will not bring up an interactive session, but will just hang. I find the best solution seems to be to use 'cygstart' (better than using the '-i' option):
cygstart /cygdrive/c/Python27/python.exe
And that seems to work with ipython as well (if installed):
cygstart /cygdrive/c/Python27/Scripts/ipython.exe
Not answering the initial question, but for those who want to use Python interactive session from within Cygwin terminal (for example in mintty) - start Python with "-i" option to tell it explicitly that it needs to run in interactive mode:
$ python -i
The neat way is also to create an alias in your .bashrc (knowing that it is only read for interactive terminal sessions anyway):
alias python='python -i'
Otherwise, Python will not know that it runs in the console, because all Cygwin pty-based terminals (mintty, rxvt and xterm) are recognized as pipes by Windows, not as the console. Therefore, Python thinks there is no console and enters non-interactive mode. So, if you still want interactive mode instead, you need to explicitly tell Python to use it. However, it still won't behave as it normally should - one still won't be able to use HOME or LEFT ARROW keys, and so on.
Perhaps if you flush the output
import sys
V = range(100000)
for x in V:
print x
sys.stdout.flush()