We Keep Coding

Running batch file with subprocess.call does not work and freezes IPython console

This is a frequent question, but reading the other threads did not solve the problem for me.
I provide the full paths to make sure I have not made any path formulation errors.
import subprocess
# create batch script
myBat = open(r'.\Test.bat','w+') # create file with writing access
myBat.write('''echo hello
pause''') # write commands to file
myBat.close()
Now I tried running it via three different ways, found them all here on SO. In each case, my IDE Spyder goes into busy mode and the console freezes. No terminal window pops up or anything, nothing happens.
subprocess.call([r'C:\\Users\\felix\\folders\\Batch_Script\\Test.bat'], shell=True)
subprocess.Popen([r'C:\\Users\\felix\\folders\\Batch_Script\Test.bat'], creationflags=subprocess.CREATE_NEW_CONSOLE)
p = subprocess.Popen("Test.bat", cwd=r"C:\\Users\\felix\\folders\\Batch_Script\\")
stdout, stderr = p.communicate()
Each were run with and without the shell=True setting, also with and without raw strings, single backslashes and so on. Can you spot why this wont work?

Spyder doesn't always handle standard streams correctly so it doesn't surprise me that you see no output when using subprocess.call because it normally runs in the same console. It also makes sense why it does work for you when executed in an external cmd prompt.
Here is what you should use if you want to keep using the spyder terminal, but call up a new window for your bat script
subprocess.call(["start", "test.bat"], shell=True)
start Starts a separate Command Prompt window to run a specified program or command. You need shell=True because it's a cmd built-in not a program itself. You can then just pass it your bat file as normal.

You should use with open()...
with open(r'.\Test.bat','w+') as myBat:
myBat.write('echo hello\npause') # write commands to file
I tested this line outside of ide (by running in cmd) and it will open a new cmd window
subprocess.Popen([r'Test.bat'], creationflags=subprocess.CREATE_NEW_CONSOLE)

Hey I have solution of your problem :)
don't use subprocess instead use os
Example :
import os
myBatchFile = f"{start /max} + yourFile.bat"
os.system(myBatchFile)
# "start /max" will run your batch file in new window in fullscreen mode
Thank me later if it helped :)

Why Does Python Call a Subprocess Command Incorrectly?

This is a follow up on a previous question as I have made progress(which is irrelevant at this point). It is worth noting that I am learning python and barely know what I am doing, however, I am familiar with programming. I am trying to call an SCP command in the windows terminal through python. However, it is not doing the desired effect. The script runs smoothly with no errors and it prints the debug commands as I have written them. However, the SCP call does not actually go through on the other end. To make sure I have the right command, I have it set to print the same command that it called afterwards. When I copy this printed command and paste it into the windows command terminal, it gives the desired effect. Why is the same command not working correctly in python? Thanks. This is my script:
import subprocess
subprocess.run(['scp', 'c:/users/<name>/desktop/OOGA.txt', 'pi#<IP>:here/'], shell=True)
print ("done")
print ('scp', 'c:/users/<name>/desktop/OOGA.txt', 'pi#<IP>:here/')

Try using raw string if shell is set to True:
from subprocess import run as subrun
status = subrun(r'scp c:/users/<name>/desktop/OOGA.txt pi#<IP>:here/',shell=True)
print("Done")
print(status)

What's the best way to execute PowerShell scripts from Python

All the previous posts on this topic deal with specific challenges for their use case. I thought it would be useful to have a post only dealing with the cleanest way to run PowerShell scripts from Python and ask if anyone has an better solution than what I found.
What seems to be the generally accepted solution to get around PowerShell trying to interpret different control characters in your command differently to what's intended is to feed your Powershell command in using a file:
ps = 'powershell.exe -noprofile'
pscommand = 'Invoke-Command -ComputerName serverx -ScriptBlock {cmd.exe \
/c "dir /b C:\}'
psfile = open(pscmdfile.ps1, 'w')
psfile.write(pscommand)
psfile.close()
full_command_string = ps + ' pscmdfile.ps1'
process = subprocess.Popen(full_command_string , shell=True, \
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
When your python code needs to change the parameters for the Powershell command each time you invoke it you end up writing and deleting a lot of temporary files for subprocess.Popen to run. It works perfectly but it's unnecessary and not very clean. It's really nice to be able to tidy up and wanted to get suggestions on any improvements I could make to the solution I found.
Instead of writing a file to disk containing the PS command create a virtual file using the io module. Assuming that the "date" and "server" strings are being fed in as part of a loop or function that contains this code, not including the imports of course:
import subprocess
import io
from string import Template
raw_shellcmd = 'powershell.exe -noprofile '
--start of loop with server and date variables populated--
raw_pslistcmd = r'Invoke-Command -ComputerName $server -ScriptBlock ' \
r'{cmd.exe /c "dir /b C:\folder\$date"}'
pslistcmd_template = Template(raw_pslistcmd)
pslistcmd = pslistcmd_template.substitute(server=server, date=date)
virtualfilepslistcommand = io.BytesIO(pslistcmd)
shellcmd = raw_shellcmd + virtualfilepslistcommand.read()
process = subprocess.Popen(shellcmd, shell=True, stdout=subprocess.PIPE, \
stderr=subprocess.PIPE)
--end of loop--

Arguably the best approach is to use powershell.exe -Command rather than writing the PowerShell command to a file:
pscommand = 'Invoke-Command ...'
process = subprocess.Popen(['powershell.exe', '-NoProfile', '-Command', '"&{' + pscommand + '}"'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
Make sure double quotes in the pscommand string are properly escaped.
Note that shell=True is required only in certain edge cases, and should not be used in your scenario. From the documentation:
On Windows with shell=True, the COMSPEC environment variable specifies the default shell. The only time you need to specify shell=True on Windows is when the command you wish to execute is built into the shell (e.g. dir or copy). You do not need shell=True to run a batch file or console-based executable.

After spending a fair amount of time on this.
I think that running powershell commands from python may not make sense to a lot of people, especially people who work exclusively in windows environments. There are numerous clear advantages to python over powershell however so the ability to do all your business logic in python and then selectively execute powershell on remote servers is truly a great thing.
I've now been through several improvements of my "winrmcntl" module which I can't share due to company policy unfortunately but here is my advice to anyone who would like to do something similar. The module should take as input an unmodified PS command or scriptblock as you'd run it if you were typing directly in PS on the destination box. A few tricks:
To avoid permission difficulties, ensure the user running your python script and hence the one running powershell.exe via process.Popen is the user that has the correct permissions on the windows box you're invoke-command is pointing at. We use an enterprise scheduler which has windows vms as agents on which the python code lives which takes care of that.
You will sometimes rarely but still get the odd esoteric exception from powershell land, if they're anything like the one in particular I saw the odd time, microsoft scratch their heads at a little and get you to do time consuming application stack tracing. This is not only time consuming but very difficult to get right because it's resource intensive and you don't know when the exception will next occur. In my opinion, it's much better and easier to parse the output of the exception and retry up to x number of times if a certain text appears in those exceptions. I keep a list of strings in my winrmcntl module which currently contains a single string.
If you want to not have to "massage" the powershell commands as they traverse the python -> windows -> powershell -> powershell stack to make them work as expected on destination boxes, the most consistent method I've found is to write your one liners and scriptblocks alike into a ps_buffer.ps1 file which you then feed to powershell on the source box so that every process.popen looks exactly the same but the content of ps_buffer.ps1 changes with each execution.
powershell.exe ps_buffer.ps1
To keep your python code nice and clean, it's great having your list of powershell one liners in a json file or similar as well as pointers to scriptblocks you want to run saved into static files. You load up your json file as an ordered dict and cycle through issuing commands based on what you're doing.
Can't be overstated, as far as is possible try to be on the latest stable version of PS but more than that, it's imperative to be on the same version on client and server.
"scriptblock" and "server" are the values fed to this module or function
import subprocess
from string import Template
scriptblock = 'Get-ChildItem' #or a PS scriptblock as elaborate as you need
server = 'serverx'
psbufferfile = os.path.join(tempdir, 'pscmdbufferfile_{}.ps1'.format(server))
fullshellcmd = 'powershell.exe {}'.format(psbufferfile)
raw_pscommad = 'Invoke-Command -ComputerName $server -ScriptBlock {$scriptblock}'
pscmd_template = Template(raw_pscommand)
pscmd = pscmd_template.substitute(server=server, scriptblock=scriptblock)
try:
with open(psbufferfile, 'w') as psbf:
psbf.writelines(pscmd)
....
try:
process = subprocess.Popen(fullshellcmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output, error = process.communicate()
....

Fetch PowerShell script from GitHub and execute it

Running into issues executing a PowerShell script from within Python.
The Python itself is simple, but it seems to be passing in \n when invoked and errors out.
['powershell.exe -ExecutionPolicy Bypass -File', '$Username = "test";\n$Password = "password";\n$URL
This is the code in full:
import os
import subprocess
import urllib2
fetch = urllib2.urlopen('https://raw.githubusercontent.com/test')
script = fetch.read()
command = ['powershell.exe -ExecutionPolicy Bypass -File', script]
print command #<--- this is where I see the \n.
#\n does not appear when I simply 'print script'
So I have two questions:
How do I correctly store the script as a variable without writing to disk while avoiding \n?
What is the correct way to invoke PowerShell from within Python so that it would run the script stored in $script?

How do I correctly store the script as a variable without writing to disk while avoiding \n?
This question is essentially a duplicate of this one. With your example it would be okay to simply remove the newlines. A safer option would be to replace them with semicolons.
script = fetch.read().replace('\n', ';')
What is the correct way to invoke PowerShell from within Python so that it would run the script stored in $script?
Your command must be passed as an array. Also you cannot run a sequence of PowerShell statements via the -File parameter. Use -Command instead:
rc = subprocess.call(['powershell.exe', '-ExecutionPolicy', 'Bypass', '-Command', script])

I believe this is happening because you are opening up PowerShell and it is automatically formatting it a specific way.
You could possibly do a for loop that goes through the command output and print without a /n.

python script to edit a file in vim

I want to make a python script that:
opens a file, executes the command i,
then writes 2 lines of code, hits escape
executes the command ZZ.
I was thinking along the lines of os.system("vi program") then os.system("i") and os.system("code"), but that didn't work because you can only execute commands. Thank you!

It's not clear why you want to do this. To truly run an interactive program, you'll have to create a pseudo-tty and manage it from your python script - not for the faint of heart.
If you just want to insert text into an existing file, you can do that directly from python, using the file commands. Or you could invoke a program like sed, the "stream editor", that is intended to do file editing in a scripted fashion. The sed command supports a lot of the ex command set (which is the same base command set that vi uses) so i, c, s, g, a, all work.

THE CODE
import pyautogui
from multiprocessing import Process
import os
vi_proc = Process(target = lambda: os.system("vi program"))
vi_proc.start()
pyautogui.typewrite("i")
pyautogui.typewrite("This code\n")
pyautogui.typewrite("really sucks!")
pyautogui.press("esc")
pyautogui.typewrite("ZZ")
vi_proc.join()
THE BLABLABLA
Well, I really not understand WHY, but I coded a working solution. I used PyAutoGUI, a really simple library that allow you to emulate key and mouse presses and movements.
You may also need to install some sysyem package, like libjpeg8-dev. Furthermore, probably you have also to issue the command xhost + temporarily before installation.
That said, in bash it will be simply:
echo -e "This code\nreally sucks!" > program

If you really want to run VIM from the command-line, you can use the VIM -c option. Something like this:
gvim -c "normal oFirst line" -c "normal oSecond line" -c "ZZ" foo.txt
(Adjust using o, O, i or I as according to where you want the line inserted).
There must be an easier way to insert two lines in a file, though.