We Keep Coding

Running batch file with subprocess.call does not work and freezes IPython console

This is a frequent question, but reading the other threads did not solve the problem for me.
I provide the full paths to make sure I have not made any path formulation errors.
import subprocess
# create batch script
myBat = open(r'.\Test.bat','w+') # create file with writing access
myBat.write('''echo hello
pause''') # write commands to file
myBat.close()
Now I tried running it via three different ways, found them all here on SO. In each case, my IDE Spyder goes into busy mode and the console freezes. No terminal window pops up or anything, nothing happens.
subprocess.call([r'C:\\Users\\felix\\folders\\Batch_Script\\Test.bat'], shell=True)
subprocess.Popen([r'C:\\Users\\felix\\folders\\Batch_Script\Test.bat'], creationflags=subprocess.CREATE_NEW_CONSOLE)
p = subprocess.Popen("Test.bat", cwd=r"C:\\Users\\felix\\folders\\Batch_Script\\")
stdout, stderr = p.communicate()
Each were run with and without the shell=True setting, also with and without raw strings, single backslashes and so on. Can you spot why this wont work?

Spyder doesn't always handle standard streams correctly so it doesn't surprise me that you see no output when using subprocess.call because it normally runs in the same console. It also makes sense why it does work for you when executed in an external cmd prompt.
Here is what you should use if you want to keep using the spyder terminal, but call up a new window for your bat script
subprocess.call(["start", "test.bat"], shell=True)
start Starts a separate Command Prompt window to run a specified program or command. You need shell=True because it's a cmd built-in not a program itself. You can then just pass it your bat file as normal.

You should use with open()...
with open(r'.\Test.bat','w+') as myBat:
myBat.write('echo hello\npause') # write commands to file
I tested this line outside of ide (by running in cmd) and it will open a new cmd window
subprocess.Popen([r'Test.bat'], creationflags=subprocess.CREATE_NEW_CONSOLE)

Hey I have solution of your problem :)
don't use subprocess instead use os
Example :
import os
myBatchFile = f"{start /max} + yourFile.bat"
os.system(myBatchFile)
# "start /max" will run your batch file in new window in fullscreen mode
Thank me later if it helped :)

Pass a file to Python via the Right-Click 'Open With' context menu in windows, to then open in another program

I use a piece of software that when you close, saves your current configuration, however, next time I open the software by clicking on a file associated with it, it tries to run that file through the same configuration, which I don't want. I have therefore created a python script to first open up the app data for that program, reset the configuration back to default, and then open the program again.
This works fine if I just try to load the program without a file, but I would like to be able to click on a file associated with that program (usually I can just double click on the file to open it in the program), then use the 'open with' function in windows to select my script, and then open the file in the program once my script has cleared out the last configuration.
Currently, if I do this, it clears the configuration and opens the program but with no file loaded.
Essentially I am asking how I can pass the file to python, and then get python to pass that file to the third party application.
I am using the 'os.startfile('link to .exe') function to open the program, but how do I get the file path into python as an argument/string by clicking on the file and opening it with my script?
path = 'path/to/file/selected' # passed to python from selecting the file in windows explorer before starting script
# execute some code
os.startfile('path')
I'm a bit of a beginner when it comes to python so any help would be appreciated!
Using python 3.6 on windows 10

You can access command line arguments passed to your script using sys.argv. As long as you want to pass these arguments to some third-party application(which is an executable binary) you should use the subprocess module as os.startfile is meant to start a file with its associated application.
import sys
import subprocess
def main():
if len(sys.argv) == 1:
path = sys.argv[0]
subprocess.run(['/path/to/my.exe', path])
else:
print('Usage myscript.py <path>')
if __name__ == "__main__":
main()

If I understand your question correctly it could be done in the following fashion, relying on the subprocess module:
subprocess.Popen(["/path/to/application", "/path/to/file/to/open"])
See documentation for more details.

Python check_output call to eval (with arguments) fails

I'm using check_output to do all my SSH and GitHub setup, and I'm attempting to execute eval $(ssh-agent), both to start the agent and to parse the output if I need the process id.
from subprocess import check_output
out = check_output(["eval", "$(ssh-agent)"])
print(out)
But regardless of how I escape things, I get the same error.
Traceback (most recent call last):
File "gitSetup.py", line 3, in <module>
out = check_output(["eval", "$(ssh-agent)"])
File "/usr/lib/python2.7/subprocess.py", line 216, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1047, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
I'm wondering if I'm not escaping or trimming my arguments properly; if anyone sees my error, I'd appreciate some insight!

Even if you managed to fix the superficial syntax problems, a subprocess which runs successfully simply then terminates without a trace; it doesn't attempt to (and couldn't, even if it wanted to) modify the environment of the parent Python process. This is a common FAQ.
You could run the eval in the parent process which starts your Python script, or perhaps figure out how to communicate with ssh-agent directly from Python. Its output is usually a simple sequence of variable assignments, which you can parse yourself.
from subprocess import check_output
from os import environ
eval_string = check_output(['ssh-agent'])
for line in eval_string.rstrip('\n').split('\n'):
for expr in line.rstrip(';').split(';'):
if expr.startswith((' export ', 'echo ')):
continue
var, value = expr.strip().split('=', 1)
environ[var] = value
If the output from ssh-agent contains shell quoting, you will need to perform additional parsing on it (basically, trim the quotes around the value string). But this is already rather clunky and brittle, so perhaps revert to setting up the environment before launching Python instead of trying to splash some sophistication onto this kludge.
In more detail, ssh-agent and a precious few other shell utilities have a very specific design in order for them to be able to communicate with their parent process. Precisely because a subprocess cannot make any changes in the environment of its parent process, it instead prints a piece of code for its parent process to execute. By default, it prints sh code like this:
SSH_AUTH_SOCK=/tmp/ssh-MUyniqn10506/agent.10506; export SSH_AUTH_SOCK;
SSH_AGENT_PID=10507; export SSH_AGENT_PID;
echo Agent pid 10507;
There is also an option for it to print similar code in csh syntax (this is a historical shell which thankfully isn't used much any longer) but, alas, no option for producing Python code. (It would not be hard to make ssh-agent do that, per se.)
(The above output sample copy/pasted from http://blog.joncairns.com/2013/12/understanding-ssh-agent-and-ssh-add/ which contains a fuller explanation.)

Issues with subprocess.Popen() replacing os.system()

I am trying to generate a set of files through a python script on a 48-core cluster which actually has a master-and 3 slave machines. I need to generate a set of files, run some scripts on them, collect results and then delete them. I again repeat the process- regenerate files, execute, delete etc.,
When I delete and regenerate files with the same name, I see that the slave machine complains that it cannot find the files.
I am running python script through os.system()
I learnt from this post that it is better to use subprocess.Popen() rather than os.system so that it actually waits for my script to generate my files, before proceeding with the execution. I could use os.system("pause") or time.sleep(whatever) for waiting, but I want to convert my os.systems to subprocess.popens or subprocess.calls and I am stuck here.
I ran through the python documentation and tried out subprocess.Popen('ls'), but I am not able to get a simple thing like subprocess.Popen('cd /whatever_directory') working.
It might sound silly but how do I execute such a simple command like cd through subprocess rather than os.system('cd')?
Then, I actually want to convert the following into subprocess. How do I do it?
import os,optparse
from optparse import OptionParser
parser.add_option("-m", "--mod",dest='module', help='Enter the entity name')
parser.add_option("-f", "--folder", dest="path",help="Enter the path")
module=options.module
path=options.path
os.system('python %s/python_repeat_deckgen_remote.py -m %s' %(path,module))
I just replaced os.system with subprocess.Popen.
But it gave me a lot of complaints:
File "/usr/lib64/python2.6/subprocess.py", line 633, in __init__
errread, errwrite)
File "/usr/lib64/python2.6/subprocess.py", line 1139, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

As NPE already noted, new processes don't affect the existing one (which means os.system('cd /some/where') also has no effect on the current process). In this case, though, I think you're tripping over the fact that os.system invokes a shell to interpret the command you pass in, while subprocess.Popen does not do so by default. But you can tell it to do so:
proc = subprocess.Popen(
'python %s/python_repeat_deckgen_remote.py -m %s' % (path, module),
shell = True)
status = proc.wait()
If you're invoking a shell built-in command or using shell expansions, it's necessary to invoke the shell (assuming you're not willing to simulate it):
>>> import subprocess
>>> x = subprocess.Popen('echo $$')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/local/lib/python2.7/subprocess.py", line 1249, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
>>> x = subprocess.Popen('echo $$', shell = True); x.wait()
81628
0
>>>
but you can bypass the shell—which can help with security issues—if your problem permits, by breaking up the command arguments into a list:
>>> x = subprocess.Popen(['echo', '$$']); x.wait()
$$
0
>>>
Note that this time the output here is the string $$, rather than a process ID, since this time the shell did not interpret the string.
For your original example, for instance, you might use:
proc = subprocess.Popen(['python',
os.path.join(path, 'python_repeat_deckgen_remote.py'),
'-m',
module])
which avoids issues if path and/or module contain characters special to the shell.
(You might even want to call subprocess.Popen with cwd = path and eliminate the call to os.path.join, although that depends on other things.)

I am not able to get a simple thing like subprocess.Popen('cd /whatever_directory') working.
Each process's current working directory is independent of other processes in the system.
When you start a subprocess (using either of these mechanisms), and get it to cd into another directory, that has no effect on the parent process, i.e. on your Python script.
To change the current working directory of your script, you should use os.chdir().

I might be not answering you question directly, but for such tasks which require running subprocess, I always use plumbum.
IMO, It makes the task much simpler and more intuitive, including running on remote machines.
Using plumbum, in order to set subprocess's working directory, you can run the command in a with local.cwd(path): my_cmd() context.

Problem when invoking command prompt from a python script

I would like some help towards invoking a command prompt (& passing some argument to the command prompt) from a python script.
I use pyqt4 for developing the UI and on the UI I have a run button. On selection of run button, I would like to invoke a command prompt and pass on some script name as the argument.
self.connect(run_button, SIGNAL('clicked()'), self.runscript) # this is my run button signal and i'm calling the runscript()
def runscript(self):
print 'Inside Run Script'
os.chdir('C:\PerfLocal_PAL')
try:
subprocess.call(['C:\windows\system32\cmd.exe'])
except:
print 'Exception Caused.'
When I click on run button, the application dies and it does not invoke the command prompt at all. I tried with os.system as well same result.
also, I would like to know how to pass the argument to the call function?
Any help towards this is highly appreciated.
Thanks,

To correctly define file paths in Python on Windows, you need to do one of three things:
Use forward slashes: "C:/PerfLocal_PAL" (Python understands forward slashes regardless of platform)
Use raw strings: r"C:\PerfLocal_PAL"
Escape the backslashes: "C:\\PerfLocal_PAL"
This affects both your chdir call and your subprocess.call invocation.
However, you will also have trouble due to the fact that your parent process is a GUI application, and hence has no console streams for stdin, stdout and stderr. Try using the following instead to get a completely separate command window:
subprocess.call("start", shell=True)
You may also want to use the "/D" argument of start to set your working directory, rather than changing the cwd of the parent process:
subprocess.call(["start", "/DC:\\PerfLocal_PAL"], shell=True)

Have you tried debugging this at all? Which line does the script fail on? Does it actually start the runscript function at all?
Regarding passing arguments to cmd.exe, have a look at the documentation for subprocess.call. It will show you that you can have a second argument providing the command line parameters to the program, e.g.
subprocess.call(["C:\windows\system32\cmd.exe", "scriptname.bat"])

One problem is that subprocess.call will block until it is complete, and cmd.exe will not return until you exit it. That answers the 'just dies' but may not explain the console never appearing. Start with this:
subprocess.Popen(['C:\Windows\system32\cmd.exe'])
That at least will not block. If you can get it to appear, try your arguments, like this:
subprocess.Popen(['C:\Windows\system32\cmd.exe', 'program_or_script', 'arg1'])

Your signal connection and your subprocess call seems to be fine.
Change your chdir call to:
os.chdir(r'C:\PerfLocal_PAL')
I guess the error you are getting is of the form (when you launch your application from the command prompt):
WindowsError: [Error 123] The filename, directory name, or volume label syntax is incorrect: 'C:\PerfLocal_PAL'