I am trying to call several "blocking" .bat files from my python program. First thing I need to do is change the directory that the CMD is opened. After I have the CMD pointing to the desired location I will call two bat files. I want them to execute sequentially.
def launchAdminConsole():
print('Going to launch admin console')
changeDir = 'cd dir1\\dir2\\bin \n 1.bat \n 2.bat'
os.system("start /wait cmd /c {"+changeDir+"}")
print("Admin Console launced")
According to this question using the /wait should make the command prompt wait but for me, it just pops up and goes away so I am not sure if the bat file is executed or not.
Also I am not sure if I have formed the command line code correctly. I googled about how to execute several commands in a single instance of cmd from python but none of the results helped me much so I took a guess on my own and did the above code.
I need the command prompt to open and run the two bat files and then give the control back to python. I do not need to fetch the output of the bat files or something I just need to know if the two files are executed or not. As I said before they are all blocking bat files so if run correctly the command prompt will not be able to close so quickly. I hope you guys got my requirement else comment below I will explain more.
Edit :
Updated my code as follows
def launchAdminConsole2():
print('Going to launch admin console')
changeDir = 'cd dir1\\dir2\\bin'
runOnce1 = '1.bat'
runOnce2 = '2.bat'
p = subprocess.Popen(changeDir,shell=True)
p.wait()
print(p.returncode)
p = subprocess.call([changeDir, runOnce, runOnce1])
p.wait()
print("Admin Console launced")
The return code for the change directory returns 0 but it is saying that 1.bat is not found. I am sure that if the directory has changed that file will be present in the given location.
The error is
File "C:\Users\nirma\AppData\Local\Programs\Python\Python37-32\lib\subprocess.py", line 304, in call
with Popen(*popenargs, **kwargs) as p:
File "C:\Users\nirma\AppData\Local\Programs\Python\Python37-32\lib\subprocess.py", line 756, in __init__
restore_signals, start_new_session)
File "C:\Users\nirma\AppData\Local\Programs\Python\Python37-32\lib\subprocess.py", line 1155, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
You can run several commands using cmd.exe /c if you separate them with ampersands:
cmd.exe /c "cd \dir & 1.bat & 2.bat"
Try passing that to subprocess.call and see how it goes.
Related
I'm using check_output to do all my SSH and GitHub setup, and I'm attempting to execute eval $(ssh-agent), both to start the agent and to parse the output if I need the process id.
from subprocess import check_output
out = check_output(["eval", "$(ssh-agent)"])
print(out)
But regardless of how I escape things, I get the same error.
Traceback (most recent call last):
File "gitSetup.py", line 3, in <module>
out = check_output(["eval", "$(ssh-agent)"])
File "/usr/lib/python2.7/subprocess.py", line 216, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1047, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
I'm wondering if I'm not escaping or trimming my arguments properly; if anyone sees my error, I'd appreciate some insight!
Even if you managed to fix the superficial syntax problems, a subprocess which runs successfully simply then terminates without a trace; it doesn't attempt to (and couldn't, even if it wanted to) modify the environment of the parent Python process. This is a common FAQ.
You could run the eval in the parent process which starts your Python script, or perhaps figure out how to communicate with ssh-agent directly from Python. Its output is usually a simple sequence of variable assignments, which you can parse yourself.
from subprocess import check_output
from os import environ
eval_string = check_output(['ssh-agent'])
for line in eval_string.rstrip('\n').split('\n'):
for expr in line.rstrip(';').split(';'):
if expr.startswith((' export ', 'echo ')):
continue
var, value = expr.strip().split('=', 1)
environ[var] = value
If the output from ssh-agent contains shell quoting, you will need to perform additional parsing on it (basically, trim the quotes around the value string). But this is already rather clunky and brittle, so perhaps revert to setting up the environment before launching Python instead of trying to splash some sophistication onto this kludge.
In more detail, ssh-agent and a precious few other shell utilities have a very specific design in order for them to be able to communicate with their parent process. Precisely because a subprocess cannot make any changes in the environment of its parent process, it instead prints a piece of code for its parent process to execute. By default, it prints sh code like this:
SSH_AUTH_SOCK=/tmp/ssh-MUyniqn10506/agent.10506; export SSH_AUTH_SOCK;
SSH_AGENT_PID=10507; export SSH_AGENT_PID;
echo Agent pid 10507;
There is also an option for it to print similar code in csh syntax (this is a historical shell which thankfully isn't used much any longer) but, alas, no option for producing Python code. (It would not be hard to make ssh-agent do that, per se.)
(The above output sample copy/pasted from http://blog.joncairns.com/2013/12/understanding-ssh-agent-and-ssh-add/ which contains a fuller explanation.)
When I run emconfigure ./configure I get the following error:
ERROR:root:Exception thrown when invoking Popen in configure with
args: "./configure"!
Traceback (most recent call last):
File "/usr/local/bin/emconfigure", line 13, in <module>
emconfigure.run()
File "/usr/local/Cellar/emscripten/1.37.21/libexec/emconfigure.py", line 46, in run
shared.Building.configure(sys.argv[1:])
File "/usr/local/Cellar/emscripten/1.37.21/libexec/tools/shared.py", line 1533, in configure
process = Popen(args, stdout=None if EM_BUILD_VERBOSE_LEVEL >= 2 else stdout, stderr=None if EM_BUILD_VERBOSE_LEVEL >= 1 else stderr, env=env)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1335, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Break-down:
This one can be tricky to track down because of the depth of the call stack. Let's break it apart:
At the "next-to-top level": a file you're trying to execute cannot be found.
One level lower than that things become clear: subprocess.Popen is trying to execute args.
What's in args? - basically ./configure (as passed by emscripten). export the EM_BUILD_VERBOSE=3 environmental variable and then stderr should display this.
So, the next-to-top level ./configure is directing things in an invalid fashion. Now it's time for a guess at the top level (not knowing what's in the ./configure script): is some line of the shell command in the ./configure script file itself trying to execute something non-existent? E.g.: some point after #! /bin/sh ? - try debugging just ./compile.
It is worth noting that a subprocess can have issues with relative paths (not getting PATH), and permissions (./configure is locked-down), but your error very well may persist even after /you/make/an/absolute/call/to/configure and 777 the permissions of configure. - In that case the previous paragraph is the likeliest culprit.
Editorializing:
The reason this error is interesting as a general one for emscripten is that part (much) of emscripten's raison d'etre is to port [legacy] C/++ code to new JS apps. Well, old C/++ builds get old in the sense of this question. If that's true, this is will be a generally-occurring problem, and, if it is general the community could use a general pattern for resolution.
If you're using Docker, your host computer is Windows, and you mounted it to a Windows directory, make sure that your files aren't \r\n line endings but instead \n line endings. Changing this fixed it for me.
In this case, if you're using git, you might try setting git config core.autocrlf input to stop checking out to \r\n. If you need to convert a file, try this answer out.
I am trying to get a command to run within python that otherwise works fine as a bat file or in the cmd command prompt.
Python version 2.7.9.
The following works fine in the command prompt:
"C:\Program Files (x86)\Common Files\Aquatic Informatics\AQUARIUS\runreport.exe" -Report="NHC_chart_1Week_1Series" -TimeSeries="Precip Total.24hr#FK Intake Forecast" -Server="aqserver-van" -username="admin" -password="admin" -label="Battery Voltage Data" -OutputFile="C:\Aquarius\Reports\Altagas\Summary\24hr_Precip_Forecast.pdf"
This basically calls a program (runreport.exe) and passes a bunch of arguments to it. If things go well a file is created.
It seems I should be able to do the same in python with subprocess.call
I have tried many different versions of the code but none of them run the program correctly. Below is my current code:
import subprocess
run_report_program_path=r'C:\Program Files (x86)\Common Files\Aquatic Informatics\AQUARIUS\runreport.exe'
reportname="-Report=NHC_chart_1Week_1Series"
timeseries="-TimeSeries=Precip Total.24hr#FK Intake Forecast"
server="-Server=aqserver-van"
user="-username=admin"
passwrd="-password=admin"
output="-OutputFile=C:\Aquarius\Reports\Altagas\Summary\24hr_Precip_Forecast.pdf"
result=subprocess.check_output([run_report_program_path, reportname, timeseries, server, user, passwrd, output], shell= True)
The program runreport's does 'pop up' but runs to quick and doesn't output the desired result, instead I get the error below. Does anyone see the issue?
Traceback (most recent call last):
File "C:\aquarius\scripts\API_import_and_trigger_reports_py\python_scripts\call_run_reports\test2.py", line 10, in <module>
result=subprocess.check_output([run_report_program_path, reportname, timeseries, server, user, passwrd, output], shell= True)
File "C:\Python27\lib\subprocess.py", line 573, in check_output
raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command '['C:\\Program Files (x86)\\Common Files\\Aquatic Informatics\\AQUARIUS\\runreport.exe', '-Report=NHC_chart_1Week_1Series', '-TimeSeries=Precip Total.24hr#FK Intake Forecast', '-Server=aqserver-van', '-username=admin', '-password=admin', '-OutputFile=C:\\Aquarius\\Reports\\Altagas\\Summary\x14hr_Precip_Forecast.pdf']' returned non-zero exit status 1
Also, in the command prompt it usually returns a nice set of errors (like time series not found, or report created successfully), I also don't get these. Is it possible to get these returned within the python session?
subprocess.check_output raises an exception if your program does not return zero. Try insert an r before the output string, that maybe is why your program fails.
Also it passes the output as a return value, so you would not see any output even if your program passed.
Changing the output to
output=r'-OutputFile=C:\Aquarius\Reports\Altagas\Summary\24hr_Precip_Forecast.pdf'
Fixed the issue. Thanks you
I am trying to generate a set of files through a python script on a 48-core cluster which actually has a master-and 3 slave machines. I need to generate a set of files, run some scripts on them, collect results and then delete them. I again repeat the process- regenerate files, execute, delete etc.,
When I delete and regenerate files with the same name, I see that the slave machine complains that it cannot find the files.
I am running python script through os.system()
I learnt from this post that it is better to use subprocess.Popen() rather than os.system so that it actually waits for my script to generate my files, before proceeding with the execution. I could use os.system("pause") or time.sleep(whatever) for waiting, but I want to convert my os.systems to subprocess.popens or subprocess.calls and I am stuck here.
I ran through the python documentation and tried out subprocess.Popen('ls'), but I am not able to get a simple thing like subprocess.Popen('cd /whatever_directory') working.
It might sound silly but how do I execute such a simple command like cd through subprocess rather than os.system('cd')?
Then, I actually want to convert the following into subprocess. How do I do it?
import os,optparse
from optparse import OptionParser
parser.add_option("-m", "--mod",dest='module', help='Enter the entity name')
parser.add_option("-f", "--folder", dest="path",help="Enter the path")
module=options.module
path=options.path
os.system('python %s/python_repeat_deckgen_remote.py -m %s' %(path,module))
I just replaced os.system with subprocess.Popen.
But it gave me a lot of complaints:
File "/usr/lib64/python2.6/subprocess.py", line 633, in __init__
errread, errwrite)
File "/usr/lib64/python2.6/subprocess.py", line 1139, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
As NPE already noted, new processes don't affect the existing one (which means os.system('cd /some/where') also has no effect on the current process). In this case, though, I think you're tripping over the fact that os.system invokes a shell to interpret the command you pass in, while subprocess.Popen does not do so by default. But you can tell it to do so:
proc = subprocess.Popen(
'python %s/python_repeat_deckgen_remote.py -m %s' % (path, module),
shell = True)
status = proc.wait()
If you're invoking a shell built-in command or using shell expansions, it's necessary to invoke the shell (assuming you're not willing to simulate it):
>>> import subprocess
>>> x = subprocess.Popen('echo $$')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/local/lib/python2.7/subprocess.py", line 1249, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
>>> x = subprocess.Popen('echo $$', shell = True); x.wait()
81628
0
>>>
but you can bypass the shell—which can help with security issues—if your problem permits, by breaking up the command arguments into a list:
>>> x = subprocess.Popen(['echo', '$$']); x.wait()
$$
0
>>>
Note that this time the output here is the string $$, rather than a process ID, since this time the shell did not interpret the string.
For your original example, for instance, you might use:
proc = subprocess.Popen(['python',
os.path.join(path, 'python_repeat_deckgen_remote.py'),
'-m',
module])
which avoids issues if path and/or module contain characters special to the shell.
(You might even want to call subprocess.Popen with cwd = path and eliminate the call to os.path.join, although that depends on other things.)
I am not able to get a simple thing like subprocess.Popen('cd /whatever_directory') working.
Each process's current working directory is independent of other processes in the system.
When you start a subprocess (using either of these mechanisms), and get it to cd into another directory, that has no effect on the parent process, i.e. on your Python script.
To change the current working directory of your script, you should use os.chdir().
I might be not answering you question directly, but for such tasks which require running subprocess, I always use plumbum.
IMO, It makes the task much simpler and more intuitive, including running on remote machines.
Using plumbum, in order to set subprocess's working directory, you can run the command in a with local.cwd(path): my_cmd() context.
I am attempting to build a large system through a python script. I first need to set up the environment for Visual Studio. Having problems I decided to see if I could just set up and launch Visual Studio. I first set several environment variables and then call C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\vcvarsall.bat x64.
Once this finishes I call devenv /useenv. If I do these from the command prompt everything works fine and I can do what I need to do in VS. My python code for doing this is:
import os
vcdir=os.environ['ProgramFiles(x86)']
arch = 'x64'
command = 'CALL "' +vcdir+'\\Microsoft Visual Studio 11.0\\VC\\vcvarsall.bat" '+arch
os.system(command)
command = "CALL devenv /useenv"
os.system(command)
If I run this, the bat file will run and when it tries the devenv command I get that it is not recognized. It looks like to bat file runs in a different subprocess than the one that the script is running in. I really need to get this running in my current process. My eventual goal is to do the entire build inside the python script and there will be many calls to devenv to do a major portion of the build.
Thank you.
I had the exact same problem as you. I was trying to run vcvarsall.bat as part of a build script written in python and I needed the environment created by vcvarsall. I found a way to do it. First create a wrapper script called setup_environment.bat:
call "C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\vcvarsall.bat" amd64
set > environment.txt
Then in your python script where you would have called vcvarsall.bat, run the wrapper script, then read the environment variables from the text file into your current environment:
this_dir = os.path.dirname(os.path.realpath(__file__)) # path of the currently executing script
os.system('call ' + this_dir + '\setup_environment.bat') # run the wrapper script, creates environment.txt
f = open('environment.txt','r')
lines = f.read().splitlines()
f.close()
os.remove('environment.txt')
for line in lines:
pair = line.split('=',1)
os.environ[pair[0]] = pair[1]
Here is a simple example, should work out of box:
import os
import platform
def init_vsvars():
vswhere_path = r"%ProgramFiles(x86)%/Microsoft Visual Studio/Installer/vswhere.exe"
vswhere_path = os.path.expandvars(vswhere_path)
if not os.path.exists(vswhere_path):
raise EnvironmentError("vswhere.exe not found at: %s", vswhere_path)
vs_path = os.popen('"{}" -latest -property installationPath'.format(vswhere_path)).read().rstrip()
vsvars_path = os.path.join(vs_path, "VC\\Auxiliary\\Build\\vcvars64.bat")
output = os.popen('"{}" && set'.format(vsvars_path)).read()
for line in output.splitlines():
pair = line.split("=", 1)
if(len(pair) >= 2):
os.environ[pair[0]] = pair[1]
if "windows" in platform.system().lower():
init_vsvars()
os.system("where cl.exe")
Please note that environment variables don't have effect after python script exits.
What you are trying to do won't work. vcvarsall.bat sets environment variables, which only work inside a particular process. And there is no way for a Python process to run a CMD.exe process within the same process space.
I have several solutions for you.
One is to run vcvarsall.bat, then figure out all the environment variables it sets and make Python set them for you. You can use the command set > file.txt to save all the environment variables from a CMD shell, then write Python code to parse this file and set the environment variables. Probably too much work.
The other is to make a batch file that runs vcvarsall.bat, and then fires up a new Python interpreter from inside that batch file. The new Python interpreter will be in a new process, but it will be a child process under the CMD.exe process, and it will inherit all the environment variables.
Or, hmm. Maybe the best thing is just to write a batch file that runs vcvarsall.bat and then runs the devenv command. Yeah, that's probably simplest, and simple is good.
The important thing to note is that in the Windows batch file language (and DOS batch file before it), when you execute another batch file, variables set by the other batch file are set in the same environment. In *NIX you need to use a special command source to run a shell script in the same environment, but in batch that's just how it works. But you will never get variables to persist after a process has terminated.
Building on the answer by #derekswanson08 I came up with this which is a bit more fully-fledged. Uses wvwhere.exe which comes with VS 2017.
def init_vsvars():
cprint("")
cprint_header("Initializing vs vars")
if "cygwin" in platform.system().lower():
vswhere_path = "${ProgramFiles(x86)}/Microsoft Visual Studio/Installer/vswhere.exe"
else:
vswhere_path = r"%ProgramFiles(x86)%/Microsoft Visual Studio/Installer/vswhere.exe"
vswhere_path = path.expandvars(vswhere_path)
if not path.exists(vswhere_path):
raise EnvironmentError("vswhere.exe not found at: %s", vswhere_path)
vs_path = common.run_process(".", vswhere_path,
["-latest", "-property", "installationPath"])
vs_path = vs_path.rstrip()
vsvars_path = os.path.join(vs_path, "VC/Auxiliary/Build/vcvars64.bat")
env_bat_file_path = "setup_build_environment_temp.bat"
env_txt_file_path = "build_environment_temp.txt"
with open(env_bat_file_path, "w") as env_bat_file:
env_bat_file.write('call "%s"\n' % vsvars_path)
env_bat_file.write("set > %s\n" % env_txt_file_path)
os.system(env_bat_file_path)
with open(env_txt_file_path, "r") as env_txt_file:
lines = env_txt_file.read().splitlines()
os.remove(env_bat_file_path)
os.remove(env_txt_file_path)
for line in lines:
pair = line.split("=", 1)
os.environ[pair[0]] = pair[1]
What you should be using is a module in the standard library called subprocess
I have linked you to an example in the standard library here.
Here is an example with your situation.
import shlex, subprocess
args = shlex.split(your_command)
p = subprocess.Popen(args)
That should execute it also return stderr if you need to know what happened with the call. your command might even be a third bat file that encompasses the two bat files so you get all the environment variables.
Another solution is to call vcvarsall.bat within the same os.system than the build command:
os.system("cd " + vcFolder + " & vcvarsall.bat amd64 & cd " + buildFolder + " & make")