I'm trying to use subprocess.run to build my CMake project but the code finishing without errors but its not working.
the code is:
import subprocess
git_repo_remvoe_destination = '/home/yaodav/Desktop/'
git_repo_clone_destination = git_repo_remvoe_destination + 'test_clone_git'
test_path = git_repo_clone_destination + "/test/"
cmake_debug_path = test_path + "cmake-debug-build"
cmake_build_command = " cmake -Bcmake-build-debug -H. -DCMAKE_BUILD_TYPE=debug -DCMAKE_C_COMPILER=/usr/local/bin/gcc " \
"-DCMAKE_CXX_COMPILER=/usr/local/bin/c++ -Bcmake-build-debug -H. " \
"-DSYTEM_ARCH=x86_64-pc-linux-gnu-gcc-7.4.0"
cmake_link_command = "cmake --build cmake-build-debug --target all"
cmake_command = ['cd '+test_path, cmake_build_command, cmake_link_command]
out = subprocess.run(cmake_command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
I tried this answer but it didn't work
how to I do that?
2 Issues.
First you should call subprocess.run() once for each command instead of putting three different commands in a list.
Second: The cd ... command just changes the present working directory in one sub process and the consecutive command will not be any more in the same directory.
However there is a simple solution to it.
subprocess.run has a cwd parameter ( https://docs.python.org/2/library/subprocess.html#popen-constructor ) that allows you to specify the directory in which a subprocess should be executed.
So Following should do:
out = subprocess.run(cmake_build_command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, cwd=test_path, shell=True)
out += subprocess.run(cmake_link_command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, cwd=test_path, shell=True)
Related
I want to delete all the below files:
20200922_051424_00011_v4wzh_db508ed0-b8b9-488b-a796-773d1fb4045c_08
20200922_051424_00011_v4wzh_db508ed0-b8b9-488b-a796-773d1fb4045c_04 20200922_051424_00011_v4wzh_db508ed0-b8b9-488b-a796-773d1fb4045c_09
20200922_051424_00011_v4wzh_db508ed0-b8b9-488b-a796-773d1fb4045c_05 20200922_051424_00011_v4wzh_db508ed0-b8b9-488b-a796-773d1fb4045c_10
In Linux I simply do:
rm 20200922_051424_00011_v4wzh_db508ed0-b8b9-488b-a796-773d1fb4045c_*
But when I am doing the same using python script. It is just deleting first file matching the pattern but not all of them:
temp = subprocess.Popen('rm 20200922_051424_00011_v4wzh_db508ed0-b8b9-488b-a796-773d1fb4045c_*', shell=True, stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
Can anyone tell me the reason why its not working and also what should I do?
Complete python function is:
def remove(filename):
try:
cmd = 'rm ' + filename
print(cmd)
temp = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
stdout, stderr = temp.communicate()
if stderr:
print('Error while running rm command.')
print("Result of running rm command: ", stdout)
except CalledProcessError as e:
pass
Since you're in python, why not remove them directly from python rather than calling a shell command?
for filename in glob.glob(pattern):
os.remove(filename)
Documentation:
os.remove()
glob.glob()
I need to gzip files of size more than 10 GB using python on top of shell commands and hence decided to use subprocess Popen.
Here is my code:
outputdir = '/mnt/json/output/'
inp_cmd='gzip -r ' + outputdir
pipe = Popen(["bash"], stdout =PIPE,stdin=PIPE,stderr=PIPE)
cmd = bytes(inp_cmd.encode('utf8'))
stdout_data,stderr_data = pipe.communicate(input=cmd)
It is not gzip-ing the files within output directory.
Any way out?
The best way is to use subprocess.call() instead of subprocess.communicate().
call() waits till the command is executed completely while in Popen(), one has to extrinsically use wait() method for the execution to finish.
Have you tried it like this:
output_dir = "/mnt/json/output/"
cmd = "gzip -r {}".format(output_dir)
proc = subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
stdin=subprocess.PIPE,
shell=True,
)
out, err = proc.communicate()
I'm trying to execute 10 python scripts from python code and open each of them in a new shell window.
My code :
for i in range(10):
name_of_file = "myscript"+str(i)+".py"
cmd = "python " + name_of_file
os.system("gnome-terminal -e 'bash -c " + cmd + "'")
But each script file are not executing, I get only the live interpreter of python in the new terminal...
Thank you guys
I would suggest using the subprocess module (https://docs.python.org/2/library/subprocess.html).
In this way, you'll write something like the following:
import subprocess
cmd = ['gnome-terminal', '-x', 'bash', '-c']
for i in range(10):
name_of_file = "myscript"+str(i)+".py"
your_proc = subprocess.Popen(cmd + ['python %s' % (name_of_file)])
# or if you want to use the "modern" way of formatting string you can write
# your_proc = subprocess.Popen(cmd + ['python {}'.format(name_of_file)])
...
and you have more control over the processes you start.
If you want to keep using os.system(), build your command string first, then pass it to the function. In your case would be:
cmd = 'gnome-terminal -x bash -c "python {}"'.format(name_of_file)
os.system(cmd)
something along these lines.
Thanks to #anishsane for some suggestions!
I think that it is to do with the string quoting of the argument to os.system. Try this:
os.system("""gnome-terminal -e 'bash -c "{}"'""".format(cmd))
I have an .R file saved locally at the following path:
Rfilepath = "C:\\python\\buyback_parse_guide.r"
The command for RScript.exe is:
RScriptCmd = "C:\\Program Files\\R\\R-2.15.2\\bin\\Rscript.exe --vanilla"
I tried running:
subprocess.call([RScriptCmd,Rfilepath],shell=True)
But it returns 1 -- and the .R script did not run successfully. What am I doing wrong? I'm new to Python so this is probably a simple syntax error... I also tried these, but they all return 1:
subprocess.call('"C:\Program Files\R\R-2.15.2\bin\Rscript.exe"',shell=True)
subprocess.call('"C:\\Program Files\\R\\R-2.15.2\\bin\\Rscript.exe"',shell=True)
subprocess.call('C:\Program Files\R\R-2.15.2\bin\Rscript.exe',shell=True)
subprocess.call('C:\\Program Files\\R\\R-2.15.2\\bin\\Rscript.exe',shell=True)
Thanks!
The RScriptCmd needs to be just the executable, no command line arguments. So:
RScriptCmd = "\"C:\\Program Files\\R\\R-2.15.2\\bin\\Rscript.exe\""
Then the Rfilepath can actually be all of the arguments - and renamed:
RArguments = "--vanilla \"C:\\python\\buyback_parse_guide.r\""
It looks like you have a similar problem to mine. I had to reinstall RScript to a path which has no spaces.
See: Running Rscript via Python using os.system() or subprocess()
This is how I worked out the communication between Python and Rscript:
part in Python:
from subprocess import PIPE,Popen,call
p = subprocess.Popen([ path/to/RScript.exe, path/to/Script.R, Arg1], stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
out = p.communicate()
outValue = out[0]
outValue contains the output-Value after executing the Script.R
part in the R-Script:
args <- commandArgs(TRUE)
argument1 <- as.character(args[1])
...
write(output, stdout())
output is the variable to send to Python
When I execute the script in R, it is:
$ R --vanilla --args test_matrix.csv < hierarchical_clustering.R > out.txt
In Python, it works if I use:
process = subprocess.call("R --vanilla --args "+output_filename+"_DM_Instances_R.csv < /home/kevin/AV-labels/Results/R/hierarchical_clustering.R > "+output_filename+"_out.txt", shell=True)
But this method doesn't provide the process.wait() function.
So, I would like to use the subprocess.Popen, I tried:
process = subprocess.Popen(['R', '--vanilla', '--args', "\'"+output_filename+"_DM_Instances_R.csv\'", '<', '/home/kevin/AV-labels/Results/R/hierarchical_clustering.R'])
But it didn't work, Python just opened R but didn't execute my script.
Instead of 'R', give it the path to Rscript. I had the same problem. Opens up R but doesn't execute my script. You need to call Rscript (instead of R) to actually execute the script.
retcode = subprocess.call("/Pathto/Rscript --vanilla /Pathto/test.R", shell=True)
This works for me.
Cheers!
I've solved this problem by putting everything into the brackets..
process = subprocess.Popen(["R --vanilla --args "+output_filename+"_DM_Instances_R.csv < /home/kevin/AV-labels/Results/R/hierarchical_clustering.R > "+output_filename+"_out.txt"], shell=True)
process.wait()
A couple of ideas:
You might want to consider using the Rscript frontend, which makes
running scripts easier; you can pass the script filename directly
as a parameter, and do not need to read the script in through standard input.
You don't need the shell for just redirecting standard output to a file, you can
do that directly with subprocess.Popen.
Example:
import subprocess
output_name = 'something'
script_filename = 'hierarchical_clustering.R'
param_filename = '%s_DM_Instances_R.csv' % output_name
result_filename = '%s_out.txt' % output_name
with open(result_filename, 'wb') as result:
process = subprocess.Popen(['Rscript', script_filename, param_filename],
stdout=result);
process.wait()
You never actually execute it fully ^^ try the following
process = subprocess.Popen(['R', '--vanilla', '--args', '\\%s_DM_Instances_R.csv\\' % output_filename, '<', '/home/kevin/AV-labels/Results/R/hierarchical_clustering.R'], stdout=subprocess.PIPE, stdin=subprocess.PIPE, shell=True)
process.communicate()#[0] is stdout
Keven's solution works for my requirement. Just to give another example about #Kevin's solution. You can pass more parameters to the rscript with python-style string:
import subprocess
process = subprocess.Popen(["R --vanilla --args %s %d %.2f < /path/to/your/rscript/transformMatrixToSparseMatrix.R" % ("sparse", 11, 0.98) ], shell=True)
process.wait()
Also, to make things easier you could create an R executable file. For this you just need to add this in the first line of the script:
#! /usr/bin/Rscript --vanilla --default-packages=utils
Reference: Using R as a scripting language with Rscript or this link