We Keep Coding

Python - Using subprocess to Pipe R Script, resultant Winerror 2

I'm trying to run an R Script through Python using subprocess but unfortunately, I'm continually getting the following error:
WindowsError: [Error 2] The system cannot find the file specified
This is extremely frustrating as I've checked the path over everything multiple times (done it without C:/, moved to different directories, changed the R script I'm trying to run, etc). I've also checked the CompSec Environment Variable, and it is also correct (through Windows Systems, although I never checked it directly in my IDE, Spyder). I've tried doing it in Python 2.7 and Python 3.5, and neither version works.
The code goes as follows:
import subprocess
def sort_ByInputColumn(inputPath, inputFileTermination, sortColumn, outputPath, outputFileTermination):
scriptPath = "C:/Users/Kyle/Documents/Rscript_SortByInputColumn.R"
subprocess.call(["Rscript", scriptPath, inputPath, inputFileTermination, sortColumn, outputPath, outputFileTermination])
fileName = 'Alabama'
outputPath = "C:/Users/Kyle/Documents/HillData/Data/Output/Module2/"
sortColumn = str(16)
inputTermination = fileName + 'Module2NN_WorkCounty_Work.csv'
outputFileTermination = fileName + 'Module2NN_SortedWorkCounty.csv'
sort_ByInputColumn(outputPath, inputTermination, sortColumn, outputPath, outputFileTermination)
The fact that I get this error no matter what code I try to run (even blatantly copy-pasting this tutorial to try and make it work) makes me feel like something deeper (or something extremely obvious) is going on that I'm not seeing.
Would appreciate any feedback on the matter.

To solve this most annoying and horrendous problem, I reinstalled R into a directory with no spaces and called Rscript with its full pathing (that is "C:/R/R-3.3.1/bin/Rscript.exe" from C:/Program Files/R/R-3.3.1/bin/Rscript.exe" because Program Files has a space and this kills subprocess because it's command line based, I guess). This time, it worked.
See this similar question for a hint as to where I got the inspiration for this.

Python doesn't delete temporary file if the program fails

I'm writing a program, which, inter alia, works with temporary file, created using tempfile library.
The temporary file creates and fills in function:
def func():
mod_script = tempfile.NamedTemporaryFile(dir='special')
dest = open(mod_script, 'w')
# filling dest
return mod_script
(I use open() and not with open() because I execute the temporary file after calling func())
After some operations with mod_script outside func(), I call mod_script.close(). And all works fine.
But I have one problem. If my program fails (or if I interrupt it), the temporary file doesn't remove.
How do I fix it ?
I really don't want to write try...except...finally clauses because I'll have to write it so many times (there are many points, where my program can fail).

First, use a with statement, and pass delete=False to the constructor.
Then you need to put the necessary error handling in your program. Catch exceptions (see try..finally) and clean up during program exit whether it is successful or crashes.
Alternatively, keep the file open while executing it to prevent the automatic deletion-on-close from deleting it before you have executed it. This may have issues on Windows where it tends to have conflicts using files that are open.

Running a bash file with Python

I've got a bash file that I normally execute using Cygwin.
I need to run this file from my Python code.
I tried this:
for bashfile in files:
p = Popen(bashfile, cwd=dname) #dname is the current directory of the script
stdout, stderr = p.communicate()
I've also seen a similar question here, but when trying to run it that way it says that it can't find the directory of my bash file...
Any ideas? Thanks! :-)
Edit: bashfile has a full path.

Do you need its output to get it directly to Python? If not this may be very fast and easy solution:
os.system("""here some code you use to execute in Terminal""")

You can also try this, though it does (and will no matter what you try) matter where the directory is. This, as far as the output goes, may be a little bit cleaner than the os method.
import commands
cmd="bash ./script.sh"
commands.getoutput(cmd)
If the case is that you need to change the directory:
cmd = "/path/to/your/script/script.sh"
The added benefit of using this method, versus say, os is that you can assign the output to a variable...
fun_times = commands.getoutput("bash ./script.sh")
whereas...
not_fun_times = os.system("./script.sh")
will throw an error.
etc, etc.

How come my Python code doesn't work?

from celery.decorators import task
from celery.decorators import task
#task()
def add(x, y):
r = open("./abc.txt","w")
r.write("sdf")
r.close()
return x + y
That's my tasks.py file.
>>> import tasks
>>> r = tasks.add.delay(3,5)
>>> r.result
8
As you can see, the function works. However, the file does not create.
Why?
I've tried changing multiple file paths, due to possible permission issues. but no luck.

If the file was not being written, you would get an exception, so the function cannot possibly complete.
Since the function is returning 8, it follows that the file is being written somewhere.
Perhaps the file is written in a different directory to the one you are expecting
The only other possibility I can think of is that the add function that is being run is not the one that you have shown here

I think the problem is that you're running this by importing a module. The . in the file path is relative to where the module lives, not your current working directory. Try giving it a full path name.
If that doesn't work, show us exactly where you're running the script from and an ls -la on that directory. And if that still doesn't show anything abnormal. Do a
find / -name abc.txt

The working directory may not be the one you expect.
The task may be done on another host, if you use multiple hosts.

In Python, how do I make a temp file that persists until the next run?

I need to create a folder that I use only once, but need to have it exist until the next run. It seems like I should be using the tmp_file module in the standard library, but I'm not sure how to get the behavior that I want.
Currently, I'm doing the following to create the directory:
randName = "temp" + str(random.randint(1000, 9999))
os.makedirs(randName)
And when I want to delete the directory, I just look for a directory with "temp" in it.
This seems like a dirty hack, but I'm not sure of a better way at the moment.
Incidentally, the reason that I need the folder around is that I start a process that uses the folder with the following:
subprocess.Popen([command], shell=True).pid
and then quit my script to let the other process finish the work.

Creating the folder with a 4-digit random number is insecure, and you also need to worry about collisions with other instances of your program.
A much better way is to create the folder using tempfile.mkdtemp, which does exactly what you want (i.e. the folder is not deleted when your script exits). You would then pass the folder name to the second Popen'ed script as an argument, and it would be responsible for deleting it.

What you've suggested is dangerous. You may have race conditions if anyone else is trying to create those directories -- including other instances of your application. Also, deleting anything containing "temp" may result in deleting more than you intended. As others have mentioned, tempfile.mkdtemp is probably the safest way to go. Here is an example of what you've described, including launching a subprocess to use the new directory.
import tempfile
import shutil
import subprocess
d = tempfile.mkdtemp(prefix='tmp')
try:
subprocess.check_call(['/bin/echo', 'Directory:', d])
finally:
shutil.rmtree(d)

"I need to create a folder that I use only once, but need to have it exist until the next run."
"Incidentally, the reason that I need the folder around is that I start a process ..."
Not incidental, at all. Crucial.
It appears you have the following design pattern.
mkdir someDirectory
proc1 -o someDirectory # Write to the directory
proc2 -i someDirectory # Read from the directory
if [ %? == 0 ]
then
rm someDirectory
fi
Is that the kind of thing you'd write at the shell level?
If so, consider breaking your Python application into to several parts.
The parts that do the real work ("proc1" and "proc2")
A Shell which manages the resources and processes; essentially a Python replacement for a bash script.

A temporary file is something that lasts for a single program run.
What you need is not, therefore, a temporary file.
Also, beware of multiple users on a single machine - just deleting anything with the 'temp' pattern could be anti-social, doubly so if the directory is not located securely out of the way.
Also, remember that on some machines, the /tmp file system is rebuilt when the machine reboots.

You can also automatically register an function to completely remove the temporary directory on any exit (with or without error) by doing :
import atexit
import shutil
import tempfile
# create your temporary directory
d = tempfile.mkdtemp()
# suppress it when python will be closed
atexit.register(lambda: shutil.rmtree(d))
# do your stuff...
subprocess.Popen([command], shell=True).pid

tempfile is just fine, but to be on a safe side you'd need to safe a directory name somewhere until the next run, for example pickle it. then read it in the next run and delete directory. and you are not required to have /tmp for the root, tempfile.mkdtemp has an optional dir parameter for that. by and large, though, it won't be different from what you're doing at the moment.

The best way of creating the temporary file name is either using tempName.TemporaryFile(mode='w+b', suffix='.tmp', prifix='someRandomNumber' dir=None)
or u can use mktemp() function.
The mktemp() function will not actually create any file, but will provide a unique filename (actually does not contain PID).