We Keep Coding

PermissionError: [Errno 13] Permission denied when saving HoloMap to GIF

I am trying to create an animated gig from a series of heat maps with HoloViews.
I need to do this in a Python script, i. e. specifically not in a Jupyter notebook.
When saving the image, Python throws an error because it cannot create a temporary file in the temp-folder of the current user (this is under Windows). Happens regardless of the user, even when I run Python as admin.
When I stop in the debugger and change the temp-file path to some other place, e. g. Desktop, that works, but the resulting holo.gif in the working directory is empty (0 bytes). The temporary gif, though, is correctly animated, so I guess the code is basically OK.
[Edit: Not so sure anymore. I ran this the night through on 26.531 heat maps each of which consisted of a 5x5 grid. The process did not finish (i. e. did not hit the breakppoint at Image.py line 1966). Is there a way to do what I want that is less painfully slow?]
Answers to similar problems on StackOverflow did point to permission problems (but what kind of problem could that be if it doesn't even work for an admin?) and suggest saving to another location, which is impossible here as I have no control over where matplotlib will try to create temporary files.
The problem is specifically with gif's, I can create *.png or *.html output without error. (AFAIK, the difference is that gif-creation uses ImageMagick.)
Here's the code (construction of underlying heat map data left out):
import holoviews as hv
hv.extension('matplotlib')
renderer = hv.renderer('matplotlib')
renderer.fps = 3
heatMapDict = {
k: hv.HeatMap(measurements[k].sensors) for k in range(len(measurements))
}
holo = hv.HoloMap(heatMapDict, kdims='index')
renderer.save(holo, 'holo', fmt='gif')
And the traceback:
INFO:matplotlib.animation:Animation.save using <class 'matplotlib.animation.PillowWriter'>
Traceback (most recent call last):
File "cm3.py", line 69, in <module>
renderer.save(holo, 'holo', fmt='gif')
File "C:\Users\y2046\AppData\Local\Programs\Python\Python37\lib\site-packages\holoviews\plotting\renderer.py", line 554, in save
rendered = self_or_cls(plot, fmt)
File "C:\Users\y2046\AppData\Local\Programs\Python\Python37\lib\site-packages\holoviews\plotting\mpl\renderer.py", line 108, in __call__
data = self._figure_data(plot, fmt, **({'dpi':self.dpi} if self.dpi else {}))
File "C:\Users\y2046\AppData\Local\Programs\Python\Python37\lib\site-packages\holoviews\plotting\mpl\renderer.py", line 196, in _figure_data
data = self._anim_data(anim, fmt)
File "C:\Users\y2046\AppData\Local\Programs\Python\Python37\lib\site-packages\holoviews\plotting\mpl\renderer.py", line 246, in _anim_data
anim.save(f.name, writer=writer, **anim_kwargs)
File "C:\Users\y2046\AppData\Local\Programs\Python\Python37\lib\site-packages\matplotlib\animation.py", line 1174, in save
writer.grab_frame(**savefig_kwargs)
File "C:\Users\y2046\AppData\Local\Programs\Python\Python37\lib\contextlib.py", line 119, in __exit__
next(self.gen)
File "C:\Users\y2046\AppData\Local\Programs\Python\Python37\lib\site-packages\matplotlib\animation.py", line 232, in saving
self.finish()
File "C:\Users\y2046\AppData\Local\Programs\Python\Python37\lib\site-packages\matplotlib\animation.py", line 583, in finish
duration=int(1000 / self.fps))
File "C:\Users\y2046\AppData\Local\Programs\Python\Python37\lib\site-packages\PIL\Image.py", line 1966, in save
fp = builtins.open(filename, "w+b")
PermissionError: [Errno 13] Permission denied: 'C:\\Users\\y2046\\AppData\\Local\\Temp\\tmp4im5ozo8.gif'
Addendum:
I'm coming to think that this is not a permission problem after all. Perhaps it has to do with reentrancy and file-locking under Windows? The Python process in fact may create files in the temp directory, as proved by inserting the following test code before calling renderer.save():
import os
import builtins
filename = 'C:\\Users\\y2046\\AppData\\Local\\Temp\\test.txt'
fp = builtins.open(filename, "w+b")
try:
fp.write("first".encode('utf-8'))
finally:
fp.close()
os.remove(filename)
I should test this under Linux. If it works there, there must be a bug in the Pillow writer.

It looks like there is something broken with HoloViews. I have opened issue #3151 with them.

Permission Error for python on windows

I'm trying to download some files at regular intervals, remove the old ones and replace them with new files. First time it runs well, but the second time it throws an error.
def check_update():
print ('looking for update')
shutil.rmtree(config.destination)
shutil.os.mkdir(config.destination)
threading.Timer(60.0,check_update).start()
def get_videos():
response = requests.get(config.api)
data = response.json()
files = list()
l = len(data)
for i in range(l):
files.append(data[i]['filename'])
return files
def get_newfiles(myfiles):
for i in range(len(myfiles)):
url = config.videos+myfiles[i]
filename = wget.download(url)
def move_files(myfiles):
for i in range(len(myfiles)):
file = myfiles[i]
shutil.move(config.source_files+file,config.destination)
def videos():
files = set(get_videos())
myfiles = list(files)
get_newfiles(myfiles)
move_files(myfiles)
videos()
print ("files are updated")
res = requests.get(config.api)
data = res.json()
return data
data = check_update()
Here is the error.
File "C:\Program Files (x86)\Python36-32\lib\threading.py", line 916, in _bootstrap_inner
self.run()
File "C:\Program Files (x86)\Python36-32\lib\threading.py", line 1182, in run
self.function(*self.args, **self.kwargs)
File "tornado.py", line 8, in check_update
shutil.rmtree(config.destination)
File "C:\Program Files (x86)\Python36-32\lib\shutil.py", line 494, in rmtree
return _rmtree_unsafe(path, onerror)
File "C:\Program Files (x86)\Python36-32\lib\shutil.py", line 389, in _rmtree_unsafe
onerror(os.unlink, fullname, sys.exc_info())
File "C:\Program Files (x86)\Python36-32\lib\shutil.py", line 387, in _rmtree_unsafe
os.unlink(fullname)
permissionError: [WinError 32] The process cannot access the file because it is being used by another process:
How can I overcome this?

The error occurs when attempting to delete config.destination dir. That happens because either the dir itself or one (or more) of its children (may be a dir or a file) is opened in another process (could also be the current one).Typical usecases that frequently lead to this situation:
A cmd console is open in that dir. Just cd outside the dir and try removing it again
A running program has a file opened
As an example could be a Notepad (or an IDE) that has opened a source file located in that dir.
But since it looks like you work with videos, maybe You wanted to check whether a downloaded file works and opened it in a video player.
No matter what the case, closing that program would fix the issue
This is specific to you: I don't know how wget.download works, but if it's not blocking (although according to the code it doesn't seem to be the case) maybe one video from your previous run is still downloading, hence it's open. Closing that python process would do (whether waiting til it finishes or killing it from Task Manager)
Note: When searching for the cause, you should try removing the dir from a file manager (e.g. Windows Explorer) to avoid the overhead introduced by the script.

Error: IOStream has no fileno - SUMO

I am trying to run SUMO through traci interface. I copy pasted this example from this link. The code is as follows
import os, sys
import subprocess
if 'SUMO_HOME' in os.environ:
tools = os.path.join(os.environ['SUMO_HOME'], 'tools')
sys.path.append(tools)
else:
sys.exit("please declare environment variable 'SUMO_HOME'")
PORT = 8813
sumoBinary = "C:/Program Files (x86)/DLR/Sumo/bin/sumo-gui"
sumoProcess = subprocess.Popen([sumoBinary, "-c", "example.sumocfg", \
"--remote-port", str(PORT)], stdout=sys.stdout, stderr=sys.stderr)
import traci
import traci.constants as tc
traci.init(PORT)
traci.vehicle.subscribe(vehID, (tc.VAR_ROAD_ID, tc.VAR_LANEPOSITION))
print(traci.vehicle.getSubscriptionResults(vehID))
for step in range(3):
print("step", step)
traci.simulationStep()
print(traci.vehicle.getSubscriptionResults(vehID))
traci.close()
When I try to run the code, it throws me the following error
File "C:\Anaconda3\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 685, in runfile
execfile(filename, namespace)
File "C:\Anaconda3\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 85, in execfile
exec(compile(open(filename, 'rb').read(), filename, 'exec'), namespace)
File "C:/Users/Raja/Documents/vehicomPhd/SUMOTraffic/traci.py", line 22, in <module>
"--remote-port", str(PORT)], stdout=sys.stdout, stderr=sys.stderr)
File "C:\Anaconda3\lib\subprocess.py", line 823, in __init__
errread, errwrite) = self._get_handles(stdin, stdout, stderr)
File "C:\Anaconda3\lib\subprocess.py", line 1037, in _get_handles
c2pwrite = msvcrt.get_osfhandle(stdout.fileno())
File "C:\Anaconda3\lib\site-packages\IPython\kernel\zmq\iostream.py", line 205, in fileno
raise UnsupportedOperation("IOStream has no fileno.")
UnsupportedOperation: IOStream has no fileno.
Anyone know what is wrong.

Looks like you're running in an ipython notebook. They have non-standard "standard" I/O streams that can't be used like a "true" file object (because they're really data queues, not pipes, so they don't have a file descriptor to use for low level I/O).
You can't use them with libraries (like subprocess) that perform low level I/O; the error is there to tell you this.. You'll need to use a real file-like object, possibly something as simple as sending output to a tempfile.TemporaryFile and then copying output from the file to stdout if that's what you need.
It's possible it would work by just not passing Popen stdout and stderr arguments at all; the default behavior for subprocess is to use the same stdout and stderr as the parent, so if there are valid file handles opened (even if the notebook replaced sys.stdout/sys.stderr for Python use), it might "just work" (where "just work" includes the possibility that data sent to the underlying file descriptors 0 and 1 is discarded, so you never see it).
Or just don't run in the ipython notebook.

The problem was that my command line used Python 2 whereas the spyder IDE used python 3. Since I wont be needing to pass any more arguments to the stdin, I removed the option and simply tried to open the sumo-gui with subprocess as follows. It works now.
PORT = 8813
sumoBinary = 'C:/Program Files (x86)/DLR/Sumo/bin/sumo-gui'
sumoProcess = subprocess.Popen([sumoBinary, "-c", "Kaiserslautern.sumocfg", \
"--remote-port", str(PORT)])

Why subprocess.call () is showing error in linux?

Let us consider Linux platform where I need to execute a program called smart.exe which uses input.dat file. Both the files are placed in the same directory with each file having the same file permission 777.
Now if I run the following command in the terminal window smart.exe is fully executed without any error.
$./smart.exe input.dat
On the other hand, if I use the following python script called my_script.py placed in the same directory, then I get an error.
my_script.py has the following code:
#!/usr/bin/python
import os, subprocess
exit_code = subprocess.call("./smart.exe input.dat", shell = False)
The error is as follows:
File "my_script.py", line 4, in <module>
exit_code = subprocess.call("./smart.exe input.dat", shell = False)
File "/usr/lib64/python2.6/subprocess.py", line 478, in call
p = Popen(*popenargs, **kwargs)
File "/usr/lib64/python2.6/subprocess.py", line 642, in __init__
errread, errwrite)
File "/usr/lib64/python2.6/subprocess.py", line 1234, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Can someone please tell me why this is happening. Please note that the smart.exe should take around 10 sec to fully complete. This may be a clue for the problem.
Please also advise if there is any other way to run smart.exe from my_script.py. Your solution is much appreciated!

You should decide if you want shell support or not.
If you want the shell to be used (which is not necessary here), you should use exit_code = subprocess.call("./smart.exe input.dat", shell=True). Then the shell interprets your command line.
If you don't want it (as you don't need it and want to avoid unnecessary complexity), you should do exit_code = subprocess.call(["./smart.exe", "input.dat"], shell=False).
(And there is no point naming your binarys .exe under Linux.)

rar file module not working in python 2.7

i have a code like this
import rarfile
pwd = None
rar = rarfile.RarFile(source_filename)
rar.extractall(dest_dir,None,pwd) # error from here
this code in working in ubuntu.
when i run this on windows i get error like this
Traceback (most recent call last):
File "1_bete_rar.pyw", line 132, in extract
File "1_bete_rar.pyw", line 176, in unrar_file
File "rarfile.pyc", line 586, in extractall
File "rarfile.pyc", line 1112, in _extract
File "rarfile.pyc", line 1704, in custom_popen
File "subprocess.pyc", line 711, in __init__
File "subprocess.pyc", line 948, in _execute_child
WindowsError: [Error 2] The system cannot find the file specified
what is the problem with my code? how can i extract rar file with python in windows?

As the rarfile FAQ states (and as is made evident by traces of subprocess in the stack trace),
[rarfile] depends on unrar command-line utility to do the actual decompression.
Note that by default it expect it to be in PATH. If unrar launching fails, you need to fix this.
So get UnRAR from http://www.rarlab.com/rar_add.htm and put it somewhere in your PATH (such as the directory you're running your script from).

Looks like source_filename isn't pointing to a valid RAR file, do this little check before, just to be sure:
import os.path
os.path.isfile(source_filename) # what's the value returned?
If the file exists, then check if the path is in the correct format. For example, this won't work:
source_filename = 'c:\documents\file.rar'
Try this instead:
source_filename = 'c:\\documents\\file.rar'
Or even better, use raw strings:
source_filename = r'c:\documents\file.rar'

One common problem using python with Windows is that the path separator is \, but this is a special character that needs to be escaped in Python. If you print the source_filename you should be able to see if this is set correctly.
e.g.
source_filename = 'c:\users\prosserc\documents\myfile.txt'
will not work correctly. Here are a couple of alternatives:
Use a raw string:
source_filename = r'c:\users\prosserc\documents\myfile.txt'
or use os.path.join to join to an eviornment variable such as user_profile
source_filename = os.path.join(os.getenv('userprofile'), 'documents', 'myfile.txt')