I'm trying to download some files at regular intervals, remove the old ones and replace them with new files. First time it runs well, but the second time it throws an error.
def check_update():
print ('looking for update')
shutil.rmtree(config.destination)
shutil.os.mkdir(config.destination)
threading.Timer(60.0,check_update).start()
def get_videos():
response = requests.get(config.api)
data = response.json()
files = list()
l = len(data)
for i in range(l):
files.append(data[i]['filename'])
return files
def get_newfiles(myfiles):
for i in range(len(myfiles)):
url = config.videos+myfiles[i]
filename = wget.download(url)
def move_files(myfiles):
for i in range(len(myfiles)):
file = myfiles[i]
shutil.move(config.source_files+file,config.destination)
def videos():
files = set(get_videos())
myfiles = list(files)
get_newfiles(myfiles)
move_files(myfiles)
videos()
print ("files are updated")
res = requests.get(config.api)
data = res.json()
return data
data = check_update()
Here is the error.
File "C:\Program Files (x86)\Python36-32\lib\threading.py", line 916, in _bootstrap_inner
self.run()
File "C:\Program Files (x86)\Python36-32\lib\threading.py", line 1182, in run
self.function(*self.args, **self.kwargs)
File "tornado.py", line 8, in check_update
shutil.rmtree(config.destination)
File "C:\Program Files (x86)\Python36-32\lib\shutil.py", line 494, in rmtree
return _rmtree_unsafe(path, onerror)
File "C:\Program Files (x86)\Python36-32\lib\shutil.py", line 389, in _rmtree_unsafe
onerror(os.unlink, fullname, sys.exc_info())
File "C:\Program Files (x86)\Python36-32\lib\shutil.py", line 387, in _rmtree_unsafe
os.unlink(fullname)
permissionError: [WinError 32] The process cannot access the file because it is being used by another process:
How can I overcome this?
The error occurs when attempting to delete config.destination dir. That happens because either the dir itself or one (or more) of its children (may be a dir or a file) is opened in another process (could also be the current one).Typical usecases that frequently lead to this situation:
A cmd console is open in that dir. Just cd outside the dir and try removing it again
A running program has a file opened
As an example could be a Notepad (or an IDE) that has opened a source file located in that dir.
But since it looks like you work with videos, maybe You wanted to check whether a downloaded file works and opened it in a video player.
No matter what the case, closing that program would fix the issue
This is specific to you: I don't know how wget.download works, but if it's not blocking (although according to the code it doesn't seem to be the case) maybe one video from your previous run is still downloading, hence it's open. Closing that python process would do (whether waiting til it finishes or killing it from Task Manager)
Note: When searching for the cause, you should try removing the dir from a file manager (e.g. Windows Explorer) to avoid the overhead introduced by the script.
Related
I am trying to create an animated gig from a series of heat maps with HoloViews.
I need to do this in a Python script, i. e. specifically not in a Jupyter notebook.
When saving the image, Python throws an error because it cannot create a temporary file in the temp-folder of the current user (this is under Windows). Happens regardless of the user, even when I run Python as admin.
When I stop in the debugger and change the temp-file path to some other place, e. g. Desktop, that works, but the resulting holo.gif in the working directory is empty (0 bytes). The temporary gif, though, is correctly animated, so I guess the code is basically OK.
[Edit: Not so sure anymore. I ran this the night through on 26.531 heat maps each of which consisted of a 5x5 grid. The process did not finish (i. e. did not hit the breakppoint at Image.py line 1966). Is there a way to do what I want that is less painfully slow?]
Answers to similar problems on StackOverflow did point to permission problems (but what kind of problem could that be if it doesn't even work for an admin?) and suggest saving to another location, which is impossible here as I have no control over where matplotlib will try to create temporary files.
The problem is specifically with gif's, I can create *.png or *.html output without error. (AFAIK, the difference is that gif-creation uses ImageMagick.)
Here's the code (construction of underlying heat map data left out):
import holoviews as hv
hv.extension('matplotlib')
renderer = hv.renderer('matplotlib')
renderer.fps = 3
heatMapDict = {
k: hv.HeatMap(measurements[k].sensors) for k in range(len(measurements))
}
holo = hv.HoloMap(heatMapDict, kdims='index')
renderer.save(holo, 'holo', fmt='gif')
And the traceback:
INFO:matplotlib.animation:Animation.save using <class 'matplotlib.animation.PillowWriter'>
Traceback (most recent call last):
File "cm3.py", line 69, in <module>
renderer.save(holo, 'holo', fmt='gif')
File "C:\Users\y2046\AppData\Local\Programs\Python\Python37\lib\site-packages\holoviews\plotting\renderer.py", line 554, in save
rendered = self_or_cls(plot, fmt)
File "C:\Users\y2046\AppData\Local\Programs\Python\Python37\lib\site-packages\holoviews\plotting\mpl\renderer.py", line 108, in __call__
data = self._figure_data(plot, fmt, **({'dpi':self.dpi} if self.dpi else {}))
File "C:\Users\y2046\AppData\Local\Programs\Python\Python37\lib\site-packages\holoviews\plotting\mpl\renderer.py", line 196, in _figure_data
data = self._anim_data(anim, fmt)
File "C:\Users\y2046\AppData\Local\Programs\Python\Python37\lib\site-packages\holoviews\plotting\mpl\renderer.py", line 246, in _anim_data
anim.save(f.name, writer=writer, **anim_kwargs)
File "C:\Users\y2046\AppData\Local\Programs\Python\Python37\lib\site-packages\matplotlib\animation.py", line 1174, in save
writer.grab_frame(**savefig_kwargs)
File "C:\Users\y2046\AppData\Local\Programs\Python\Python37\lib\contextlib.py", line 119, in __exit__
next(self.gen)
File "C:\Users\y2046\AppData\Local\Programs\Python\Python37\lib\site-packages\matplotlib\animation.py", line 232, in saving
self.finish()
File "C:\Users\y2046\AppData\Local\Programs\Python\Python37\lib\site-packages\matplotlib\animation.py", line 583, in finish
duration=int(1000 / self.fps))
File "C:\Users\y2046\AppData\Local\Programs\Python\Python37\lib\site-packages\PIL\Image.py", line 1966, in save
fp = builtins.open(filename, "w+b")
PermissionError: [Errno 13] Permission denied: 'C:\\Users\\y2046\\AppData\\Local\\Temp\\tmp4im5ozo8.gif'
Addendum:
I'm coming to think that this is not a permission problem after all. Perhaps it has to do with reentrancy and file-locking under Windows? The Python process in fact may create files in the temp directory, as proved by inserting the following test code before calling renderer.save():
import os
import builtins
filename = 'C:\\Users\\y2046\\AppData\\Local\\Temp\\test.txt'
fp = builtins.open(filename, "w+b")
try:
fp.write("first".encode('utf-8'))
finally:
fp.close()
os.remove(filename)
I should test this under Linux. If it works there, there must be a bug in the Pillow writer.
It looks like there is something broken with HoloViews. I have opened issue #3151 with them.
Question: How can I change file permissions on a Windows 10 PC with a Python script?
I have written a Python script that takes folders, which are created by proprietary software, and moves them to a network drive with shutil.move().
It seems that the proprietary software creates folders that are read-only by default. I need to change the file permissions for these folders in order for shutil.move() to delete the folders after they are copied to the network drive.
I have searched on SO to discover that os.chmod(path, 0o777) only works to grant access on Unix systems. On Windows, it modifies the read-only attribute of a file or folder. This question seems to yield a solution, which I tried as follows:
import win32security
import ntsecuritycon as con
account = r"admin"
userx, domain, type = win32security.LookupAccountName ("", account)
sd = win32security.GetFileSecurity(path, win32security.DACL_SECURITY_INFORMATION)
dacl = sd.GetSecurityDescriptorDacl() # instead of dacl = win32security.ACL()
dacl.AddAccessAllowedAce(win32security.ACL_REVISION, con.FILE_GENERIC_READ | con.FILE_GENERIC_WRITE, userx)
sd.SetSecurityDescriptorDacl(1, dacl, 0) # may not be necessary
win32security.SetFileSecurity(path, win32security.DACL_SECURITY_INFORMATION, sd)
But it does not seem to work. Also, I don't understand what I am doing with the modules win32security and ntsecuritycon. Maybe someone can give an easy explanation.
edit: ok so i looked at stuff. This is the exception that gets raised:
Traceback (most recent call last):
File "copyscript.py", line 108, in <module>
copyscript()# the loop needs to be called as a function to delete all assigned variables after each loop
File "copyscript.py", line 93, in copyscript
shutil.move(run, str(target_dir2))#move files renamed to user folder
File "C:\Users\admin\AppData\Local\Programs\Python\Python35\lib\shutil.py", line 550, in move
rmtree(src)
File "C:\Users\admin\AppData\Local\Programs\Python\Python35\lib\shutil.py", line 488, in rmtree
return _rmtree_unsafe(path, onerror)
File "C:\Users\admin\AppData\Local\Programs\Python\Python35\lib\shutil.py", line 378, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "C:\Users\admin\AppData\Local\Programs\Python\Python35\lib\shutil.py", line 383, in _rmtree_unsafe
onerror(os.unlink, fullname, sys.exc_info())
File "C:\Users\admin\AppData\Local\Programs\Python\Python35\lib\shutil.py", line 381, in _rmtree_unsafe
os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'THG126.D\\AcqData\\sample_info.xml'
the full path of this file is D:\MSD_Data\THG126.D\AcqData\sample_info.xml.
the user account is named "admin" and it belongs to the "Administrators" group.
"admin" is the owner and has "full control" according to the "advanced security settings" for MSD_Data, THG126.D, sample_info.xml and the python script.
i have also tried running the script via CLI using "run as administrator". The same error occurs.
i looked at all files in the folders and found that only sample_info.xml has RA attributes, whereas all other have only A, so i added
path2 = r"D:\\MSD_data\\"+run+r"\\AcqData\\sample_info.xml"
subprocess.check_call(["attrib", "-r", path2, "/S", "/D"])
to the script and it seems to work now. I need to wait a little for new folders to be generated by the other software to see if the script is working correctly now.
The problem seems to have been that a file had the attribute "RA", which means "read-only" and "archived". Even though the used user account wqas the owner of all files and folders, shutil.move() fails when it tries to delete the file after copying to the target location.
A workaround to this problem is to use
subprocess.check_call(["attrib", "-r", path])
to remove the read-only file attribute. This resolved my issue. If you still have trouble with shutil.move() you could also try this solution.
I'm trying to code a little script that watches a defined directory with a while-loop. Every file or directory that is in this directory is compressed to RAR and moved to another directory after the process is completed.
My problem: everytime I copy a file or folder to this directory, the script doesn't wait and startes the process the second it sees a new file or folder. But when the files or folders are bigger than a few kilobytes the loop breaks with a permission error.
Since I'm a Python beginner I don't know which module to use. Is there a checking module to see if the file or folder that the tool wants to process is used by another process? Or am I going in the wrong direction?
Edit: added the code for directory-only listening:
watchDir = "L:\\PythonTest\\testfolder\\"
finishedDir = "L:\\PythonTest\\finishedfolders\\"
rarfilesDir = "L:\\PythonTest\\rarfiles\\"
rarExe = "L:\\PythonTest\\rar.exe"
rarExtension = ".rar"
rarCommand = "a"
while True:
dirList = [name for name in os.listdir(watchDir) if os.path.isdir(os.path.join(watchDir,name))]
for entryName in dirList:
if not os.path.exists((os.path.join(finishedDir,entryName))):
sourcePath = os.path.join(watchDir,entryName)
entryNameStripped = entryName.replace(" ", "")
os.chdir(watchDir)
archiveName = rarfilesDir+entryNameStripped+rarExtension
subprocesscall = [rarExe, rarCommand, archiveName, entryName]
subprocess.call(subprocesscall, shell=True)
shutil.move(sourcePath,finishedDir)
When I run the script and try to add a file of several GB (named #filename# in the following lines) these errors occur:
Creating archive L:\PythonTest\rarfiles\#filename#.rar
Cannot open #filename#
The process cannot access the file, since it's used by another process.
Adding #filename# OK
WARNING: Cannot open 1 file
Done
Traceback (most recent call last):
File "C:\Python34\lib\shutil.py", line 522, in move
os.rename(src, real_dst)
PermissionError: [WinError 5] Access denied: #filepath#
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "L:/Python Test/test.py", line 35, in <module>
shutil.move(sourcePath,finishedDir)
File "C:\Python34\lib\shutil.py", line 531, in move
copytree(src, real_dst, symlinks=True)
File "C:\Python34\lib\shutil.py", line 342, in copytree
raise Error(errors)
shutil.Error: #filepath#
instead of using os.listdir, you can use os.walk, os.walk yields 3 tuple dirpath(path of directory,filenames(all files in that dirpath),dirnames(all the sub directories in dirpath)
for x,y,z in os.walk('path-of-directory'):
do you stuff with x,y,z the three tuples
I'm learning web programming with Python, and still basically going through lectures/tutorial.
I'm trying to upload a file to a server. This is my code:
import ftplib
import sys
filename = sys.argv[1]
connect = ftplib.FTP("***.**.***.**")
connect.login("testuser","pass")
file = open(filename, "rb")
connect.storbinary("STOR " + filename, file)
connect.quit()
and this is the error I have:
File "C:\Users\test\putfile.py", line 8, in <module>
connect.storbinary("STOR " + filename, file)
File "C:\Python27\lib\ftplib.py", line 471, in storbinary
conn = self.transfercmd(cmd, rest)
File "C:\Python27\lib\ftplib.py", line 376, in transfercmd
return self.ntransfercmd(cmd, rest)[0]
File "C:\Python27\lib\ftplib.py", line 339, in ntransfercmd
resp = self.sendcmd(cmd)
File "C:\Python27\lib\ftplib.py", line 249, in sendcmd
return self.getresp()
File "C:\Python27\lib\ftplib.py", line 224, in getresp
raise error_perm, resp
ftplib.error_perm: 550 Permission denied.
testuser should have the permission to write files, since the folder is owned by him, and he has root privilege(was added in sudoer file).
the same thing happens if I add the line:
connect.cwd('/testfolder')
I will get error_perm: 550 Failed to change directory.
However I can still read the existing files just fine (with
connect.retrlines("RETR " + filename))
I'm pretty new about Python as well as Linux, so I don't have idea what I'm doing. I need some help.
Perhaps this can help:
With FTP is not sufficient be owner of files and directories.
The service and daemon FTP must be correctly configured in order to write and create files etc.
For example in Ubuntu:
Edit /etc/vsftpd.conf
And in the line
;write_enable=YES
Delete the semicolon
Finally restart the service:
sudo service vsftpd restart
I would check if you are in the right location. I got the same problem, and then I realised that I was in a different location that I intended, in the root folder, above "/public_html", so there was no folder that I wanted to enter, and I didn't have permissions to store any files.
You can check where you are this way:
print connect.pwd()
and what the contents of the current directory are:
print ftplib.FTP.dir(connect)
So, if you are in the root folder ("/"), above the "/public_html" and you want to change current directory to "/testfolder" you need to use:
connect.cwd('/public_html/testfolder')
Have you checked the access permission on the FTP server? I just ran into this same problem. This issue happened cause I did not have the permission to read the folder that I want to upload my files into.
There are a few things one can check if encountered with this error.
Check the current directory of the ftp server that you are trying to access using connect.pwd(). Make sure you have the write access to this directory. You can try copy pasting manually to verify the same.
Make sure you only provide the filename and not the complete path. For me, this was causing an issue. For example, filename = "upload_img.jpg" instead of filename = "D:/apth/to/upload_img.jpg". Workaround will be to extract the CWD using os.split() and then set the CWD using the os.chdir()
I have a python cgi script that runs an application via subprocess over and over again (several thousand times). I keep getting the same error...
Traceback (most recent call last):
File "/home/linuser/Webpages/cgi/SnpEdit.py", line 413, in <module>
webpage()
File "/home/linuser/Webpages/cgi/SnpEdit.py", line 406, in main
displayOmpResult(form['odfFile'].value)
File "/home/linuser/Webpages/cgi/SnpEdit.py", line 342, in displayContainerDiv
makeSection(position,sAoiInput)
File "/home/linuser/Webpages/cgi/SnpEdit.py", line 360, in displayData
displayTable(i,j,lAmpAndVars,dOligoSet[key],position)
File "/home/linuser/Webpages/cgi/SnpEdit.py", line 247, in displayTable
p = subprocess.Popen(['/usr/bin/pDat',sInputFileLoc,sOutputFileLoc],stdout=fh, stderr=fh)
File "/usr/lib/python2.6/subprocess.py", line 633, in __init__
errread, errwrite)
File "/usr/lib/python2.6/subprocess.py", line 1039, in _execute_child
errpipe_read, errpipe_write = os.pipe()
OSError: [Errno 24] Too many open files
The function causing it is below.
def displayTable(sData):
# convert the data to the proper format
sFormattedData = convertToFormat(sData)
# write the formatted data to file
sInputFile = tempfile.mkstemp(prefix='In_')[1]
fOpen = open(sInputFile,'w')
fOpen.write(sFormattedData)
fOpen.close()
sOutputFileLoc = sInputFile.replace('In_','Out_')
# run app, requires two files; an input and an output
# temp file to holds stdout stderr of subprocess
fh = tempfile.TemporaryFile(mode='w',dir=tempfile.gettempdir())
p = subprocess.Popen(['/usr/bin/pDat',sInputFileLoc,sOutputFileLoc],stdout=fh, stderr=fh)
p.communicate()
fh.close()
# open output file and print parsed data into a list of dictionaries
sOutput = open(sOutputFileLoc).read()
lOutputData = parseOutput(sOutput)
displayTableHeader(lOutputData)
displaySimpleTable(lOutputData)
As far as I can tell, I'm closing the files properly. When I run...
import resource
print resource.getrlimit(resource.RLIMIT_NOFILE)
I get...
(1024, 1024)
Do I have to increase this value? I read that subprocess opens several file descriptors. I tried adding "close_fds = True" and I tried using the with statement when creating my file but the result was the same. I suspect the problem may be with the application that I'm subprocessing, pDat, but this program was made by someone else. It requires two inputs; an input file and the location of where you want the output file written to. I suspect it may not be closing the output file that it creates. Aside from this, I can't see what I might be doing wrong. Any suggestions? Thanks.
EDIT:
I'm on ubuntu 10.04 running python 2.6.5 and apache 2.2.14
Instead of this...
sInputFile = tempfile.mkstemp(prefix='In_')[1]
fOpen = open(sInputFile,'w')
fOpen.write(sFormattedData)
fOpen.close()
I should have done this...
iFileHandle,sInputFile = tempfile.mkstemp(prefix='In_')
fOpen = open(sInputFile,'w')
fOpen.write(sFormattedData)
fOpen.close()
os.close(iFileHandle)
The mkstemp function makes OS level handles to a file and I wasn't closing them. The solution is described in more detail here...
http://www.logilab.org/blogentry/17873
You want to add close_fds=True to the popen call (just in case).
Then, here:
# open output file and print parsed data into a list of dictionaries
sOutput = open(sOutputFileLoc).read()
lOutputData = parseOutput(sOutput)
...I might remember wrong, but unless you use the with syntax, I do not think that the output file descriptor has been closed.
UPDATE: the main problem is that you need to know which files are open. On Windows this would require something like Process Explorer. In Linux it's a bit simpler; you just have to invoke the CGI from command line, or be sure that there is only one instance of the CGI running, and fetch its pid with ps command.
Once you have the pid, run a ls -la on the content of the /proc/<PID>/fd directory. All open file descriptors will be there, with the name of the files they point to. Knowing that file so-and-so is opened 377 times, that goes a long way towards finding out where exactly that file is opened (but not closed).