I know this question is asked and answered many places on this site but none of the resolutions work for me. Basically, I need shutil.copy to work the same way as shutil.move works (and it does) but when I do .copy I get the Error 13 Permissions error on the source file.
I can't do shutil.copytree because I need not the content of the first file folder to be copied but the file folder itself (with the contents inside). I've tried checking the permissions, modifying, nothing works.
I've also tried using the full path including the folder names (no variables) and I still get the same error.
This is in Windows 7 and 8.1
Here is the code:
import os
import csv
import re
from os import path
from distutils.dir_util import copy_tree
import shutil
# Open the csv file
f = open("Consumers.csv")
csv_f = csv.reader(f)
#Loop throught the csv file
for eachrow in csv_f:
#loop through the folders in the directory
for foldname in os.listdir():
#If the folder name is the same as a field in the first column of the csv file
if (foldname == eachrow[0]):
#Name the second column filed name "bucket."
#This is also the name of a folder in the same directory
bucket = eachrow[1]
#copy the first folder (and contents) into the second folder
shutil.copy (foldname, bucket)
And the error:
PermissionError Traceback (most recent call last)
<ipython-input-36-61e009418603> in <module>
25
26 #copy the first folder (and contents) into the second folder
---> 27 shutil.copy (foldname, bucket)
28
29
~\Anaconda3\lib\shutil.py in copy(src, dst, follow_symlinks)
243 if os.path.isdir(dst):
244 dst = os.path.join(dst, os.path.basename(src))
--> 245 copyfile(src, dst, follow_symlinks=follow_symlinks)
246 copymode(src, dst, follow_symlinks=follow_symlinks)
247 return dst
~\Anaconda3\lib\shutil.py in copyfile(src, dst, follow_symlinks)
118 os.symlink(os.readlink(src), dst)
119 else:
--> 120 with open(src, 'rb') as fsrc:
121 with open(dst, 'wb') as fdst:
122 copyfileobj(fsrc, fdst)
PermissionError: [Errno 13] Permission denied: 'Last1_First1_11111'
Any help would be greatly appreciated!
I've done something similar once. So maybe this helps, if I understand you correctly.
You can modify the behaviour of copytree via the ignore= option. ignore= takes a callable that in turn takes the currently (i.e. during the recursion) visited path and a list of that path's content as arguments. It has to return an iterable that copytree then ignores.
I've used the following imports (you can adjust that):
import shutil
import os
Take this as ignore function:
def ignore(path, content_list):
return [
content
for content in content_list
if os.path.isdir(os.path.join(path, content))
]
It packs all subdirectories of the visited path in the ignore list. The effect is that copytree stops its recursion after the first directory and the files in it!
Instead of
shutil.copy (foldname, bucket)
use
shutil.copytree(foldname, bucket, ignore=ignore)
It worked as designed for me. But I'm a bit unsure how your path structure exactly looks like. It would be good if foldname and bucket were absolute paths.
To ensure that I would add the following to the imports (the Path class makes working with paths extremely easy):
from pathlib import Path
Then supplement the loop through the folder with
#loop through the folders in the directory
for foldname in os.listdir():
fold_path = Path(foldname).resolve()
and use (instead of the version above)
shutil.copytree(fold_path, (fold_path.parent / bucket),
ignore=ignore,
dirs_exist_ok=True)
That's my understanding of what you are trying to achieve. (It's always good to illustrate questions with small examples to ensure that your intend is really clear.)
EDIT: Complete program (without comments and imports that aren't needed in this part):
import csv
import os
import shutil
from pathlib import Path
def ignore(path, content_list):
return [
content
for content in content_list
if os.path.isdir(os.path.join(path, content))
]
f = open("Consumers.csv")
csv_f = csv.reader(f)
for eachrow in csv_f:
for foldname in os.listdir():
fold_path = Path(foldname).resolve()
if foldname == eachrow[0]:
bucket = eachrow[1]
shutil.copytree(fold_path,
(fold_path.parent / bucket),
ignore=ignore,
dirs_exist_ok=True)
Related
I see many answers on here to similar questions but I cannot seem to adapt it quite yet due to my budding Python skill. I'd like to save the time of individually grabbing the data sets that contain what I need for analysis in R, but my scripts either don't run or seem to do what I need.
I need to 1) loop through a sea of subfolders in a parent folder, 2) loop through the bagillion .csv files in those subs and pick out the 1 that matters (matching text below) and 3) copy it over to a new clean folder with only what I want.
What I have tried:
1)
import os, shutil, glob
src_fldr = 'C:/project_folder_withsubfolders';
dst_fldr = 'C:/project_folder_withsubfolders/subfolder_to_dump_into';
try:
os.makedirs(dst_fldr); ## it creates the destination folder
except:
print ("Folder already exist or some error");
for csv_file in glob.glob(src_fldr+'*statistics_Intensity_Sum_Ch=3_Img=1.csv*'):
shutil.copy2(csv_file, dst_fldr);
where the text statistics_Intensity_Sum etc is the exact pattern I need for the file to copy over
this didn't actually copy anything over
Making a function that will do this:
srcDir = 'C:/project_folder_withsubfolders'
dstDir = 'C:/project_folder_withsubfolders/subfolder_to_dump_into'
def moveAllFilesinDir(srcDir, dstDir):
files = os.listdir(srcDir)
for f in files:
if f.find("statistics_Intensity_Sum_Ch=3_Img=1"):
shutil.move(f, dstDir)
else:
shutil.move(f, srcDir)
moveAlllFilesinDir(srcDir, dstDir)
This returned the following error:
File "C:\Users\jbla12\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 806, in move
os.rename(src, real_dst)
FileNotFoundError: [WinError 2] The system cannot find the file specified: 'F1 converted' -> 'C:/Users/jbla12/Desktop/R Analyses/p65_project/sum_files\\F1 converted'
That's because that's a sub-folder I want it to go through! I've tried other methods but don't have record of them in my scripts.
SOLVED:
Special thanks to "Automate the Boring Stuff"
import shutil
import os
dest = 'C:/Users/jbla12/Desktop/R Analyses/p65_project/sum_files/'
src = 'C:/Users/jbla12/Desktop/R Analyses/p65_project/'
txt_ID = 'statistics_Intensity_Sum_Ch=3_Img=1.csv'
def moveSpecFiles(txt_ID, src, dest):
#src is the original file(s) destination
#dest is the destination for the files to end up in
#spec_txt is what the files end with that you want to ID
for foldername, subfolders, filenames in os.walk(src):
for file in filenames:
if file.endswith(txt_ID):
shutil.copy(os.path.join(foldername, file), dest)
print('Your files are ready sir/madam!')
moveSpecFiles(txt_ID, src, dest)
I'm new to Python, and am trying to copy photos (.jpg) to a folder after renaming the images in numerical order. I have written the following script to rename all the photos in my directory (all 32,000 of them) to what they are (weeds):
import os
os.chdir('E:\\weeds')
i=1
for file in os.listdir():
src=file
dst="weed"+str(i)+".jpg"
os.rename(src,dst)
i+=1
Here's a sample of the output naming:
I am then trying to copy the first 250 of these photos into a new directory, as shown below:
import os, shutil
#Copying files to folders
original_weed = 'E:\\weeds'
train_weed = 'E:\\weeds_train'
#Training dataset: Weeds
fnames = ['weed{}.jpg'.format(i) for i in range(250)]
for fname in fnames:
src = os.path.join(original_weed, fname)
dst = os.path.join(train_weed, fname)
shutil.copyfile(src, dst)
The following error is produced:
FileNotFoundError Traceback (most recent call last)
<ipython-input-7-f08fff292dd8> in <module>
17 src = os.path.join(original_weed, fname)
18 dst = os.path.join(train_weed, fname)
---> 19 shutil.copyfile(src, dst)
~\anaconda3\lib\shutil.py in copyfile(src, dst, follow_symlinks)
259 os.symlink(os.readlink(src), dst)
260 else:
--> 261 with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
262 # macOS
263 if _HAS_FCOPYFILE:
FileNotFoundError: [Errno 2] No such file or directory: 'E:\\weed0.jpg'
What do I need to do so that I can get the file to be identified for copying?
There is an error in your code - the first part of the script names pictures with index starting from 1:
os.chdir('E:\\weeds')
i=1
for file in os.listdir():
While the next part of the code iterates through the result of the range function, which starts at 0 and ends in 249 in your case. Check the docs for the range function for further info.
The error which you got indicates that there is no file named weed0.jpg in the directory, which is indeed true.
The easiest fix you can do is to set the initial value of i to 0 in the first part of the code, and it should work properly.
I want to copy files that have specific date. I can filter out the date. Copying makes problems.
import os
from os import walk
import time
from datetime import date, timedelta
import zipfile
import io
import shutil
src = 'K:\\Userfiles'
dest = 'L:\\Userfiles'
date1 = date.today() - timedelta(2)
for root, dirs, files in os.walk(src):
for file in files:
if ( 'zip' in file):
x = file[-18:-8]
d = date1.strftime('%Y-%m-%d')
if x == d:
shutil.copyfile(file, dest)
ERROR is: FileNotFoundError: [Errno 2] No such file or directory.
Traceback (most recent call last):
File "C:/Python37/datetime_finder.py", line 28, in shutil.copyfile(file, 'K:\Userfiles\Ucar\UNZIP')
File "C:\Python37\lib\shutil.py", line 120, in copyfile with open(src, 'rb') as fsrc: FileNotFoundError: [Errno 2] No such file or directory: 'getContents_2019-01-27.csv.zip
Taken from https://docs.python.org/3/library/shutil.html#shutil.copyfile
shutil.copyfile(src, dst, *, follow_symlinks=True)
Copy the contents (no metadata) of the file named src to a file named dst and return dst. src and dst are path names given as strings. dst must be the complete target file name; look at shutil.copy() for a copy that accepts a target directory path.
If I am not mistaken, you are missing setting dest value inside your inner for loop, so shutil.copyfile fails as '' (empty string) does not make sense as second argument. As side note if you want to copy only .zip files it is better to use:
if file.endswith('zip'):
instead of
if ('zip' in file):
which is also True for example for 'my_list_of_zip_files.txt', also keep in mind case-sensitiveness, so it might be better to use following if
if file.lower().endswith('zip'):
You are getting the error in this line shutil.copyfile(file, dest)
Mentioning the full path should fix the problem.
Ex:
shutil.copyfile(os.path.join(root, file), ".")
I am trying to open the file from folder and read it but it's not locating it. I am using Python3
Here is my code:
import os
import glob
prefix_path = "C:/Users/mpotd/Documents/GitHub/Python-Sample-
codes/Mayur_Python_code/Question/wx_data/"
target_path = open('MissingPrcpData.txt', 'w')
file_array = [os.path.abspath(f) for f in os.listdir(prefix_path) if
f.endswith('.txt')]
file_array.sort() # file is sorted list
for f_obj in range(len(file_array)):
file = os.path.abspath(file_array[f_obj])
join_file = os.path.join(prefix_path, file) #whole file path
for filename in file_array:
log = open(filename, 'r')#<---- Error is here
Error: FileNotFoundError: [Errno 2] No such file or directory: 'USC00110072.txt'
You are not giving the full path to a file to the open(), just its name - a relative path.
Non-absolute paths specify locations in relation to current working directory (CWD, see os.getcwd).
You would have to either os.path.join() correct directory path to it, or os.chdir() to the directory that the files reside in.
Also, remember that os.path.abspath() can't deduce the full path to a file just by it's name. It will only prefix its input with the path of the current working directory, if the given path is relative.
Looks like you are forgetting to modify the the file_array list. To fix this, change the first loop to this:
file_array = [os.path.join(prefix_path, name) for name in file_array]
Let me reiterate.
This line in your code:
file_array = [os.path.abspath(f) for f in os.listdir(prefix_path) if f.endswith('.txt')]
is wrong. It will not give you a list with correct absolute paths. What you should've done is:
import os
import glob
prefix_path = ("C:/Users/mpotd/Documents/GitHub/Python-Sample-"
"codes/Mayur_Python_code/Question/wx_data/")
target_path = open('MissingPrcpData.txt', 'w')
file_array = [f for f in os.listdir(prefix_path) if f.endswith('.txt')]
file_array.sort() # file is sorted list
file_array = [os.path.join(prefix_path, name) for name in file_array]
for filename in file_array:
log = open(filename, 'r')
You are using relative path where you should be using an absolute one. It's a good idea to use os.path to work with file paths. Easy fix for your code is:
prefix = os.path.abspath(prefix_path)
file_list = [os.path.join(prefix, f) for f in os.listdir(prefix) if f.endswith('.txt')]
Note that there are some other issues with your code:
In python you can do for thing in things. You did for thing in range(len(things)) it's much less readable and unnecessary.
You should use context managers when you open a file. Read more here.
I have a functionality in my Python which backups a particular directory everyday. The backup steps include compressing the directory. I am using shutil.make_archive to compress the directory.
But now I am facing a permission issue when the directory I compress contains files which I do not have access to.
In this case, I just want to skip that file and then go ahead with compression of the remaining files. How do I achieve this ?
I searched SO and came across this answer, which shows how to zip a directory from using zipfile library. In this case, I can just check if a particular file throws an exception and ignore it.
Is this the only way or is there an easier way ?
My code (just uses shutil.make_archive):
shutil.make_archive(filename, "zip", foldername)
The code in the answer which I have modified for my use:
def zipdir(path, ziph):
# ziph is zipfile handle
for root, dirs, files in os.walk(path):
for file in files:
try:
ziph.write(os.path.join(root, file))
except PermissionError as e:
continue
if __name__ == '__main__':
zipf = zipfile.ZipFile('Python.zip', 'w', zipfile.ZIP_DEFLATED)
zipdir('tmp/', zipf)
zipf.close()
Please tell me if there is an easier/efficient solution to my issue.
I know this is ancient history, but you can use copytree and pass in a custom copy function that suppresses the permissions errors:
import os
#Adapted from python shutil source code.
def copyIgnorePermissionError(src, dst, *, follow_symlinks=True):
"""Copy data and metadata. Return the file's destination.
Metadata is copied with copystat(). Please see the copystat function
for more information.
The destination may be a directory.
If follow_symlinks is false, symlinks won't be followed. This
resembles GNU's "cp -P src dst".
"""
if os.path.isdir(dst):
dst = os.path.join(dst, os.path.basename(src))
#try the copy. If it gives us any lip about permission errors,
#we're going to just suppress them here. If for some reason
#a non-permission error happens, then raise it.
try:
copyfile(src, dst, follow_symlinks=follow_symlinks)
copystat(src, dst, follow_symlinks=follow_symlinks)
except PermissionError:
pass
except Exception as e:
raise(e)
finally:
return dst
#copy the source to some safe destination
shutil.copytree(aSource, aCopyDestination, copy_function=copyIgnorePermissionError)
That will make a copy of all the files and directories that you can access at your permission level. Then you can use shutil.make_archive to create the archive of this copy, then delete the copy.