We Keep Coding

Python 3.10: FileNotFoundError - Existing Path With Unicode Characters

Problem statement:
While automatically copying files between input directories, and output directories my program fails on a path that contains unicode (most likely Korean) characters.
The whole script is publicly available under: This Link
The file that causes the error is also publicly available: File That Causes the Error
The specific part of the code that fails seems to be:
for root, _, filenames in os.walk(maybe_dir):
for file in filenames:
# Prepare relative paths:
relative_dir = os.path.relpath(root, maybe_dir)
relative_file = os.path.join(relative_dir, file)
# Get unique filename:
unique_filename = uuid.uuid4().hex
unique_filename_with_ext = unique_filename + file_extension
new_path_and_filename = os.path.join(
full_output_path, unique_filename_with_ext
)
current_file = os.path.abspath(os.path.join(root, file))
# Copying files:
shutil.copy(current_file, new_path_and_filename)
The error:
Traceback (most recent call last):
File "F:\Projects\SC2DatasetPreparator\src\directory_flattener.py", line 96, in <module>
directory_flattener(
File "F:\Projects\SC2DatasetPreparator\src\directory_flattener.py", line 60, in directory_flattener
shutil.copy(current_file, new_path_and_filename)
File "D:\Programs\Python3_10\lib\shutil.py", line 417, in copy
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "D:\Programs\Python3_10\lib\shutil.py", line 254, in copyfile
with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: 'F:\\Projects\\SC2DatasetPreparator\\processing\\directory_flattener\\input\\2017_IEM_XI_World_Championship_Katowice\\IEM XI - World Championship - StarCraft II Replays\\RO24\\Group A\\Solar Vs herO\\Ùë¦ý+ñÝü¼ ý×¼Û¦£Ù¦£ ýºÇÛÁ¼ - ÝåáÙäêÙ¿+Ýè© (Û¦ÁÝùêýØÿ ý£áýé¦) (2) Solar vs Hero game 1.SC2Replay'
The error itself is unexpected as I am using absolute paths and the script works for 6 other directories before it fails on that specific file.
The path clearly exists and can be accessed manually:
Steps to attempt to reproduce the error are as follows:
Clone the repository: Branch 1.1.0_testing
Place the File That Causes the Error in ./processing/directory_flattener/input/test_dir
Run the script
Closing Remarks:
It seems that the script worked before on Python 3.7 because I have verified the output that I have received before updating to Python 3.10 and within the directory mapping that is created the files with unicode characters in their path are present:
{
"ce2f4610891e472190a0852c617b35e8": "RO24\\Group A\\Solar Vs herO\\\u00d9\u00eb\u00a6\u00fd+\u00f1\u00dd\u00fc\u00bc \u00fd\u00d7\u00bc\u00db\u00a6\u00a3\u00d9\u00a6\u00a3 \u00fd\u00ba\u00c7\u00db\u00c1\u00bc - \u00dd\u00e5\u00e1\u00d9\u00e4\u00ea\u00d9\u00bf+\u00dd\u00e8\u00a9 (\u00db\u00a6\u00c1\u00dd\u00f9\u00ea\u00fd\u00d8\u00ff \u00fd\u00a3\u00e1\u00fd\u00e9\u00a6) (2) Solar vs Hero game 1.SC2Replay",
"dcc82d633910479c95d06ef418fcf2e0": "RO24\\Group A\\Solar Vs herO\\\u00fd\u00fb\u00a6\u00d9\u00a6\u00e4\u00fd\u00e4\u00f1 \u00d9\u00aa\u00bc\u00dd\u00f6\u00e4 - \u00d9\u00d7\u00ff\u00d9\u00ec\u00f6 Solar vs Hero game 2.SC2Replay",
}
While searching for an answer I have only stumbled upon similar problems in Python 2 where the .decode() method was suggested as a solution. Applying such measures did not help to solve the issue.

The cause for this error was the Maximum Path Length Limitation which limited the ability to use paths longer than 260 characters on Windows.
The error was fixed by adding a prefix of "\\?\" to the path that was used to access and copy the file.
This means that the following line of code was changed:
current_file = os.path.abspath(os.path.join(root, file))
Into this:
current_file = "\\\\?\\" + os.path.abspath(os.path.join(root, file))

No such file or directory error in python while looping through several files

We are using a function to match a file pattern and if it matches we are running some jobs and eventually removing the file form the folder. But the file pattern match code is failing with "FileNotFoundError: [Errno 2] No such file or directory" when the for loop is running and some of the files are removed, this is happening in so less time that we are not able to replicate the issue in lower environment. Any idea how to catch the error or resolve it. In this case 'ABCD.XAF.XYJ250.A252.dat' file was present in the folder but removed as part of some other job.
def poke(self, context):
full_path = self.filepath
file_pattern = re.compile(self.filepattern)
os.chdir(full_path)
for files in sorted(os.listdir(full_path),key=os.path.getmtime):
if not re.match(file_pattern, files):
self.log.info(files)
self.log.info(file_pattern)
else:
print("File Sensed "+files)
return True
return False
Error:
File "File_Sensor_Operator.py", line 25, in poke
for files in sorted(os.listdir(full_path),key=os.path.getmtime):
File "/usr/lib64/python3.6/genericpath.py", line 55, in getmtime
return os.stat(filename).st_mtime
FileNotFoundError: [Errno 2] No such file or directory: 'ABCD.XAF.XYJ250.A252.dat'

IIUC, the error is happening because after the for loop starts, you have removed one of the files which means that it is no longer in the iterable returned bysorted(os.listdir(full_path),key=os.path.getmtime) which then throws the error.
So I think it should go away if you change that section of your code to this:
all_files = sorted(os.listdir(full_path),key=os.path.getmtime)
for files in all_files:
if not re.match(file_pattern, files):
That way the iterable is assigned to a variable so the state is preserved as you loop through it.

Your issue is that you collect all the files, delete one, then try to sort them by each files attributes (whereof one doesn't exist anymore)
import os
with open("dat.dat", "w+"):
pass
file_list = os.listdir(os.path.dirname(__file__))
os.remove("dat.dat")
for file_name in sorted(file_list, key=os.path.getmtime):
print(file_name)
You can fix it by assigning the filenames and attributes to a variable before sorting.
import os
with open("dat.dat", "w+"):
pass
file_list = {
filename:os.path.getmtime(filename)
for filename in os.listdir(os.path.dirname(__file__))
if os.path.exists(filename)
}
os.remove("dat.dat")
for file_name in sorted(file_list.keys(), key=file_list.get):
print(file_name)

Transfering files from one directory to another via python

I'm trying to move multiple files from one directory to another.
The function I'm trying to make should move files if they begin with a value in sample_list. My issue is that there are multiple files which begins with a value in sample_list, this seems to be causing issues for shutil.
import shutil
import os
source = './train/'
dest1 = './test/'
files = [
'195_reg_6762_1540.npz',
'1369_reg_7652_-2532.npz',
'195_reg_1947_-484.npz',
'1336_reg_6209_1217.npz',
'1198_reg_3784_-934.npz',
'12_reg_3992_-10.npz',
'1369_reg_3214_-91.npz']
test_samples = [195, 1493, 409, 339, 12, 1336]
#Move files which begin with values in test_samples [edited orig post to fix typo]
for f in files:
for i in test_samples:
if (f.startswith(str(i))):
shutil.move(source+f, dest1)
Raises the error:
File "/home/usr/anaconda3/lib/python3.7/shutil.py", line 564, in move
raise Error("Destination path '%s' already exists" % real_dst)
Error: Destination path '/home/usr/Documents/project/data/test/195_reg_1947_-484.npz' already exists
It always fails if there is more than one file with a value in test samples to move.
What would be the correct way to move files which begin with values in test_samples from source directory to target directory.

This occurs when you've already successfully run the script (or parts of it once). Once the file exists in the destination directory, shutil throws an exception if you try to replace it.
See this modification that checks if the file already exists in the destination directory and, if so, deletes it and moves it back from the origin directory (note that the file must exist again in the origin directory). [Do you mean to copy instead of move?]
for f in files:
for i in test_samples:
if (f.startswith(str(i))):
if not os.path.exists(dest1+f):
shutil.move(source+f, dest1)
else:
os.remove(dest1+f)
shutil.move(source+f,dest1)

CSV file creation error in python

I am getting some error while writing contents to csv file in python
import sys
reload(sys)
sys.setdefaultencoding('utf8')
import csv
a = [['1/1/2013', '1/7/2013'], ['1/8/2013', '1/14/2013'], ['1/15/2013', '1/21/2013'], ['1/22/2013', '1/28/2013'], ['1/29/2013', '1/31/2013']]
f3 = open('test_'+str(a[0][0])+'_.csv', 'at')
writer = csv.writer(f3,delimiter = ',', lineterminator='\n',quoting=csv.QUOTE_ALL)
writer.writerow(a)
Error
Traceback (most recent call last):
File "test.py", line 10, in <module>
f3 = open('test_'+str(a[0][0])+'_.csv', 'at')
IOError: [Errno 2] No such file or directory: 'test_1/1/2013_.csv'
How to fix it and what is the error?

You have error message - just read it.
The file test_1/1/2013_.csv doesn't exist.
In the file name that you create - you use a[0][0] and in this case it result in 1/1/2013.
Probably this two signs '/' makes that you are looking for this file in bad directory.
Check where are this file (current directory - or in .test_1/1 directory.

It's probably due to the directory not existing - Python will create the file for you if it doesn't exist already, but it won't automatically create directories.
To ensure the path to a file exists you can combine os.makedirs and os.path.dirname.
file_name = 'test_'+str(a[0][0])+'_.csv'
# Get the directory the file resides in
directory = os.path.dirname(file_name)
# Create the directories
os.makedirs(directory)
# Open the file
f3 = open(file_name, 'at')
If the directories aren't desired you should replace the slashes in the dates with something else, perhaps a dash (-) instead.
file_name = 'test_' + str(a[0][0]).replace('/', '-') + '_.csv'

IOError: [Errno 13] Permission denied: Bulk renaming script

This is one of my first python scripts designed to go into a directory and rename all the files of a certain extension to the same thing. For instance, go into a music directory and change the name of all the album artwork files to something like folder.jpg so that they're easily recognized and displayed by music playing software like foobar2000.
I'm currently getting the error in the title. I've tried:
os.chmod(pathname, 0777)
and
os.chmod(pathname, stat.S_IWOTH)
to allow other entities to edit the directory I'm looking at, but the error remains. I'm not too familiar with Windows's permissions system or with Python so this is mystifying me. Any idea where I'm going wrong?
#Changes file names in every subdirectory of the target directory
import os
import sys
import shutil
#Get target path
pathname = raw_input('Enter path for music directory (ex. C:\\Music): ')
try:
os.chdir(pathname)
except:
print('Failed. Invalid directory?')
sys.exit()
#Get variables for renaming
fn = raw_input('Enter desired file name for all converted files: ')
ft = raw_input('Enter the file extension you want the program to look for (ex. .jpg): ')
outname = fn + ft
#Create path tree
for path, subdirs, files in os.walk (pathname):
#Search tree for files with defined extention
for name in files:
if name.lower().endswith(ft):
#Rename the files
src = os.path.join(path, name)
print (src)
dst = os.path.join(path, outname)
print dst
shutil.move(src, dst)
print('Complete')
A second more benign issue is that when I use a print statement to check on what files are being edited, it seems that before the first .jpg is attempted, the program processes and renamed a file called d:\test\folder.jpg which doesn't exist. This seems to succeed and the program fails in the last loop the second time through, when an existing file is processed. The program runs like this:
>>> ================================ RESTART ================================
>>>
Enter path for music directory (ex. C:\Music): d:\test
Enter desired file name for all converted files: folder
Enter the file extension you want the program to look for (ex. .jpg): .jpg
d:\test\folder.jpg
d:\test\folder.jpg
d:\test\Anathema\Alternative 4\AA.jpg
d:\test\Anathema\Alternative 4\folder.jpg
Traceback (most recent call last):
File "D:\Documents\Programming\Python scripts\Actually useful\bulk_filename_changer.py", line 29, in <module>
shutil.move(src, dst)
File "C:\Python27\lib\shutil.py", line 301, in move
copy2(src, real_dst)
File "C:\Python27\lib\shutil.py", line 130, in copy2
copyfile(src, dst)
File "C:\Python27\lib\shutil.py", line 83, in copyfile
with open(dst, 'wb') as fdst:
IOError: [Errno 13] Permission denied: 'd:\\test\\Anathema\\Alternative 4\\folder.jpg'
>>>
The glitch seems to be benign because it doesn't cause an error, but it might be related to an error I have in the code that leads to the IOError I get when the first "real" file is processed. Beats me...

I am assuming that d: is not a CD drive or something not writable. If so then have you checked the properties of your files? Are any of them read only? There would also need to be more than one of the file type in the folder to cause this, I believe.