We Keep Coding

Python 3.10: FileNotFoundError - Existing Path With Unicode Characters

Problem statement:
While automatically copying files between input directories, and output directories my program fails on a path that contains unicode (most likely Korean) characters.
The whole script is publicly available under: This Link
The file that causes the error is also publicly available: File That Causes the Error
The specific part of the code that fails seems to be:
for root, _, filenames in os.walk(maybe_dir):
for file in filenames:
# Prepare relative paths:
relative_dir = os.path.relpath(root, maybe_dir)
relative_file = os.path.join(relative_dir, file)
# Get unique filename:
unique_filename = uuid.uuid4().hex
unique_filename_with_ext = unique_filename + file_extension
new_path_and_filename = os.path.join(
full_output_path, unique_filename_with_ext
)
current_file = os.path.abspath(os.path.join(root, file))
# Copying files:
shutil.copy(current_file, new_path_and_filename)
The error:
Traceback (most recent call last):
File "F:\Projects\SC2DatasetPreparator\src\directory_flattener.py", line 96, in <module>
directory_flattener(
File "F:\Projects\SC2DatasetPreparator\src\directory_flattener.py", line 60, in directory_flattener
shutil.copy(current_file, new_path_and_filename)
File "D:\Programs\Python3_10\lib\shutil.py", line 417, in copy
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "D:\Programs\Python3_10\lib\shutil.py", line 254, in copyfile
with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: 'F:\\Projects\\SC2DatasetPreparator\\processing\\directory_flattener\\input\\2017_IEM_XI_World_Championship_Katowice\\IEM XI - World Championship - StarCraft II Replays\\RO24\\Group A\\Solar Vs herO\\Ùë¦ý+ñÝü¼ ý×¼Û¦£Ù¦£ ýºÇÛÁ¼ - ÝåáÙäêÙ¿+Ýè© (Û¦ÁÝùêýØÿ ý£áýé¦) (2) Solar vs Hero game 1.SC2Replay'
The error itself is unexpected as I am using absolute paths and the script works for 6 other directories before it fails on that specific file.
The path clearly exists and can be accessed manually:
Steps to attempt to reproduce the error are as follows:
Clone the repository: Branch 1.1.0_testing
Place the File That Causes the Error in ./processing/directory_flattener/input/test_dir
Run the script
Closing Remarks:
It seems that the script worked before on Python 3.7 because I have verified the output that I have received before updating to Python 3.10 and within the directory mapping that is created the files with unicode characters in their path are present:
{
"ce2f4610891e472190a0852c617b35e8": "RO24\\Group A\\Solar Vs herO\\\u00d9\u00eb\u00a6\u00fd+\u00f1\u00dd\u00fc\u00bc \u00fd\u00d7\u00bc\u00db\u00a6\u00a3\u00d9\u00a6\u00a3 \u00fd\u00ba\u00c7\u00db\u00c1\u00bc - \u00dd\u00e5\u00e1\u00d9\u00e4\u00ea\u00d9\u00bf+\u00dd\u00e8\u00a9 (\u00db\u00a6\u00c1\u00dd\u00f9\u00ea\u00fd\u00d8\u00ff \u00fd\u00a3\u00e1\u00fd\u00e9\u00a6) (2) Solar vs Hero game 1.SC2Replay",
"dcc82d633910479c95d06ef418fcf2e0": "RO24\\Group A\\Solar Vs herO\\\u00fd\u00fb\u00a6\u00d9\u00a6\u00e4\u00fd\u00e4\u00f1 \u00d9\u00aa\u00bc\u00dd\u00f6\u00e4 - \u00d9\u00d7\u00ff\u00d9\u00ec\u00f6 Solar vs Hero game 2.SC2Replay",
}
While searching for an answer I have only stumbled upon similar problems in Python 2 where the .decode() method was suggested as a solution. Applying such measures did not help to solve the issue.

The cause for this error was the Maximum Path Length Limitation which limited the ability to use paths longer than 260 characters on Windows.
The error was fixed by adding a prefix of "\\?\" to the path that was used to access and copy the file.
This means that the following line of code was changed:
current_file = os.path.abspath(os.path.join(root, file))
Into this:
current_file = "\\\\?\\" + os.path.abspath(os.path.join(root, file))

Python - set path to the directory in project

I created a project where my main.py script is in root folder Project. I have a utils directory inside and I want to get the path to this directory. I made a function which saves a .pdf file in my utils directory:
with open(os.path.abspath('utils/' + self.object.object_name+ '.pdf'), 'wb') as f:
f.write(pdf)
but I get an error:
IOError: [Errno 2] No such file or directory: '/home/documents/office/projects/me/utils/d.pdf'
How can I change os.path.abspath to do it the right way?
I want to get into utils directory always.

What you need is defining the path. And you can keep it relative.
yourPath = './utils/'
yourFileName = self.object.object_name+ '.pdf'
yourFullFileName = yourPath+yourFileName
and finally
with open(yourFullFileName, 'wb') as f:
f.write(pdf)
UPDATE
Following the small course Scott Hunter gave me, (cf his comment) I make amends. It follows that the correct way to build your path is :
working_dir = os.path.dirname(__file__)# __file__ is the full name of your working script
yourFullFileName = os.path.join(working_dir, 'util', self.object.object_name + "." + 'pdf')
By doing so, there is no assumption about the (operating-system dependent) separator.

PermissionError: [Errno 13] Permission denied # PYTHON

i am trying to copy a file from one folder to another, but i am getting "PermissionError: [Errno 13] Permission denied". I am working within my home directory and i am the administrator of the PC. Went through many other previous posts .. tried all the options that are to my knowledge (newbie to programming) ... need some help.
import os
import shutil
src = "C:\\Users\\chzia\\Scripts\\test" # the file lab.txt is in this folder that needs to be copied to testcp folder.
dst = "C:\\Users\\chzia\\Scripts\\testcp"
for file in os.listdir(src):
src_file = os.path.join(src, file)
dst_file = os.path.join(dst, file)
#shutil.copymode(src, dst) # i have tried these options too same error
#shutil.copyfile(src, dst) # i have tried these options too same error
shutil.copy(src, dst)
My target is to create an .exe that copies a file from the network location to a specific folder on a pc where the .exe is run.
Thanks in advance for all the support and help.

Perhaps try to use shutil.copyfile instead:
shutil.copyfile(src, dst)
Similar old topic on Why would shutil.copy() raise a permission exception when cp doesn't?

I am sure I am late, but I ran into the same problem.
I noticed that in my case the problem is that the subfolder already exists.
If I delete the folder at the start (it is OK in my case).
import os
import shutil
dst = "C:\\Users\\chzia\\Scripts\\testcp" # target folder
def checkFolder(path):
try:
os.stat(path)
shutil.rmtree(path)
except:
os.mkdir(path)
checkFolder(dst)

If you Googled the exception and ended-up here, remember to provide the absolute/full path when using copy & copyfile from shutil. For example,
abs_src_path = os.path.abspath(relative_file_path)
abs_dst_path = os.path.abspath(relative_dst_path)
shutil.copy(abs_src_path , abs_dst_path)
In the question above that is already done, but you might be the one who is mislead by the error message.

Copying a file to an existing directory results in IOError [Error 21] is a directory

I get this error:
IOError [Error 21] is a directory
when I try to copy a file to an existing directory. I do it like this:
shutil.copyfile(src, dst)
where src is a file and dst is an existing directory. What I'm doing wrong?

You're using the wrong function. You might want "copy":
https://docs.python.org/2/library/shutil.html

You have already answered yourself in the question.
dst should be the path to the copied file. So if you want to copy the file to /var/lib/my/ and your file is called f1 then dst should be /var/lib/my/f1.txt
Try to use shutil.copy as suggested here by john-estess
shutil.copy(src, dst)
or try to fix this using the following snippet
shutil.copyfile(src, '%s/%s' % (dst, src.split('/')[-1]))
Assuming src is the path of the file you want to copy, such as /var/log/apache/access.log, and dst is the path to the directory, where you want to copy the file, for example, /var/lib/my then the new destination is /var/lib/my/access.log.

See code snippet to see how the source is the path to a file, AND the destination is also a path to a file (that doesn't yet exist until you run this function).
for file in orig_files:
shutil.copyfile(SOURCE_PATH + file_name, DEST_PATH + file_name)

Bug in Python Renaming Program.....No such file or Directory (Fnmatch)

I'm trying to build a little renaming program to help save me time in the future.
Basically it will go through directories I point it too and rename files if they meet certain criteria.
I have written what I need but I have a bug in the very beginning that I can't figure out.
Here is the code:
import os
import fnmatch
for file in os.listdir("""/Users/Desktop/TESTME"""):
if fnmatch.fnmatch(file,'MISC*'):
os.rename(file, file[4:12] + '-13-Misc.jpg')
When I try to run it I am getting this:
Traceback (most recent call last):
File "/Users/Documents/Try.py", line 6, in <module>
os.rename(file, file[4:12] + '-13-Misc.jpg')
OSError: [Errno 2] No such file or directory
I also tried this:
if fnmatch.fnmatch(file,'MISC*'):
fun = file[4:12] + '-13-Misc.jpg'
os.rename(file, fun)
But I get the same thing.
It's not recognizing the file as a file.
Am I going about this the wrong way?

You'll need to include the full path to the filenames you are trying to rename:
import os
import fnmatch
directory = "/Users/Desktop/TESTME"
for file in os.listdir(directory):
if fnmatch.fnmatch(file, 'MISC*'):
path = os.path.join(directory, file)
target = os.path.join(directory, file[4:12] + '-13-Misc.jpg'
os.rename(path, target)
The os.path.join function intelligently joins path elements into a whole, using the correct directory separator for your platform.

The function os.listdir() only returns the file names of the files in the given directory, not their full paths. You can use os.path.join(directory, file_name) to reconstruct the full path of the file.
You could also do this in bash:
cd /Users/Desktop/TESTME/
for f in MISC*; do mv "$f" "${f:4:8}-13-Misc.jpg"; done