We Keep Coding

Python 3.10: FileNotFoundError - Existing Path With Unicode Characters

Problem statement:
While automatically copying files between input directories, and output directories my program fails on a path that contains unicode (most likely Korean) characters.
The whole script is publicly available under: This Link
The file that causes the error is also publicly available: File That Causes the Error
The specific part of the code that fails seems to be:
for root, _, filenames in os.walk(maybe_dir):
for file in filenames:
# Prepare relative paths:
relative_dir = os.path.relpath(root, maybe_dir)
relative_file = os.path.join(relative_dir, file)
# Get unique filename:
unique_filename = uuid.uuid4().hex
unique_filename_with_ext = unique_filename + file_extension
new_path_and_filename = os.path.join(
full_output_path, unique_filename_with_ext
)
current_file = os.path.abspath(os.path.join(root, file))
# Copying files:
shutil.copy(current_file, new_path_and_filename)
The error:
Traceback (most recent call last):
File "F:\Projects\SC2DatasetPreparator\src\directory_flattener.py", line 96, in <module>
directory_flattener(
File "F:\Projects\SC2DatasetPreparator\src\directory_flattener.py", line 60, in directory_flattener
shutil.copy(current_file, new_path_and_filename)
File "D:\Programs\Python3_10\lib\shutil.py", line 417, in copy
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "D:\Programs\Python3_10\lib\shutil.py", line 254, in copyfile
with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: 'F:\\Projects\\SC2DatasetPreparator\\processing\\directory_flattener\\input\\2017_IEM_XI_World_Championship_Katowice\\IEM XI - World Championship - StarCraft II Replays\\RO24\\Group A\\Solar Vs herO\\Ùë¦ý+ñÝü¼ ý×¼Û¦£Ù¦£ ýºÇÛÁ¼ - ÝåáÙäêÙ¿+Ýè© (Û¦ÁÝùêýØÿ ý£áýé¦) (2) Solar vs Hero game 1.SC2Replay'
The error itself is unexpected as I am using absolute paths and the script works for 6 other directories before it fails on that specific file.
The path clearly exists and can be accessed manually:
Steps to attempt to reproduce the error are as follows:
Clone the repository: Branch 1.1.0_testing
Place the File That Causes the Error in ./processing/directory_flattener/input/test_dir
Run the script
Closing Remarks:
It seems that the script worked before on Python 3.7 because I have verified the output that I have received before updating to Python 3.10 and within the directory mapping that is created the files with unicode characters in their path are present:
{
"ce2f4610891e472190a0852c617b35e8": "RO24\\Group A\\Solar Vs herO\\\u00d9\u00eb\u00a6\u00fd+\u00f1\u00dd\u00fc\u00bc \u00fd\u00d7\u00bc\u00db\u00a6\u00a3\u00d9\u00a6\u00a3 \u00fd\u00ba\u00c7\u00db\u00c1\u00bc - \u00dd\u00e5\u00e1\u00d9\u00e4\u00ea\u00d9\u00bf+\u00dd\u00e8\u00a9 (\u00db\u00a6\u00c1\u00dd\u00f9\u00ea\u00fd\u00d8\u00ff \u00fd\u00a3\u00e1\u00fd\u00e9\u00a6) (2) Solar vs Hero game 1.SC2Replay",
"dcc82d633910479c95d06ef418fcf2e0": "RO24\\Group A\\Solar Vs herO\\\u00fd\u00fb\u00a6\u00d9\u00a6\u00e4\u00fd\u00e4\u00f1 \u00d9\u00aa\u00bc\u00dd\u00f6\u00e4 - \u00d9\u00d7\u00ff\u00d9\u00ec\u00f6 Solar vs Hero game 2.SC2Replay",
}
While searching for an answer I have only stumbled upon similar problems in Python 2 where the .decode() method was suggested as a solution. Applying such measures did not help to solve the issue.

The cause for this error was the Maximum Path Length Limitation which limited the ability to use paths longer than 260 characters on Windows.
The error was fixed by adding a prefix of "\\?\" to the path that was used to access and copy the file.
This means that the following line of code was changed:
current_file = os.path.abspath(os.path.join(root, file))
Into this:
current_file = "\\\\?\\" + os.path.abspath(os.path.join(root, file))

Compress photo and put to another directory with tinify and python

I'm slowly starting to study about python and wanted to write simple script which use tinify api, takes photos from unoptimalized directiory, compress it and put to optimalized directory. So far works partly, I mean weirdly I need to keep photos in main directory and unoptimalized one. If I dont have another copy in one of these directories, I have error. another thing is that after I launch script, only first photo is compressed and put inside optimalized directory, and then error appears.
So far I'm experimenting on two photos: lew.jpg and kot.jpg
My directory structure is like this:
Main root directory with script, and two other directories inside (optimalized and unoptimalized)
def optimalizeImages():
for fname in os.listdir('./unoptimalized'):
if fname.endswith(".jpg"):
print(fname)
source = tinify.from_file(fname)
print("processing", fname)
os.chdir("./optimalized")
source.to_file("optimalized-" + fname)
print("changed", fname)
optimalizeImages()
Error after processing first image:
Traceback (most recent call last):
File "python.py", line 20, in <module>
optimalizeImages()
File "python.py", line 11, in optimalizeImages
source = tinify.from_file(fname)
File "/home/grzymowiczma/.local/lib/python2.7/site-packages/tinify/__init__.py", line 79, in from_file
return Source.from_file(path)
File "/home/grzymowiczma/.local/lib/python2.7/site-packages/tinify/source.py", line 13, in from_file
with open(path, 'rb') as f:
IOError: [Errno 2] No such file or directory: 'kot.jpg'
and if i keep photos only in root directory, no error but also no any action, if i keep them only in unoptimalized i get same error:
Traceback (most recent call last):
File "python.py", line 20, in <module>
optimalizeImages()
File "python.py", line 11, in optimalizeImages
source = tinify.from_file(fname)
File "/home/grzymowiczma/.local/lib/python2.7/site-packages/tinify/__init__.py", line 79, in from_file
return Source.from_file(path)
File "/home/grzymowiczma/.local/lib/python2.7/site-packages/tinify/source.py", line 13, in from_file
with open(path, 'rb') as f:
IOError: [Errno 2] No such file or directory: 'lew.jpg'

There are two issues here, both because you are using relative filenames. os.listdir() gives you those relative filenames, without a path. Instead of ./unoptimalized/kot.jpg, you get just kot.jpg.
So when you try to load the image:
source = tinify.from_file(fname)
you tell the library to load image.jpg without any context other than the current working directory. And that's not the right directory, not if os.listdir('./unoptimalized') worked to list all image files; that indicates that the current working directory is in the parent directory of both unoptimalized and optimalized.
You solved that by putting an image file with the same name in the parent directory, but that's not quite the right way to solve this. More on this below.
The next issue is that you change the working directory:
os.chdir("./optimalized")
You do this for the first image, so now the current working directory has changed to optimalized. When you then loop back up for the next file, you are now in the wrong location altogether to read the next file. Now lew.jpg, which might exist in ./unoptimalized or the parent directory, can't be found because it is not there in ./optimalized.
The solution is to add on the directory name. You can do so with os.path.join(), and not changing directories:
def optimalizeImages():
input_dir = './unoptimalized'
output_dir = './optimalized'
for fname in os.listdir(input_dir):
if fname.endswith(".jpg"):
print(fname)
source = tinify.from_file(os.path.join(input_dir, fname))
print("processing", fname)
source.to_file(os.path.join(output_dir, "optimalized-" + fname))
print("changed", fname)
Note that this still depends heavily on the current working directory being correct, but at least it is now stable and stays at the parent directory of both optimalized and unoptimalized. You may want to change that to using absolute paths.
And a side note on language: In English, the correct spelling of the two terms is optimized and unoptimized. I didn't correct those in my answer to keep the focus on what is going wrong.

For loop throwing exception on file already deleted during os walk

I'm writing a script that walks through a directory and looks for files with the typical Windows installer extensions and deletes them. When I run this with a list (vs say checking for .msi or .exe), it breaks when going through the nested loop again. It seems as if it runs though my list, deletes one type of extension then runs through the loop again and attemtps to find the same extension then throws an exception. Here is the output when I simply print, but not remove a file:
> C:\Users\User\Documents\Python Scripts>python test.py < test_run.txt
> Found directory: . Found directory: .\test_files
> Deleting test.cub
> Deleting test.idt
> Deleting test.idt
> Deleting test.msi
> Deleting test.msm
> Deleting test.msp
> Deleting test.mst
> Deleting test.pcp
> Deleting test1.exe
When I attempt to run it with os.remove it gives the following:
Found directory: .
Found directory: .\test_files
Deleting test.cub
Traceback (most recent call last):
File "test.py", line 13, in <module>
os.remove(fileName)
FileNotFoundError: [WinError 2] The system cannot find the file specified: 'test.cub'
I read up on os walk and that seems to be working properly, I can't seem to figure out where this script is going wrong. The code is below:
import os
myList = [".msi", ".msm", ".msp", ".mst", ".idt", ".idt", ".cub", ".pcp", ".exe"]
rootDir = '.'
for dirName, subdrList, fileList in os.walk(rootDir):
print('Found directory: %s' %dirName)
for fileName in fileList:
for extName in myList:
if(fileName.endswith(extName)):
print('\t Deleting %s' % fileName)
os.remove(fileName)

The correct relative name of the file test.cub is .\test_files\test.cub.
The relative name you are supplying is .\test.cub.
As it says in the os.walk documentation:
To get a full path (which begins with top) to a file or directory in
dirpath, do os.path.join(dirpath, name).

Python Permission Error when reading

import os
import rarfile
file = input("Password List Directory: ")
rarFile = input("Rar File: ")
passwordList = open(os.path.dirname(file+'.txt'),"r")
with this code I am getting the error:
Traceback (most recent call last):
File "C:\Users\Nick L\Desktop\Programming\PythonProgramming\RarCracker.py", line 7, in <module>
passwordList = open(os.path.dirname(file+'.txt'),"r")
PermissionError: [Errno 13] Permission denied: 'C:\\Users\\Nick L\\Desktop'
This is weird because I have full permission to this file as I can edit it and do whatever I want, and I am only trying to read it. Every other question I read on stackoverflow was regarding writing to a file and getting a permissions error.

You're trying to open a directory, not a file, because of the call to dirname on this line:
passwordList = open(os.path.dirname(file+'.txt'),"r")
To open the file instead of the directory containing it, you want something like:
passwordList = open(file + '.txt', 'r')
Or better yet, use the with construct to guarantee that the file is closed after you're done with it.
with open(file + '.txt', 'r') as passwordList:
# Use passwordList here.
...
# passwordList has now been closed for you.
On Linux, trying to open a directory raises an IsADirectoryError in Python 3.5, and an IOError in Python 3.1:
IsADirectoryError: [Errno 21] Is a directory: '/home/kjc/'
I don't have a Windows box to test this on, but according to Daoctor's comment, at least one version of Windows raises a PermissionError when you try to open a directory.
PS: I think you should either trust the user to enter the whole directory-and-file name him- or herself --- without you appending the '.txt' to it --- or you should ask for just the directory, and then append a default filename to it (like os.path.join(directory, 'passwords.txt')).
Either way, asking for a "directory" and then storing it in a variable named file is guaranteed to be confusing, so pick one or the other.

os.path.dirname() will return the Directory in which the file is present not the file path. For example if file.txt is in path= 'C:/Users/Desktop/file.txt' then os.path.dirname(path)wil return 'C:/Users/Desktop' as output, while the open() function expects a file path.
You can change the current working directory to file location and open the file directly.
os.chdir(<File Directory>)
open(<filename>,'r')
or
open(os.path.join(<fileDirectory>,<fileName>),'r')

Bug in Python Renaming Program.....No such file or Directory (Fnmatch)

I'm trying to build a little renaming program to help save me time in the future.
Basically it will go through directories I point it too and rename files if they meet certain criteria.
I have written what I need but I have a bug in the very beginning that I can't figure out.
Here is the code:
import os
import fnmatch
for file in os.listdir("""/Users/Desktop/TESTME"""):
if fnmatch.fnmatch(file,'MISC*'):
os.rename(file, file[4:12] + '-13-Misc.jpg')
When I try to run it I am getting this:
Traceback (most recent call last):
File "/Users/Documents/Try.py", line 6, in <module>
os.rename(file, file[4:12] + '-13-Misc.jpg')
OSError: [Errno 2] No such file or directory
I also tried this:
if fnmatch.fnmatch(file,'MISC*'):
fun = file[4:12] + '-13-Misc.jpg'
os.rename(file, fun)
But I get the same thing.
It's not recognizing the file as a file.
Am I going about this the wrong way?

You'll need to include the full path to the filenames you are trying to rename:
import os
import fnmatch
directory = "/Users/Desktop/TESTME"
for file in os.listdir(directory):
if fnmatch.fnmatch(file, 'MISC*'):
path = os.path.join(directory, file)
target = os.path.join(directory, file[4:12] + '-13-Misc.jpg'
os.rename(path, target)
The os.path.join function intelligently joins path elements into a whole, using the correct directory separator for your platform.

The function os.listdir() only returns the file names of the files in the given directory, not their full paths. You can use os.path.join(directory, file_name) to reconstruct the full path of the file.
You could also do this in bash:
cd /Users/Desktop/TESTME/
for f in MISC*; do mv "$f" "${f:4:8}-13-Misc.jpg"; done