Problem statement:
While automatically copying files between input directories, and output directories my program fails on a path that contains unicode (most likely Korean) characters.
The whole script is publicly available under: This Link
The file that causes the error is also publicly available: File That Causes the Error
The specific part of the code that fails seems to be:
for root, _, filenames in os.walk(maybe_dir):
for file in filenames:
# Prepare relative paths:
relative_dir = os.path.relpath(root, maybe_dir)
relative_file = os.path.join(relative_dir, file)
# Get unique filename:
unique_filename = uuid.uuid4().hex
unique_filename_with_ext = unique_filename + file_extension
new_path_and_filename = os.path.join(
full_output_path, unique_filename_with_ext
)
current_file = os.path.abspath(os.path.join(root, file))
# Copying files:
shutil.copy(current_file, new_path_and_filename)
The error:
Traceback (most recent call last):
File "F:\Projects\SC2DatasetPreparator\src\directory_flattener.py", line 96, in <module>
directory_flattener(
File "F:\Projects\SC2DatasetPreparator\src\directory_flattener.py", line 60, in directory_flattener
shutil.copy(current_file, new_path_and_filename)
File "D:\Programs\Python3_10\lib\shutil.py", line 417, in copy
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "D:\Programs\Python3_10\lib\shutil.py", line 254, in copyfile
with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: 'F:\\Projects\\SC2DatasetPreparator\\processing\\directory_flattener\\input\\2017_IEM_XI_World_Championship_Katowice\\IEM XI - World Championship - StarCraft II Replays\\RO24\\Group A\\Solar Vs herO\\Ùë¦ý+ñÝü¼ ý×¼Û¦£Ù¦£ ýºÇÛÁ¼ - ÝåáÙäêÙ¿+Ýè© (Û¦ÁÝùêýØÿ ý£áýé¦) (2) Solar vs Hero game 1.SC2Replay'
The error itself is unexpected as I am using absolute paths and the script works for 6 other directories before it fails on that specific file.
The path clearly exists and can be accessed manually:
Steps to attempt to reproduce the error are as follows:
Clone the repository: Branch 1.1.0_testing
Place the File That Causes the Error in ./processing/directory_flattener/input/test_dir
Run the script
Closing Remarks:
It seems that the script worked before on Python 3.7 because I have verified the output that I have received before updating to Python 3.10 and within the directory mapping that is created the files with unicode characters in their path are present:
{
"ce2f4610891e472190a0852c617b35e8": "RO24\\Group A\\Solar Vs herO\\\u00d9\u00eb\u00a6\u00fd+\u00f1\u00dd\u00fc\u00bc \u00fd\u00d7\u00bc\u00db\u00a6\u00a3\u00d9\u00a6\u00a3 \u00fd\u00ba\u00c7\u00db\u00c1\u00bc - \u00dd\u00e5\u00e1\u00d9\u00e4\u00ea\u00d9\u00bf+\u00dd\u00e8\u00a9 (\u00db\u00a6\u00c1\u00dd\u00f9\u00ea\u00fd\u00d8\u00ff \u00fd\u00a3\u00e1\u00fd\u00e9\u00a6) (2) Solar vs Hero game 1.SC2Replay",
"dcc82d633910479c95d06ef418fcf2e0": "RO24\\Group A\\Solar Vs herO\\\u00fd\u00fb\u00a6\u00d9\u00a6\u00e4\u00fd\u00e4\u00f1 \u00d9\u00aa\u00bc\u00dd\u00f6\u00e4 - \u00d9\u00d7\u00ff\u00d9\u00ec\u00f6 Solar vs Hero game 2.SC2Replay",
}
While searching for an answer I have only stumbled upon similar problems in Python 2 where the .decode() method was suggested as a solution. Applying such measures did not help to solve the issue.
The cause for this error was the Maximum Path Length Limitation which limited the ability to use paths longer than 260 characters on Windows.
The error was fixed by adding a prefix of "\\?\" to the path that was used to access and copy the file.
This means that the following line of code was changed:
current_file = os.path.abspath(os.path.join(root, file))
Into this:
current_file = "\\\\?\\" + os.path.abspath(os.path.join(root, file))
Related
I am taking an online course in Python. I am using version 3.8.1 on Windows. I am trying to write a program which will retrieve the size of all the files within a certain folder. My current working directory is 'c:\'. This is the code I have written:
for filename in os.listdir('c:\\mypythonscripts'):
if not os.path.isfile(os.path.join('c:\\mypythonscripts', filename)):
continue
totalSize = totalSize + os.path.getsize(os.path.join('c:\\mypythonscripts', filename))
This is the error message it produces:
Traceback (most recent call last):
File "<pyshell#44>", line 1, in <module>
for filename in os.listdir('c:\\mypythonscripts'):
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'c:\\mypythonscripts'
The spelling of the file name is correct and I am able to retrieve the file size when I use an absolute path:
for filename in os.listdir('c:\\users\\owner\\mypythonscripts'):
if not os.path.isfile(os.path.join('c:\\users\\owner\\mypythonscripts', filename)):
continue
totalSize = totalSize + os.path.getsize(os.path.join('c:\\users\\owner\\mypythonscripts', filename))
>>> totalSize
2281314
Can someone tell me what is preventing Python from recognizing the relative path?
If you're running the script from the mypythonscripts directory you can go os.listdir('.').
If you're running the script from the owner directory, you can go os.listdir('mypythonscripts').
I'm slowly starting to study about python and wanted to write simple script which use tinify api, takes photos from unoptimalized directiory, compress it and put to optimalized directory. So far works partly, I mean weirdly I need to keep photos in main directory and unoptimalized one. If I dont have another copy in one of these directories, I have error. another thing is that after I launch script, only first photo is compressed and put inside optimalized directory, and then error appears.
So far I'm experimenting on two photos: lew.jpg and kot.jpg
My directory structure is like this:
Main root directory with script, and two other directories inside (optimalized and unoptimalized)
def optimalizeImages():
for fname in os.listdir('./unoptimalized'):
if fname.endswith(".jpg"):
print(fname)
source = tinify.from_file(fname)
print("processing", fname)
os.chdir("./optimalized")
source.to_file("optimalized-" + fname)
print("changed", fname)
optimalizeImages()
Error after processing first image:
Traceback (most recent call last):
File "python.py", line 20, in <module>
optimalizeImages()
File "python.py", line 11, in optimalizeImages
source = tinify.from_file(fname)
File "/home/grzymowiczma/.local/lib/python2.7/site-packages/tinify/__init__.py", line 79, in from_file
return Source.from_file(path)
File "/home/grzymowiczma/.local/lib/python2.7/site-packages/tinify/source.py", line 13, in from_file
with open(path, 'rb') as f:
IOError: [Errno 2] No such file or directory: 'kot.jpg'
and if i keep photos only in root directory, no error but also no any action, if i keep them only in unoptimalized i get same error:
Traceback (most recent call last):
File "python.py", line 20, in <module>
optimalizeImages()
File "python.py", line 11, in optimalizeImages
source = tinify.from_file(fname)
File "/home/grzymowiczma/.local/lib/python2.7/site-packages/tinify/__init__.py", line 79, in from_file
return Source.from_file(path)
File "/home/grzymowiczma/.local/lib/python2.7/site-packages/tinify/source.py", line 13, in from_file
with open(path, 'rb') as f:
IOError: [Errno 2] No such file or directory: 'lew.jpg'
There are two issues here, both because you are using relative filenames. os.listdir() gives you those relative filenames, without a path. Instead of ./unoptimalized/kot.jpg, you get just kot.jpg.
So when you try to load the image:
source = tinify.from_file(fname)
you tell the library to load image.jpg without any context other than the current working directory. And that's not the right directory, not if os.listdir('./unoptimalized') worked to list all image files; that indicates that the current working directory is in the parent directory of both unoptimalized and optimalized.
You solved that by putting an image file with the same name in the parent directory, but that's not quite the right way to solve this. More on this below.
The next issue is that you change the working directory:
os.chdir("./optimalized")
You do this for the first image, so now the current working directory has changed to optimalized. When you then loop back up for the next file, you are now in the wrong location altogether to read the next file. Now lew.jpg, which might exist in ./unoptimalized or the parent directory, can't be found because it is not there in ./optimalized.
The solution is to add on the directory name. You can do so with os.path.join(), and not changing directories:
def optimalizeImages():
input_dir = './unoptimalized'
output_dir = './optimalized'
for fname in os.listdir(input_dir):
if fname.endswith(".jpg"):
print(fname)
source = tinify.from_file(os.path.join(input_dir, fname))
print("processing", fname)
source.to_file(os.path.join(output_dir, "optimalized-" + fname))
print("changed", fname)
Note that this still depends heavily on the current working directory being correct, but at least it is now stable and stays at the parent directory of both optimalized and unoptimalized. You may want to change that to using absolute paths.
And a side note on language: In English, the correct spelling of the two terms is optimized and unoptimized. I didn't correct those in my answer to keep the focus on what is going wrong.
I am creating a Python application that imports many JSON files. The files are in the same folder as the python script's location. Before I moved the entire folder someplace else, the files imported perfectly. Since the script creates a files if none exists, it keeps creating the file in the home directory while ignoring the one in the same folder as it is in. When I specify an absolute path (code below):
startT= time()
with open('~/Documents/CincoMinutos-master/settings.json', 'a+') as f:
f.seek(0,0) # places pointer at start of file
corrupted = False
try:
# turns all json info into vars with load
self.s_settings = json.load(f)
self.s_allVerbs = []
# --- OFFLINE MODE INIT ---
if self.s_settings['Offline Mode']: # conjugation file reading only happens if setting is on
with open('~/Documents/CincoMinutos-master/verbconjugations.json', 'r+', encoding='utf-8') as f2:
self.s_allVerbs = [json.loads(line) for line in f2]
# --- END OFFLINE MODE INIT ---
for key in self.s_settings:
if not isinstance(self.s_settings[key], type(self.s_defaultSettings[key])): corrupted = True
except Exception as e: # if any unexpected error occurs
corrupted = True
print('File is corrupted!\n',e)
if corrupted or not len(self.s_settings):
f.truncate(0) # if there are any errors, reset & recreate the file
json.dump(self.s_defaultSettings, f, indent=2, ensure_ascii=False)
self.s_settings = {key: self.s_defaultSettings[key] for key in self.s_defaultSettings}
# --- END FILE & SETTINGS VAR INIT ---
print("Finished loading file in {:4f} seconds".format(time()-startT))
It spits out a FileNotFound error.
Traceback (most recent call last):
File "/Users/23markusz/Documents/CincoMinutos-master/__main__.py", line 709, in <module>
frame = CincoMinutos(root)
File "/Users/23markusz/Documents/CincoMinutos-master/__main__.py", line 42, in __init__
with open('~/Documents/CincoMinutos-master/settings.json', 'a+') as f:
FileNotFoundError: [Errno 2] No such file or directory: '~/Documents/CincoMinutos-master/settings.json'
Keep in mind that I am perfectly able to access it with the same absolute path when I operate from terminal. Can somebody please explain what I need to do in order for the files to import correctly?
Also, I am creating this application for multiple users. While /Users/23markusz/Documents/CincoMinutos-master/verbconjugations.json does work, it will not on another user's system. This file is also in the SAME FOLDER as the script so it should import correctly.
UPDATE:
While my issue is solved using os.path.expanduser(), I still do not understand why python refuses to open a file that is within the same folder as the python script. It should automatically open the file with just the filename and not the absolute path.
"~" isn't a real directory (and would not qualify as an "absolute path"), and that's why the open doesn't work.
In order to expand the tilde to an actual directory (e.g. /Users/23markusz), you can use os.path.expanduser:
import os
...
with open(os.path.expanduser('~/Documents/CincoMinutos-master/settings.json'), 'a+') as f:
# Do stuff
I'm writing a script that walks through a directory and looks for files with the typical Windows installer extensions and deletes them. When I run this with a list (vs say checking for .msi or .exe), it breaks when going through the nested loop again. It seems as if it runs though my list, deletes one type of extension then runs through the loop again and attemtps to find the same extension then throws an exception. Here is the output when I simply print, but not remove a file:
> C:\Users\User\Documents\Python Scripts>python test.py < test_run.txt
> Found directory: . Found directory: .\test_files
> Deleting test.cub
> Deleting test.idt
> Deleting test.idt
> Deleting test.msi
> Deleting test.msm
> Deleting test.msp
> Deleting test.mst
> Deleting test.pcp
> Deleting test1.exe
When I attempt to run it with os.remove it gives the following:
Found directory: .
Found directory: .\test_files
Deleting test.cub
Traceback (most recent call last):
File "test.py", line 13, in <module>
os.remove(fileName)
FileNotFoundError: [WinError 2] The system cannot find the file specified: 'test.cub'
I read up on os walk and that seems to be working properly, I can't seem to figure out where this script is going wrong. The code is below:
import os
myList = [".msi", ".msm", ".msp", ".mst", ".idt", ".idt", ".cub", ".pcp", ".exe"]
rootDir = '.'
for dirName, subdrList, fileList in os.walk(rootDir):
print('Found directory: %s' %dirName)
for fileName in fileList:
for extName in myList:
if(fileName.endswith(extName)):
print('\t Deleting %s' % fileName)
os.remove(fileName)
The correct relative name of the file test.cub is .\test_files\test.cub.
The relative name you are supplying is .\test.cub.
As it says in the os.walk documentation:
To get a full path (which begins with top) to a file or directory in
dirpath, do os.path.join(dirpath, name).
This is one of my first python scripts designed to go into a directory and rename all the files of a certain extension to the same thing. For instance, go into a music directory and change the name of all the album artwork files to something like folder.jpg so that they're easily recognized and displayed by music playing software like foobar2000.
I'm currently getting the error in the title. I've tried:
os.chmod(pathname, 0777)
and
os.chmod(pathname, stat.S_IWOTH)
to allow other entities to edit the directory I'm looking at, but the error remains. I'm not too familiar with Windows's permissions system or with Python so this is mystifying me. Any idea where I'm going wrong?
#Changes file names in every subdirectory of the target directory
import os
import sys
import shutil
#Get target path
pathname = raw_input('Enter path for music directory (ex. C:\\Music): ')
try:
os.chdir(pathname)
except:
print('Failed. Invalid directory?')
sys.exit()
#Get variables for renaming
fn = raw_input('Enter desired file name for all converted files: ')
ft = raw_input('Enter the file extension you want the program to look for (ex. .jpg): ')
outname = fn + ft
#Create path tree
for path, subdirs, files in os.walk (pathname):
#Search tree for files with defined extention
for name in files:
if name.lower().endswith(ft):
#Rename the files
src = os.path.join(path, name)
print (src)
dst = os.path.join(path, outname)
print dst
shutil.move(src, dst)
print('Complete')
A second more benign issue is that when I use a print statement to check on what files are being edited, it seems that before the first .jpg is attempted, the program processes and renamed a file called d:\test\folder.jpg which doesn't exist. This seems to succeed and the program fails in the last loop the second time through, when an existing file is processed. The program runs like this:
>>> ================================ RESTART ================================
>>>
Enter path for music directory (ex. C:\Music): d:\test
Enter desired file name for all converted files: folder
Enter the file extension you want the program to look for (ex. .jpg): .jpg
d:\test\folder.jpg
d:\test\folder.jpg
d:\test\Anathema\Alternative 4\AA.jpg
d:\test\Anathema\Alternative 4\folder.jpg
Traceback (most recent call last):
File "D:\Documents\Programming\Python scripts\Actually useful\bulk_filename_changer.py", line 29, in <module>
shutil.move(src, dst)
File "C:\Python27\lib\shutil.py", line 301, in move
copy2(src, real_dst)
File "C:\Python27\lib\shutil.py", line 130, in copy2
copyfile(src, dst)
File "C:\Python27\lib\shutil.py", line 83, in copyfile
with open(dst, 'wb') as fdst:
IOError: [Errno 13] Permission denied: 'd:\\test\\Anathema\\Alternative 4\\folder.jpg'
>>>
The glitch seems to be benign because it doesn't cause an error, but it might be related to an error I have in the code that leads to the IOError I get when the first "real" file is processed. Beats me...
I am assuming that d: is not a CD drive or something not writable. If so then have you checked the properties of your files? Are any of them read only? There would also need to be more than one of the file type in the folder to cause this, I believe.