We Keep Coding

OSError: file not found

I'm trying to write a script that needs to rename (in the script itself, not in the folder) some .txt files to be able to use them in a loop, enumerating them.
I decided to use a dictionary, something like this:
import os
import fnmatch
dsc = {}
for filename in os.listdir('./texto'):
if fnmatch.fnmatch(filename, 'dsc_hydra*.txt'):
dsc[filename[:6]] = filename
print(dsc)
print(dsc['dsc_hydra1'])
The 'print(something)' are just to check if everything is going well.
I need to rename them because I'm using them in future functions and I don't want to address them using all that path stuff, something like:
IFOV = gi.IFOV_generic(gmatOUTsat1, matrixINPUTsat1, dsc['dsc_hydra1'], 'ifovfileMST.json', k_lim, height, width)
Using dsc['dsc_hydra1'], I get this error:
Traceback (most recent call last):
File "mainSMART_MST.py", line 429, in <module>
IFOV1= gi.IFOV_generic(gmatOUTsat1,matrixINPUTsat1,dsc['dsc_hydra1'],'ifovfileMST.jso',k_lim, height, width)
File "/home/alumno/Escritorio/HDD_Nuevo/HO(PY)/src/generateIFOV.py", line 49, in IFOV_generic
DCM11,DCM12,DCM13,DCM21,DCM22,DCM23,DCM31,DCM32,DCM33 = np.loadtxt(gmatDCM,unpack=True,skiprows = 2,dtype = float)
File "/home/alumno/.local/lib/python3.5/site-packages/numpy/lib/npyio.py", line 962, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/home/alumno/.local/lib/python3.5/site-packages/numpy/lib/_datasource.py", line 266, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/home/alumno/.local/lib/python3.5/site-packages/numpy/lib/_datasource.py", line 624, in open
raise IOError("%s not found." % path)
OSError: dsc_hydra1.txt not found.
I've already checked the folder and the file is there, why do I keep getting this error?

I had this same issue. It cannot locate the .txt file because you're in the wrong directory. Make sure that where you're trying to execute the code is within the directories of which the code needs. Hope this helps.

I had the same problem. In my case, inside the file.txt, I had a space at the end of the string. You should control the spaces! For example, inside the file.txt (space = -):
-365-
string1-
string2
-string3
if you remove all the spaces (-) it should work!

cannot write file with full path in Python

I am using Pandas on Mac, to read and write a CSV file, and the weird thing is when using full path, it has error and when using just a file name, it works. I post my code which works and which not works in my comments below, and also detailed error messages. Anyone have any good ideas?
sourceDf = pd.read_csv(path_to_csv)
sourceDf['nameFull'] = sourceDf['nameFirst'] + ' ' + sourceDf['nameLast']
sourceDf.to_csv('newMaster.csv') # working
sourceDf.to_csv('~/Downloads/newMaster.csv') # not working
Traceback (most recent call last):
File "/Users/foo/PycharmProjects/DataWranglingTest/CSVTest1.py", line 36, in <module>
add_full_name(path_to_csv, path_to_new_csv)
File "/Users/foo/PycharmProjects/DataWranglingTest/CSVTest1.py", line 28, in add_full_name
sourceDf.to_csv('~/Downloads/newMaster.csv')
File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/core/frame.py", line 1189, in to_csv
formatter.save()
File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/core/format.py", line 1442, in save
encoding=self.encoding)
File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/core/common.py", line 2831, in _get_handle
f = open(path, mode)
IOError: [Errno 2] No such file or directory: '~/Downloads/newMaster.csv'
Tried to use prefix r, but not working,
path_to_csv = r'~/Downloads/Master.csv'
path_to_new_csv = r'~/Downloads/Master_new.csv'
File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/core/frame.py", line 1189, in to_csv
formatter.save()
File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/core/format.py", line 1442, in save
encoding=self.encoding)
File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/core/common.py", line 2831, in _get_handle
f = open(path, mode)
IOError: [Errno 2] No such file or directory: '~/Downloads/Master_new.csv'
thanks in advance,
Lin

Try using os.path.join().
import os
(...)
output_filename = 'newMaster.csv'
output_path = os.path.join('Downloads', output_filename)
(...)
sourceDf.to_csv(output_path)
Use the same methodology to point pandas.read_csv() in the right direction.

You didn't specify python version.
On 3.4 you can use pathlib, otherwise use os.path.join() or quoting:
sourceDf.to_csv(r'~/Downloads/newMaster.csv')
Notice the r.
The problem is that /n is newline, which is not allowed in a path.

Using os.path.join with os.path.getsize, returning FileNotFoundError

In conjunction with my last question, I'm onto printing the filenames with their sizes next to them in a sort of list. Basically I am reading filenames from one file (which are added by the user), taking the filename and putting it in the path of the working directory to print it's size one-by-one, however I'm having an issue with the following block:
print("\n--- Stats ---\n")
with open('userdata/addedfiles', 'r') as read_files:
file_lines = read_files.readlines()
# get path for each file and find in trackedfiles
# use path to get size
print(len(file_lines), "files\n")
for file_name in file_lines:
# the actual files should be in the same working directory
cwd = os.getcwd()
fpath = os.path.join(cwd, file_name)
fsize = os.path.getsize(fpath)
print(file_name.strip(), "-- size:", fsize)
which is returning this error:
tolbiac wpm-public → ./main.py --filestatus
--- Stats ---
1 files
Traceback (most recent call last):
File "./main.py", line 332, in <module>
main()
File "./main.py", line 323, in main
parseargs()
File "./main.py", line 317, in parseargs
tracking()
File "./main.py", line 204, in tracking
fsize = os.path.getsize(fpath)
File "/usr/lib/python3.4/genericpath.py", line 50, in getsize
return os.stat(filename).st_size
FileNotFoundError: [Errno 2] No such file or directory: '/home/tolbiac/code/wpm-public/file.txt\n'
tolbiac wpm-public →
So it looks like something is adding a \n to the end of file_name, I'm not sure if thats something used in the getsize module, I tried this with os.stat, but it did the same thing.
Any suggestions? Thanks.

When you're reading in a file, you need to be aware of how the data is being seperated. In this case, the read-in file has a filename once per line seperated out by that \n operator. Need to strip it then before you use it.
for file_name in file_lines:
file_name = file_name.strip()
# rest of for loop

How to wait for a folder/file to be complete to finally process it in Python?

I'm trying to code a little script that watches a defined directory with a while-loop. Every file or directory that is in this directory is compressed to RAR and moved to another directory after the process is completed.
My problem: everytime I copy a file or folder to this directory, the script doesn't wait and startes the process the second it sees a new file or folder. But when the files or folders are bigger than a few kilobytes the loop breaks with a permission error.
Since I'm a Python beginner I don't know which module to use. Is there a checking module to see if the file or folder that the tool wants to process is used by another process? Or am I going in the wrong direction?
Edit: added the code for directory-only listening:
watchDir = "L:\\PythonTest\\testfolder\\"
finishedDir = "L:\\PythonTest\\finishedfolders\\"
rarfilesDir = "L:\\PythonTest\\rarfiles\\"
rarExe = "L:\\PythonTest\\rar.exe"
rarExtension = ".rar"
rarCommand = "a"
while True:
dirList = [name for name in os.listdir(watchDir) if os.path.isdir(os.path.join(watchDir,name))]
for entryName in dirList:
if not os.path.exists((os.path.join(finishedDir,entryName))):
sourcePath = os.path.join(watchDir,entryName)
entryNameStripped = entryName.replace(" ", "")
os.chdir(watchDir)
archiveName = rarfilesDir+entryNameStripped+rarExtension
subprocesscall = [rarExe, rarCommand, archiveName, entryName]
subprocess.call(subprocesscall, shell=True)
shutil.move(sourcePath,finishedDir)
When I run the script and try to add a file of several GB (named #filename# in the following lines) these errors occur:
Creating archive L:\PythonTest\rarfiles\#filename#.rar
Cannot open #filename#
The process cannot access the file, since it's used by another process.
Adding #filename# OK
WARNING: Cannot open 1 file
Done
Traceback (most recent call last):
File "C:\Python34\lib\shutil.py", line 522, in move
os.rename(src, real_dst)
PermissionError: [WinError 5] Access denied: #filepath#
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "L:/Python Test/test.py", line 35, in <module>
shutil.move(sourcePath,finishedDir)
File "C:\Python34\lib\shutil.py", line 531, in move
copytree(src, real_dst, symlinks=True)
File "C:\Python34\lib\shutil.py", line 342, in copytree
raise Error(errors)
shutil.Error: #filepath#

instead of using os.listdir, you can use os.walk, os.walk yields 3 tuple dirpath(path of directory,filenames(all files in that dirpath),dirnames(all the sub directories in dirpath)
for x,y,z in os.walk('path-of-directory'):
do you stuff with x,y,z the three tuples

Python FileExistsError

Here is my code for replacing all the occurrences of kyle and john with mike and jim respectively.
import os
import fileinput
import sys
rootdir ='C:/Users/sid/Desktop/app'
searchTerms={"kyle":"mike","john":"jim"}
def replaceAll(file,searchExp,replaceExp):
for line in fileinput.input(file, inplace=1):
if searchExp in line:
line = line.replace(searchExp,replaceExp)
sys.stdout.write(line)
for subdir, dirs, files in os.walk(rootdir):
for file in files:
path=subdir+'/'+file
for key,value in searchTerms.items():
replaceAll(path,key,value)
This was working fine for a test directory i created.
When i changed the rootdir to my actual java project directory, I was getting
Traceback (most recent call last):
File "C:\Users\sid\Desktop\test_iterator.py", line 19, in <module>
replaceAll(path,key,value)
File "C:\Users\sid\Desktop\test_iterator.py", line 10, in replaceAll
for line in fileinput.input(file, inplace=1):
File "C:\Python33\lib\fileinput.py", line 261, in __next__
line = self.readline()
File "C:\Python33\lib\fileinput.py", line 330, in readline
os.rename(self._filename, self._backupfilename)
FileExistsError: [WinError 183] Cannot create a file when that file already exists: 'C:/Users/sid/Desktop/app/pom.template.xml.bak'
Can someone please explain why im getting this error. I have read the post about os.rename() FileExistsError but i couldnot understand it. Can some please explain in detail.

When you use fileinput.input(..., inplace=1), the input file is renamed, and any output your code produces on sys.stdout is written to a newly created file with the original filename.
fileinput thus has to rename the original file first, by adding .bak to the name. However, it appears there already is such a file present. Probably you had a bug in your code before and the backup file was never deleted.
Verify that C:/Users/sid/Desktop/app/pom.template.xml.bak does not contain anything you want to keep, then delete it or move it back to C:/Users/sid/Desktop/app/pom.template.xml.
If, however, you keep running into this then Python has problems deleting the backup files automatically. On Windows this is usually because another process keeps opening files for their own purposes in the background. You could try deleting the backup file after a timeout:
import time, os
def replaceAll(file,searchExp,replaceExp):
for line in fileinput.input(file, inplace=1):
if searchExp in line:
line = line.replace(searchExp,replaceExp)
sys.stdout.write(line)
time.sleep(1) # wait 1 second, then delete the backup
os.remove(file + '.bak')
If your files are read-only, make them writable first:
import os, stat
def replaceAll(file,searchExp,replaceExp):
readonly = not os.stat(myFile)[0] & stat.S_IWRITE
if readonly:
os.chmod(file, stat.S_IWRITE)
for line in fileinput.input(file, inplace=1):
if searchExp in line:
line = line.replace(searchExp,replaceExp)
sys.stdout.write(line)
if readonly:
os.chmod(file, stat.S_IREAD)