Here is my code for replacing all the occurrences of kyle and john with mike and jim respectively.
import os
import fileinput
import sys
rootdir ='C:/Users/sid/Desktop/app'
searchTerms={"kyle":"mike","john":"jim"}
def replaceAll(file,searchExp,replaceExp):
for line in fileinput.input(file, inplace=1):
if searchExp in line:
line = line.replace(searchExp,replaceExp)
sys.stdout.write(line)
for subdir, dirs, files in os.walk(rootdir):
for file in files:
path=subdir+'/'+file
for key,value in searchTerms.items():
replaceAll(path,key,value)
This was working fine for a test directory i created.
When i changed the rootdir to my actual java project directory, I was getting
Traceback (most recent call last):
File "C:\Users\sid\Desktop\test_iterator.py", line 19, in <module>
replaceAll(path,key,value)
File "C:\Users\sid\Desktop\test_iterator.py", line 10, in replaceAll
for line in fileinput.input(file, inplace=1):
File "C:\Python33\lib\fileinput.py", line 261, in __next__
line = self.readline()
File "C:\Python33\lib\fileinput.py", line 330, in readline
os.rename(self._filename, self._backupfilename)
FileExistsError: [WinError 183] Cannot create a file when that file already exists: 'C:/Users/sid/Desktop/app/pom.template.xml.bak'
Can someone please explain why im getting this error. I have read the post about os.rename() FileExistsError but i couldnot understand it. Can some please explain in detail.
When you use fileinput.input(..., inplace=1), the input file is renamed, and any output your code produces on sys.stdout is written to a newly created file with the original filename.
fileinput thus has to rename the original file first, by adding .bak to the name. However, it appears there already is such a file present. Probably you had a bug in your code before and the backup file was never deleted.
Verify that C:/Users/sid/Desktop/app/pom.template.xml.bak does not contain anything you want to keep, then delete it or move it back to C:/Users/sid/Desktop/app/pom.template.xml.
If, however, you keep running into this then Python has problems deleting the backup files automatically. On Windows this is usually because another process keeps opening files for their own purposes in the background. You could try deleting the backup file after a timeout:
import time, os
def replaceAll(file,searchExp,replaceExp):
for line in fileinput.input(file, inplace=1):
if searchExp in line:
line = line.replace(searchExp,replaceExp)
sys.stdout.write(line)
time.sleep(1) # wait 1 second, then delete the backup
os.remove(file + '.bak')
If your files are read-only, make them writable first:
import os, stat
def replaceAll(file,searchExp,replaceExp):
readonly = not os.stat(myFile)[0] & stat.S_IWRITE
if readonly:
os.chmod(file, stat.S_IWRITE)
for line in fileinput.input(file, inplace=1):
if searchExp in line:
line = line.replace(searchExp,replaceExp)
sys.stdout.write(line)
if readonly:
os.chmod(file, stat.S_IREAD)
Related
import fileinput
def main()
try:
lines = fileinput.input()
res = process_lines(lines)
...more code
except Exception:
print('is your file path bad?')
if __name__ == '__main__':
main()
When I run this code with a bad path, it doesn't throw an error, yet the docs say that an OS error will be thrown if there's an IO error. How do I test for bad paths then?
fileinput.input() returns an iterator, not an ad-hoc list:
In [1]: fileinput.input()
Out[1]: <fileinput.FileInput at 0x7fa9bea55a50>
Proper usage of this function is done via a for loop:
with fileinput.input() as files:
for line in files:
process_line(line)
or using a conversion to list:
lines = list(fileinput.input())
I.e. the files are opened only when you actually iterate over this object.
Although I wouldn't recommend the second way, as it is counter to the philosophy of how such scripts are supposed to work.
You are supposed to parse as little as you need to output data, and then output it as soon as possible. This avoids issues with large inputs, and if your script is used within a larger pipeline, speeds up the processing significantly.
With regards to checking whether the path is correct or not:
As soon as you'll iterate down to the file that doesn't exist, the iterator will throw an exception:
# script.py
import fileinput
with fileinput.input() as files:
for line in files:
print(repr(line))
$ echo abc > /tmp/this_exists
$ echo xyz > /tmp/this_also_exists
$ python script.py /tmp/this_exists /this/does/not /tmp/this_also_exists
'abc\n'
Traceback (most recent call last):
File "/tmp/script.py", line 6, in <module>
for line in files:
File "/home/mrmino/.pyenv/versions/3.7.7/lib/python3.7/fileinput.py", line 252, in __next__
line = self._readline()
File "/home/mrmino/.pyenv/versions/3.7.7/lib/python3.7/fileinput.py", line 364, in _readline
self._file = open(self._filename, self._mode)
FileNotFoundError: [Errno 2] No such file or directory: '/this/does/not'
I'm trying to write a script that needs to rename (in the script itself, not in the folder) some .txt files to be able to use them in a loop, enumerating them.
I decided to use a dictionary, something like this:
import os
import fnmatch
dsc = {}
for filename in os.listdir('./texto'):
if fnmatch.fnmatch(filename, 'dsc_hydra*.txt'):
dsc[filename[:6]] = filename
print(dsc)
print(dsc['dsc_hydra1'])
The 'print(something)' are just to check if everything is going well.
I need to rename them because I'm using them in future functions and I don't want to address them using all that path stuff, something like:
IFOV = gi.IFOV_generic(gmatOUTsat1, matrixINPUTsat1, dsc['dsc_hydra1'], 'ifovfileMST.json', k_lim, height, width)
Using dsc['dsc_hydra1'], I get this error:
Traceback (most recent call last):
File "mainSMART_MST.py", line 429, in <module>
IFOV1= gi.IFOV_generic(gmatOUTsat1,matrixINPUTsat1,dsc['dsc_hydra1'],'ifovfileMST.jso',k_lim, height, width)
File "/home/alumno/Escritorio/HDD_Nuevo/HO(PY)/src/generateIFOV.py", line 49, in IFOV_generic
DCM11,DCM12,DCM13,DCM21,DCM22,DCM23,DCM31,DCM32,DCM33 = np.loadtxt(gmatDCM,unpack=True,skiprows = 2,dtype = float)
File "/home/alumno/.local/lib/python3.5/site-packages/numpy/lib/npyio.py", line 962, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/home/alumno/.local/lib/python3.5/site-packages/numpy/lib/_datasource.py", line 266, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/home/alumno/.local/lib/python3.5/site-packages/numpy/lib/_datasource.py", line 624, in open
raise IOError("%s not found." % path)
OSError: dsc_hydra1.txt not found.
I've already checked the folder and the file is there, why do I keep getting this error?
I had this same issue. It cannot locate the .txt file because you're in the wrong directory. Make sure that where you're trying to execute the code is within the directories of which the code needs. Hope this helps.
I had the same problem. In my case, inside the file.txt, I had a space at the end of the string. You should control the spaces! For example, inside the file.txt (space = -):
-365-
string1-
string2
-string3
if you remove all the spaces (-) it should work!
So I'm fairly new to python and i'm writing a script that needs to untar a file. I use this simple function.
def untar(source_filename, dest_dir):
for f in os.listdir():
print(f)
if(source_filename.endswith("tar.gz") or source_filename.endswith(".tar")):
tar = tarfile.open(source_filename)
tar.extractall(dest_dir)
tar.close()
else:
raise Exception("Could not retrieve .depends for that file.")
I added the initial for loop for debugging purposes. When I invoke it, it prints out the name of the file i need in the current working directory meaning that it does exist. Here is the whole output.
dep.tar.gz
Traceback (most recent call last):
File "init.py", line 70, in <module>
untar('dep.tar.gz', ".")
File "init.py", line 17, in untar
tar = tarfile.open(source_filename)
File "/usr/lib/python3.4/tarfile.py", line 1548, in open
return func(name, "r", fileobj, **kwargs)
File "/usr/lib/python3.4/tarfile.py", line 1646, in bz2open
compresslevel=compresslevel)
File "/usr/lib/python3.4/bz2.py", line 102, in __init__
self._fp = _builtin_open(filename, mode)
FileNotFoundError: [Errno 2] No such file or directory: 'dep.tar.gz'
Can someone tell me how it can see the file in the working directory, and then suddenly not be able to see the file in the working directory?
The program I was using to create the tar placed a space at the beginning of the filename. So python was looking for 'dep.tar.gz' and the actual filename was ' dep.tar.gz'. ty #Ben
TIL - filenames can start with spaces.
I am using Pandas on Mac, to read and write a CSV file, and the weird thing is when using full path, it has error and when using just a file name, it works. I post my code which works and which not works in my comments below, and also detailed error messages. Anyone have any good ideas?
sourceDf = pd.read_csv(path_to_csv)
sourceDf['nameFull'] = sourceDf['nameFirst'] + ' ' + sourceDf['nameLast']
sourceDf.to_csv('newMaster.csv') # working
sourceDf.to_csv('~/Downloads/newMaster.csv') # not working
Traceback (most recent call last):
File "/Users/foo/PycharmProjects/DataWranglingTest/CSVTest1.py", line 36, in <module>
add_full_name(path_to_csv, path_to_new_csv)
File "/Users/foo/PycharmProjects/DataWranglingTest/CSVTest1.py", line 28, in add_full_name
sourceDf.to_csv('~/Downloads/newMaster.csv')
File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/core/frame.py", line 1189, in to_csv
formatter.save()
File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/core/format.py", line 1442, in save
encoding=self.encoding)
File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/core/common.py", line 2831, in _get_handle
f = open(path, mode)
IOError: [Errno 2] No such file or directory: '~/Downloads/newMaster.csv'
Tried to use prefix r, but not working,
path_to_csv = r'~/Downloads/Master.csv'
path_to_new_csv = r'~/Downloads/Master_new.csv'
File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/core/frame.py", line 1189, in to_csv
formatter.save()
File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/core/format.py", line 1442, in save
encoding=self.encoding)
File "/usr/local/Cellar/python/2.7.8/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/core/common.py", line 2831, in _get_handle
f = open(path, mode)
IOError: [Errno 2] No such file or directory: '~/Downloads/Master_new.csv'
thanks in advance,
Lin
Try using os.path.join().
import os
(...)
output_filename = 'newMaster.csv'
output_path = os.path.join('Downloads', output_filename)
(...)
sourceDf.to_csv(output_path)
Use the same methodology to point pandas.read_csv() in the right direction.
You didn't specify python version.
On 3.4 you can use pathlib, otherwise use os.path.join() or quoting:
sourceDf.to_csv(r'~/Downloads/newMaster.csv')
Notice the r.
The problem is that /n is newline, which is not allowed in a path.
In conjunction with my last question, I'm onto printing the filenames with their sizes next to them in a sort of list. Basically I am reading filenames from one file (which are added by the user), taking the filename and putting it in the path of the working directory to print it's size one-by-one, however I'm having an issue with the following block:
print("\n--- Stats ---\n")
with open('userdata/addedfiles', 'r') as read_files:
file_lines = read_files.readlines()
# get path for each file and find in trackedfiles
# use path to get size
print(len(file_lines), "files\n")
for file_name in file_lines:
# the actual files should be in the same working directory
cwd = os.getcwd()
fpath = os.path.join(cwd, file_name)
fsize = os.path.getsize(fpath)
print(file_name.strip(), "-- size:", fsize)
which is returning this error:
tolbiac wpm-public → ./main.py --filestatus
--- Stats ---
1 files
Traceback (most recent call last):
File "./main.py", line 332, in <module>
main()
File "./main.py", line 323, in main
parseargs()
File "./main.py", line 317, in parseargs
tracking()
File "./main.py", line 204, in tracking
fsize = os.path.getsize(fpath)
File "/usr/lib/python3.4/genericpath.py", line 50, in getsize
return os.stat(filename).st_size
FileNotFoundError: [Errno 2] No such file or directory: '/home/tolbiac/code/wpm-public/file.txt\n'
tolbiac wpm-public →
So it looks like something is adding a \n to the end of file_name, I'm not sure if thats something used in the getsize module, I tried this with os.stat, but it did the same thing.
Any suggestions? Thanks.
When you're reading in a file, you need to be aware of how the data is being seperated. In this case, the read-in file has a filename once per line seperated out by that \n operator. Need to strip it then before you use it.
for file_name in file_lines:
file_name = file_name.strip()
# rest of for loop