I'm trying to compute the maximum from each file in my data directory, with the following code:
from os import listdir
def max_files(dir):
l = listdir(dir)
for n in l:
list_num(n)
def list_num(file):
f = open(file)
lines = f.readlines()
v=[]
for n in lines:
for s in n.split():
v.append(float(s))
mx = v[0]
maxi=[]
for i in v:
if i > mx:
mx = i
maxi.append(mx)
continue
continue
return maxi
print max_files(path)
I also checked my path, and it's completely correct. The error is:
f = open(file)
IOError: [Errno 2] No such file or directory: 'bvp.txt'
bvp.txt is the first file listed in the data directory.
Why does this problem occur, and how do I fix it?
You've run into a common confusion when using the return value of functions that return lists of files in a directory.
listdir is just returning the list of files in that directory. It's not returning the path to those files, just the file name. So unless the directory on which you're operating is the current directory, this won't work; you're trying to open each file in the current directory.
Whenever using the results of listdir, if you're not going to change working directories into that directory, you need to add the directory name back to the file before you open it. Therefore, pass the full path to the file to list_num instead of just the file name:
list_num(dir + '/' + n)
Related
We are using a function to match a file pattern and if it matches we are running some jobs and eventually removing the file form the folder. But the file pattern match code is failing with "FileNotFoundError: [Errno 2] No such file or directory" when the for loop is running and some of the files are removed, this is happening in so less time that we are not able to replicate the issue in lower environment. Any idea how to catch the error or resolve it. In this case 'ABCD.XAF.XYJ250.A252.dat' file was present in the folder but removed as part of some other job.
def poke(self, context):
full_path = self.filepath
file_pattern = re.compile(self.filepattern)
os.chdir(full_path)
for files in sorted(os.listdir(full_path),key=os.path.getmtime):
if not re.match(file_pattern, files):
self.log.info(files)
self.log.info(file_pattern)
else:
print("File Sensed "+files)
return True
return False
Error:
File "File_Sensor_Operator.py", line 25, in poke
for files in sorted(os.listdir(full_path),key=os.path.getmtime):
File "/usr/lib64/python3.6/genericpath.py", line 55, in getmtime
return os.stat(filename).st_mtime
FileNotFoundError: [Errno 2] No such file or directory: 'ABCD.XAF.XYJ250.A252.dat'
IIUC, the error is happening because after the for loop starts, you have removed one of the files which means that it is no longer in the iterable returned bysorted(os.listdir(full_path),key=os.path.getmtime) which then throws the error.
So I think it should go away if you change that section of your code to this:
all_files = sorted(os.listdir(full_path),key=os.path.getmtime)
for files in all_files:
if not re.match(file_pattern, files):
That way the iterable is assigned to a variable so the state is preserved as you loop through it.
Your issue is that you collect all the files, delete one, then try to sort them by each files attributes (whereof one doesn't exist anymore)
import os
with open("dat.dat", "w+"):
pass
file_list = os.listdir(os.path.dirname(__file__))
os.remove("dat.dat")
for file_name in sorted(file_list, key=os.path.getmtime):
print(file_name)
You can fix it by assigning the filenames and attributes to a variable before sorting.
import os
with open("dat.dat", "w+"):
pass
file_list = {
filename:os.path.getmtime(filename)
for filename in os.listdir(os.path.dirname(__file__))
if os.path.exists(filename)
}
os.remove("dat.dat")
for file_name in sorted(file_list.keys(), key=file_list.get):
print(file_name)
I am trying to open the file from folder and read it but it's not locating it. I am using Python3
Here is my code:
import os
import glob
prefix_path = "C:/Users/mpotd/Documents/GitHub/Python-Sample-
codes/Mayur_Python_code/Question/wx_data/"
target_path = open('MissingPrcpData.txt', 'w')
file_array = [os.path.abspath(f) for f in os.listdir(prefix_path) if
f.endswith('.txt')]
file_array.sort() # file is sorted list
for f_obj in range(len(file_array)):
file = os.path.abspath(file_array[f_obj])
join_file = os.path.join(prefix_path, file) #whole file path
for filename in file_array:
log = open(filename, 'r')#<---- Error is here
Error: FileNotFoundError: [Errno 2] No such file or directory: 'USC00110072.txt'
You are not giving the full path to a file to the open(), just its name - a relative path.
Non-absolute paths specify locations in relation to current working directory (CWD, see os.getcwd).
You would have to either os.path.join() correct directory path to it, or os.chdir() to the directory that the files reside in.
Also, remember that os.path.abspath() can't deduce the full path to a file just by it's name. It will only prefix its input with the path of the current working directory, if the given path is relative.
Looks like you are forgetting to modify the the file_array list. To fix this, change the first loop to this:
file_array = [os.path.join(prefix_path, name) for name in file_array]
Let me reiterate.
This line in your code:
file_array = [os.path.abspath(f) for f in os.listdir(prefix_path) if f.endswith('.txt')]
is wrong. It will not give you a list with correct absolute paths. What you should've done is:
import os
import glob
prefix_path = ("C:/Users/mpotd/Documents/GitHub/Python-Sample-"
"codes/Mayur_Python_code/Question/wx_data/")
target_path = open('MissingPrcpData.txt', 'w')
file_array = [f for f in os.listdir(prefix_path) if f.endswith('.txt')]
file_array.sort() # file is sorted list
file_array = [os.path.join(prefix_path, name) for name in file_array]
for filename in file_array:
log = open(filename, 'r')
You are using relative path where you should be using an absolute one. It's a good idea to use os.path to work with file paths. Easy fix for your code is:
prefix = os.path.abspath(prefix_path)
file_list = [os.path.join(prefix, f) for f in os.listdir(prefix) if f.endswith('.txt')]
Note that there are some other issues with your code:
In python you can do for thing in things. You did for thing in range(len(things)) it's much less readable and unnecessary.
You should use context managers when you open a file. Read more here.
I am having a hard time looping through files in a directory that is different from the directory where the script was written. I also ideally would want my script through go to through all files that start with sasa. There are a couple of files in the folder such as sasa.1, sasa.2 etc... as well as other files such as doc1.pdf, doc2.pdf
I use Python Version 2.7 with windows Powershell
Locations of Everything
1) Python Script Location ex: C:Users\user\python_project
2) Main_Directory ex: C:Users\user\Desktop\Data
3) Current_Working_Directory ex: C:Users\user\python_project
Main directory contains 100 folders (folder A, B, C, D etc..)
Each of these folders contains many files including the sasa files of interest.
Attempts at running script
For 1 file the following works:
Script is run the following way: python script1.py
file_path = 'C:Users\user\Desktop\Data\A\sasa.1
def writing_function(file_path):
with open(file_path) as file_object:
lines = file_object.readlines()
for line in lines:
print(lines)
writing_function(file_path)
However, the following does not work
Script is run the following way: python script1.py A sasa.1
import os
import sys
from os.path import join
dr = sys.argv[1]
file_name = sys.argv[2]
file_path = 'C:Users\user\Desktop\Data'
new_file_path = os.path.join(file_path, dr)
new_file_path2 = os.path.join(new_file_path, file_name)
def writing_function(paths):
with open(paths) as file_object:
lines = file_object.readlines()
for line in lines:
print(line)
writing_function(new_file_path2)
I get the following error:
with open(paths) as file_object:
IO Error: [Errno 2] No such file or directory:
'C:Users\\user\\Desktop\\A\\sasa.1'
Please note right now I am just working on one file, I want to be able to loop through all of the sasa files in the folder.
It can be something in the line of:
import os
from os.path import join
def function_exec(file):
code to execute on each file
for root, dirs, files in os.walk('path/to/your/files'): # from your argv[1]
for f in files:
filename = join(root, f)
function_exec(filename)
Avoid using the variable dir. it is a python keyword. Try print(dir(os))
dir_ = argv[1] # is preferable
No one mentioned glob so far, so:
https://docs.python.org/3/library/glob.html
I think you can solve your problem using its ** magic:
If recursive is true, the pattern “**” will match any files and zero
or more directories and subdirectories. If the pattern is followed by
an os.sep, only directories and subdirectories match.
Also note you can change directory location using
os.chdir(path)
I've created a simple script to rename my media files that have lots of weird periods and stuff in them that I have obtained and want to organize further. My script kinda works, and I will be editing it to edit the filenames further but my os.rename line throws this error:
[Windows Error: Error 2: The system cannot find the file specified.]
import os
for filename in os.listdir(directory):
fcount = filename.count('.') - 1 #to keep the period for the file extension
newname = filename.replace('.', ' ', fcount)
os.rename(filename, newname)
Does anyone know why this might be? I have a feeling that it doesn't like me trying to rename the file without including the file path?
try
os.rename(filename, directory + '/' + newname);
Triton Man has already answered your question. If his answer doesn't work I would try using absolute paths instead of relative paths.
I've done something similar before, but in order to keep any name clashes from happening I temporarily moved all the files to a subfolder. The entire process happened so fast that in Windows Explorer I never saw the subfolder get created.
Anyhow if you're interested in looking at my script It's shown below. You run the script on the command line and you should pass in as a command-line argument the directory of the jpg files you want renamed.
Here's a script I used to rename .jpg files to multiples of 10. It might be useful to look at.
'''renames pictures to multiples of ten'''
import sys, os
debug=False
try:
path = sys.argv[1]
except IndexError:
path = os.getcwd()
def toint(string):
'''changes a string to a numerical representation
string must only characters with an ordianal value between 0 and 899'''
string = str(string)
ret=''
for i in string:
ret += str(ord(i)+100) #we add 101 to make all the numbers 3 digits making it easy to seperate the numbers back out when we need to undo this operation
assert len(ret) == 3 * len(string), 'recieved an invalid character. Characters must have a ordinal value between 0-899'
return int(ret)
def compare_key(file):
file = file.lower().replace('.jpg', '').replace('dscf', '')
try:
return int(file)
except ValueError:
return toint(file)
#files are temporarily placed in a folder
#to prevent clashing filenames
i = 0
files = os.listdir(path)
files = (f for f in files if f.lower().endswith('.jpg'))
files = sorted(files, key=compare_key)
for file in files:
i += 10
if debug: print('renaming %s to %s.jpg' % (file, i))
os.renames(file, 'renaming/%s.jpg' % i)
for root, __, files in os.walk(path + '/renaming'):
for file in files:
if debug: print('moving %s to %s' % (root+'/'+file, path+'/'+file))
os.renames(root+'/'+file, path+'/'+file)
Edit: I got rid of all the jpg fluff. You could use this code to rename your files. Just change the rename_file function to get rid of the extra dots. I haven't tested this code so there is a possibility that it might not work.
import sys, os
path = sys.argv[1]
def rename_file(file):
return file
#files are temporarily placed in a folder
#to prevent clashing filenames
files = os.listdir(path)
for file in files:
os.renames(file, 'renaming/' + rename_file(file))
for root, __, files in os.walk(path + '/renaming'):
for file in files:
os.renames(root+'/'+file, path+'/'+file)
Looks like I just needed to set the default directory and it worked just fine.
folder = r"blah\blah\blah"
os.chdir(folder)
for filename in os.listdir(folder):
fcount = filename.count('.') - 1
newname = filename.replace('.', ' ', fcount)
os.rename(filename, newname)
Is there any way to make it so that IPython searches through your system path to find a file if only the name of the file is given?
I have a file foo_060112.dat that lives in a folder containing a lot of data files that are uniquely named in a folder in my path. I want to be able to simply call a load or open function on this file without specifying the full path---is it possible?
You can write your own function to search through the path and return an open file object according to the mode you select:
import sys
import os
def my_open(filename, mode):
for path in sys.path:
try:
return open(os.path.join(path, filename), mode)
except IOError:
continue
return None
Example: my_open('foo_060112.dat', 'rb')
I thinks this may help you:
import os,sys
path = os.sytem('locate foo_060112.dat')
print path
or you can also use :
_SOURCE_FILE_PATH = '/home/admin/'
_each_folder = os.walk("%s/" %(_SOURCE_FILE_PATH))
for x in _each_folder:
print x
if x[1] == []:
for y in x[2]:
if y == 'locate foo_060112.dat'
f = open(y,'r')
data = f.read()