Currently I read folders individually like so:
input_location = r'path\202206'
I would like to read in all the folders at once + not read in the zipped files.
Essentially the logic id like to perform is input_location = r'path\202206' + r'path\202207' + r'path\202207' + r'path\202208'
I cant just do input_location = r'path\ as it may read in those zip files I do not want.
Is there a way to read in the folders without reading in the zip files? Or explicitly list the folder names in one variable (input_location)?
IIUC: Collecting all directories from a directory can be done using a simple list comprehension as follows.
The glob library is nice, as it returns the full filepath, whereas a function such as os.listdir() only returns the filenames.
import os
from glob import glob
dirs = [f for f in glob('/path/to/files/*') if os.path.isdir(f)]
Output:
['/path/to/files/202207',
'/path/to/files/202206',
'/path/to/files/202208',
'/path/to/files/202209']
Then, your script can iterate over the list of directories as required.
For completeness, the directory content is a follows:
202206
202206.zip
202207
202207.zip
202208
202208.zip
202209
202209.zip
If you use pathlib from Pythons standard library you can get all entries in the folder and check if the entry is a folder.
from pathlib import Path
for entry in Path('/path/to/folder').glob('*'):
if entry.is_dir():
print(entry)
A os base approach. Notice that os.listdir returns the content of the directory in a basename form and not as a path.
import os
def my_dirs(wd):
return list(filter(os.path.isdir, (os.path.join(wd, f) for f in os.listdir('.'))))
working_dir = # add path
print(*my_dirs(working_dir), sep='\n')
Remarks: to make your program platform independent you always stuffs like os.path.join or os.sep for path manipulation
What is the best way to get a list of all files in a directory, sorted by date [created | modified], using python, on a windows machine?
I've done this in the past for a Python script to determine the last updated files in a directory:
import glob
import os
search_dir = "/mydir/"
# remove anything from the list that is not a file (directories, symlinks)
# thanks to J.F. Sebastion for pointing out that the requirement was a list
# of files (presumably not including directories)
files = list(filter(os.path.isfile, glob.glob(search_dir + "*")))
files.sort(key=lambda x: os.path.getmtime(x))
That should do what you're looking for based on file mtime.
EDIT: Note that you can also use os.listdir() in place of glob.glob() if desired - the reason I used glob in my original code was that I was wanting to use glob to only search for files with a particular set of file extensions, which glob() was better suited to. To use listdir here's what it would look like:
import os
search_dir = "/mydir/"
os.chdir(search_dir)
files = filter(os.path.isfile, os.listdir(search_dir))
files = [os.path.join(search_dir, f) for f in files] # add path to each file
files.sort(key=lambda x: os.path.getmtime(x))
Update: to sort dirpath's entries by modification date in Python 3:
import os
from pathlib import Path
paths = sorted(Path(dirpath).iterdir(), key=os.path.getmtime)
(put #Pygirl's answer here for greater visibility)
If you already have a list of filenames files, then to sort it inplace by creation time on Windows (make sure that list contains absolute path):
files.sort(key=os.path.getctime)
The list of files you could get, for example, using glob as shown in #Jay's answer.
old answer
Here's a more verbose version of #Greg Hewgill's answer. It is the most conforming to the question requirements. It makes a distinction between creation and modification dates (at least on Windows).
#!/usr/bin/env python
from stat import S_ISREG, ST_CTIME, ST_MODE
import os, sys, time
# path to the directory (relative or absolute)
dirpath = sys.argv[1] if len(sys.argv) == 2 else r'.'
# get all entries in the directory w/ stats
entries = (os.path.join(dirpath, fn) for fn in os.listdir(dirpath))
entries = ((os.stat(path), path) for path in entries)
# leave only regular files, insert creation date
entries = ((stat[ST_CTIME], path)
for stat, path in entries if S_ISREG(stat[ST_MODE]))
#NOTE: on Windows `ST_CTIME` is a creation date
# but on Unix it could be something else
#NOTE: use `ST_MTIME` to sort by a modification date
for cdate, path in sorted(entries):
print time.ctime(cdate), os.path.basename(path)
Example:
$ python stat_creation_date.py
Thu Feb 11 13:31:07 2009 stat_creation_date.py
There is an os.path.getmtime function that gives the number of seconds since the epoch
and should be faster than os.stat.
import os
os.chdir(directory)
sorted(filter(os.path.isfile, os.listdir('.')), key=os.path.getmtime)
Here's my version:
def getfiles(dirpath):
a = [s for s in os.listdir(dirpath)
if os.path.isfile(os.path.join(dirpath, s))]
a.sort(key=lambda s: os.path.getmtime(os.path.join(dirpath, s)))
return a
First, we build a list of the file names. isfile() is used to skip directories; it can be omitted if directories should be included. Then, we sort the list in-place, using the modify date as the key.
Here's a one-liner:
import os
import time
from pprint import pprint
pprint([(x[0], time.ctime(x[1].st_ctime)) for x in sorted([(fn, os.stat(fn)) for fn in os.listdir(".")], key = lambda x: x[1].st_ctime)])
This calls os.listdir() to get a list of the filenames, then calls os.stat() for each one to get the creation time, then sorts against the creation time.
Note that this method only calls os.stat() once for each file, which will be more efficient than calling it for each comparison in a sort.
In python 3.5+
from pathlib import Path
sorted(Path('.').iterdir(), key=lambda f: f.stat().st_mtime)
Without changing directory:
import os
path = '/path/to/files/'
name_list = os.listdir(path)
full_list = [os.path.join(path,i) for i in name_list]
time_sorted_list = sorted(full_list, key=os.path.getmtime)
print time_sorted_list
# if you want just the filenames sorted, simply remove the dir from each
sorted_filename_list = [ os.path.basename(i) for i in time_sorted_list]
print sorted_filename_list
from pathlib import Path
import os
sorted(Path('./').iterdir(), key=lambda t: t.stat().st_mtime)
or
sorted(Path('./').iterdir(), key=os.path.getmtime)
or
sorted(os.scandir('./'), key=lambda t: t.stat().st_mtime)
where m time is modified time.
Here's my answer using glob without filter if you want to read files with a certain extension in date order (Python 3).
dataset_path='/mydir/'
files = glob.glob(dataset_path+"/morepath/*.extension")
files.sort(key=os.path.getmtime)
# *** the shortest and best way ***
# getmtime --> sort by modified time
# getctime --> sort by created time
import glob,os
lst_files = glob.glob("*.txt")
lst_files.sort(key=os.path.getmtime)
print("\n".join(lst_files))
sorted(filter(os.path.isfile, os.listdir('.')),
key=lambda p: os.stat(p).st_mtime)
You could use os.walk('.').next()[-1] instead of filtering with os.path.isfile, but that leaves dead symlinks in the list, and os.stat will fail on them.
For completeness with os.scandir (2x faster over pathlib):
import os
sorted(os.scandir('/tmp/test'), key=lambda d: d.stat().st_mtime)
this is a basic step for learn:
import os, stat, sys
import time
dirpath = sys.argv[1] if len(sys.argv) == 2 else r'.'
listdir = os.listdir(dirpath)
for i in listdir:
os.chdir(dirpath)
data_001 = os.path.realpath(i)
listdir_stat1 = os.stat(data_001)
listdir_stat2 = ((os.stat(data_001), data_001))
print time.ctime(listdir_stat1.st_ctime), data_001
Alex Coventry's answer will produce an exception if the file is a symlink to an unexistent file, the following code corrects that answer:
import time
import datetime
sorted(filter(os.path.isfile, os.listdir('.')),
key=lambda p: os.path.exists(p) and os.stat(p).st_mtime or time.mktime(datetime.now().timetuple())
When the file doesn't exist, now() is used, and the symlink will go at the very end of the list.
This was my version:
import os
folder_path = r'D:\Movies\extra\new\dramas' # your path
os.chdir(folder_path) # make the path active
x = sorted(os.listdir(), key=os.path.getctime) # sorted using creation time
folder = 0
for folder in range(len(x)):
print(x[folder]) # print all the foldername inside the folder_path
folder = +1
Here is a simple couple lines that looks for extention as well as provides a sort option
def get_sorted_files(src_dir, regex_ext='*', sort_reverse=False):
files_to_evaluate = [os.path.join(src_dir, f) for f in os.listdir(src_dir) if re.search(r'.*\.({})$'.format(regex_ext), f)]
files_to_evaluate.sort(key=os.path.getmtime, reverse=sort_reverse)
return files_to_evaluate
Add the file directory/folder in path, if you want to have specific file type add the file extension, and then get file name in chronological order.
This works for me.
import glob, os
from pathlib import Path
path = os.path.expanduser(file_location+"/"+date_file)
os.chdir(path)
saved_file=glob.glob('*.xlsx')
saved_file.sort(key=os.path.getmtime)
print(saved_file)
Turns out os.listdir sorts by last modified but in reverse so you can do:
import os
last_modified=os.listdir()[::-1]
Maybe you should use shell commands. In Unix/Linux, find piped with sort will probably be able to do what you want.
I changed the current working directory and then I want to list all the files in that directory with extension .txt. How can I achieve this?
# !/usr/bin/python
import os
path = "/home/pawn/Desktop/projects_files"
os.chdir(path)
## checking working directory
print ("working directory "+ os.getcwd()+'\n')
Now how to list all the files in the current directory (projects_files) with extension .txt?
Use glob, which is a wildcard search for files, and you don't need to change the directory either:
import glob
for f in glob.iglob('/home/pawn/Desktop/project_files/*.txt'):
print(f)
You can use glob. With glob you can use wildcards as you would do in your system command line. For example:
import glob
print( glob.glob('*.txt') )
You would need to use glob in python.
It will help you to fetch the files of a particular format..
Usage
>>> import glob
>>> glob.glob('*.gif')
Check the documentation for more details
if you don't want to import glob as suggested in the other good answers here, you can also do it with the same os moudle:
path = "/home/pawn/Desktop/projects_files"
[f for f in os.listdir(path) if f.endswith('.txt')]
Consider the following piece of code:
files = sorted(os.listdir('dumps'), key=os.path.getctime)
The objective is to sort the listed files based on the creation time. However since the the os.listdir gives only the filename and not the absolute path the key ie, the os.path.getctime throws an exception saying
OSError: [Errno 2] No such file or directory: 'very_important_file.txt'
Is there a workaround to this situation or do I need to write my own sort function?
You can use glob.
import os
from glob import glob
glob_pattern = os.path.join('dumps', '*')
files = sorted(glob(glob_pattern), key=os.path.getctime)
files = sorted(os.listdir('dumps'), key=lambda fn:os.path.getctime(os.path.join('dumps', fn)))
files = sorted([os.path.join('dumps', file) for file in os.listdir('dumps')], key=os.path.getctime)
Getting a list of absolute paths for all files in a directory using pathlib in python3.9 on Windows
from pathlib import Path
# directory name is 'dumps'
[str(child.resolve()) for child in Path.iterdir(Path('dumps'))]
Path.iterdir() takes in a pathlib object, and so we do Path(dir) to get that object. It then spits out each file as the child, but as a relative path. child.resolve() gives the absolute path, but again as a pathlib object, so we do str() on it to return a list of strings.
You can also use os.path.join with os.path.abspath, combined with map and lambda in Python.
>>>list(map(lambda x: os.path.join(os.path.abspath('mydir'), x),os.listdir('mydir')))
This will join the absolute path of mydir with os.listdir('mydir').
The output:
['/home/ahmad/Desktop/RedBuffer/Testdata/testing.avi',
'/home/ahmad/Desktop/RedBuffer/Testdata/testing2.avi',
'/home/ahmad/Desktop/RedBuffer/Testdata/text_changing.avi',
'/home/ahmad/Desktop/RedBuffer/Testdata/text_static.avi',
'/home/ahmad/Desktop/RedBuffer/Testdata/test_img.png']
Here is another solution resulting in an np array instead of list, if it works better for someone. Still uses os
import numpy as np
import os
NPFileListFullURL=np.char.add(Folder_Path, os.listdir(Folder_Path))