Im trying to make a data parser but using python for my project but each files are inside a separate folders to each other. Currently i am able to read the first folder but i havent figured out how to read the folder after that and put it in a for loop
import os
r_path='//esw-fs01/esw_niagara_no_bck/BuildResults/master/0.1.52.68_390534/installation_area/autotestlogs_top/'
sd_path='/.'
root = os.listdir(r_path)
subdir=os.listdir(sd_path)
for entry in root:
# print(entry)
if os.path.isdir(os.path.join(r_path, entry)):
for subentry in subdir:
if os.path.isdir(os.path.join(r_path,'/ConfigurationsTest_19469')):
print(subentry)
For the second for loop, i want to iterate every folder that are in autotestlogs folder. i tried to make it but it doesnt work apparently. Please help Thanks
I think you messed your order up a little bit. If you do subdir = os.listdir(sd_path) before the loop you can't possibly get the sub directories because you need to use the parent directory to get to them.
So in your loop after you checked that an "entry" is a folder you can store the absolute path of this folder in a variable and then list it's contents with os.listdir().
Then you can loop through those and parse them.
How I would do it:
import os
r_path='//esw-fs01/esw_niagara_no_bck/BuildResults/master/0.1.52.68_390534/installation_area/autotestlogs_top/'
root = os.listdir(r_path)
for entry in root:
# print(entry)
subdir_path = os.path.join(r_path, entry) # create the absolute path of the subdir
if os.path.isdir(subdir_path): # check if it is a folder
subdir_entries = os.listdir(subdir_path) # get the content of the subdir
for subentry in subdir_entries:
subentry_path = os.path.join(subdir_path, subentry) # absolute path of the subentry
# here you can check everything you want for example if the subentry has a specific name etc
print(subentry_path)
Related
I have a code that locates files in a folder for me by name and moves them to another set path.
import os, shutil
files = os.listdir("D:/Python/Test")
one = "one"
two = "two"
oney = "fold1"
twoy="fold2"
def findfile(x,y):
for file in files:
if x in file.lower():
while x in file.lower():
src = ('D:/Python/Test/'+''.join(file))
dest = ('D:/Python/Test/'+y)
if not os.path.exists(dest):
os.makedirs(dest)
shutil.move(src,dest)
break
findfile(one,oney)
findfile(two,twoy)
In this case, this program moves all the files in the Test folder to another path depending on the name, let's say one as an example:
If there is a .png named one, it will move it to the fold1 folder. The problem is that my code does not distinguish between types of files and what I would like is that it excludes the folders from the search.
If there is a folder called one, don't move it to the fold1 folder, only move it if it is a folder! The other files if you have to move them.
The files in the folder contain the string one, they are not called exactly that.
I don't know if I have explained myself very well, if something is not clear leave me a comment and I will try to explain it better!
Thanks in advance for your help!
os.path.isdir(path)
os.path.isdir() method in Python is used to check whether the specified path is an existing directory or not. This method follows symbolic link, that means if the specified path is a symbolic link pointing to a directory then the method will return True.
Check with that function before.
Home this helps :)
I am trying to write a piece of code that will recursively iterate through the subdirectories of a specific directory and stop only when reaching files with a '.nii' extension, appending these files to a list called images - a form of a breadth first search. Whenever I run this code, however, I keep receiving [Errno 20] Not a directory: '/Volumes/ARLO/ADNI/.DS_Store'
*/Volumes/ARLO/ADNI is the folder I wish to traverse through
*I am doing this in Mac using the Spyder IDE from Anaconda because it is the only way I can use the numpy and nibabel libraries, which will become important later
*I have already checked that this folder directly contains only other folders and not files
#preprocessing all the MCIc files
import os
#import nibabel as nib
#import numpy as np
def pwd():
cmd = 'pwd'
os.system(cmd)
print(os.getcwd())
#Part 1
os.chdir('/Volumes/ARLO')
images = [] #creating an empty list to store MRI images
os.chdir('/Volumes/ARLO/ADNI')
list_sample = [] #just an empty list for an earlier version of
#the program
#Part 2
#function to recursively iterate through folder of raw MRI
#images and extract them into a list
#breadth first search
def extract(dir):
#dir = dir.replace('.DS_Store', '')
lyst = os.listdir(dir) #DS issue
for item in lyst:
if 'nii' not in item: #if item is not a .nii file, if
#item is another folder
newpath = dir + '/' + item
#os.chdir(newpath) #DS issue
extract(newpath)
else: #if item is the desired file type, append it to
#the list images
images.append(item)
#Part 3
adni = os.getcwd() #big folder I want to traverse
#print(adni) #adni is a string containing the path to the ADNI
#folder w/ all the images
#print(os.listdir(adni)) this also works, prints the actual list
"""adni = adni + '/' + '005_S_0222'
os.chdir(adni)
print(os.listdir(adni))""" #one iteration of the recursion,
#works
extract(adni)
print(images)
With every iteration, I wish to traverse further into the nested folders by appending the folder name to the growing path, and part 3 of the code works, i.e. I know that a single iteration works. Why does os keep adding the '.DS_Store' part to my directories in the extract() function? How can I correct my code so that the breadth first traversal can work? This folder contains hundreds of MRI images, I cannot do it without automation.
Thank you.
The .DS_Store files are not being created by the os module, but by the Finder (or, I think, sometimes Spotlight). They're where macOS stores things like the view options and icon layout for each directory on your system.
And they've probably always been there. The reason you didn't see them when you looked is that files that start with a . are "hidden by convention" on Unix, including macOS. Finder won't show them unless you ask it to show hidden files; ls won't show them unless you pass the -a flag; etc.
So, that's your core problem:
I have already checked that this folder directly contains only other folders and not files
… is wrong. The folder does contain at least one regular file; .DS_Store.
So, what can you do about that?
You could add special handling for .DS_Store.
But a better solution is probably to just check each file to see if it's a file or directory, by calling os.path.isdir on it.
Or, even better, use os.scandir instead of listdir, which gives you entries with more information than just the name, so you don't need to make extra calls like isdir.
Or, best of all, just throw out this code and use os.walk to recursively visit every file in every directory underneath your top-level directory.
I have a script that runs on a folder to create contour lines. Since I have roughly 2700 DEM which need to be processed, I need a way using the script to run on all folders within the parent folder saving them to an output folder. I am not sure how to script this but it would be greatly appreciated if I could get some guidance.
The following is the script I currently have which works on a single folder.
import arcpy
from arcpy import env
from arcpy.sa import *
env.workspace = "C:/DATA/ScriptTesting/test"
inRaster = "1km17670"
contourInterval = 5
baseContour = 0
outContours = "C:/DATA/ScriptTesting/test/output/contours5.shp"
arcpy.CheckOutExtension("Spatial")
Contour(inRaster,outContours, contourInterval, baseContour)
You're probably looking for os.walk(), which can recursively walk through all subdirectories of the given directory. You can either use the current working directory, or calculate your own parent folder and start from there, or whatever - but it'll give you the filenames for everything beneath what it starts with. From there, you can make a subroutine to determine whether or not to perform your script on that file.
You can get a list of all directories like this:
import arcpy
from arcpy import env
from arcpy.sa import *
import os
# pass in your root directory here
directories = os.listdir(root_dir)
Then you can iterate over this dirs:
for directory in directories:
# I assume you want the workspace attribute set to the subfolders
env.workspace = os.path.realpath(directory)
inRaster = "1km17670"
contourInterval = 5
baseContour = 0
# here you need to adjust the outputfile name if there is a file for every subdir
outContours = "C:/DATA/ScriptTesting/test/output/contours5.shp"
arcpy.CheckOutExtension("Spatial")
Contour(inRaster,outContours, contourInterval, baseContour)
As #a625993 mentioned, os.walk could be useful too if you have recursively nested directories. But as I can read from your question, you have just single subdirectories which directly contain the files and no further directories. That's why listing just the dirs underneath your root directory should be enough.
I'm a python beginner but have some basic experience, and I need someone to please help me use the os module to rename sub folders based on their parent folder. I've been searching for answers for the past week and have not had any success. I'm assuming I need to use the os.walk method to do this.
Here is my folder structure:
C:\data\test\
C:\data\test\map1
C:\data\test\map1\1617151
C:\data\test\map2
C:\data\test\map2\181719
C:\data\test\map3
C:\data\test\map3\182726
C:\data\test\map4
C:\data\test\map4\894932
I need the results to look like this.
C:\data\test\
C:\data\test\map1
C:\data\test\map1\map1
C:\data\test\map2
C:\data\test\map2\map2
C:\data\test\map3
C:\data\test\map3\map3
C:\data\test\map4
C:\data\test\map4\map4
Can someone please help?
python 2.7:
import os
os.chdir("C:\data\test\") # go to dir
sub_dirs = os.walk('.').next()[1] # get list of subdirs
for sub_dir in sub_dirs:
sub_sub_dir = os.walk('.').next[1] # get sub-subdir
os.rmdir(sub_sub_dir) # remove sub-subdir
os.makedirs(sub_dir + '\bla') # make new sub-subdir named subdir\bla
python 3+:
import os
os.chdir("C:\data\test\")
sub_dirs=next(os.walk('.'))[1]
for sub_dir in sub_dirs:
sub_sub_dir = next(os.walk('.'))[1]
os.rmdir(sub_sub_dir)
os.makedirs(sub_dir + '\bla')
Untested, but should do it.
You can get a list of all the files and it's respective folder location using this one liner:
here = '.' # Current location
files = [(root, files) for root, dirs, files in os.walk(here) if (not dirs and files)]
For the given folder structure it will return:
[
('C:\data\test\map1', ['1617151']),
...
]
You can now loop over this list and rename the files (https://docs.python.org/3/library/os.html#os.rename). You can get the parent folder's name by splitting the root string (root.split('\')[-1]).
I am new to python and have to play with directories.
Can someone help me with it?
I have to know a path with newly created folder. A new temp folder is generated by my code in which some files are copied. I have to know the path of this folder , but folder name is random , so I want to fetch the folder name of latest created folder.
What shall i do?
You can do something like this:
import os
directory = '.' # current dir
folders = os.walk(directory).next()[1]
creation_times = [(folder, os.path.getctime(folder)) for folder in folders]
creation_times.sort(key=lambda x: x[1]) # sort by creation time
Then you can pick the last element of this list:
most_recent = creation_times[-1][0]
Its a bad practice to fetch the latest folder created because some other application may create a folder right before you fetch the latest folder, but if you really want to try it out the code what elyase has demonstrated is fine.