We Keep Coding

How to go through directories to get a specific file from each directory and open that file in each directory and do some process

I have a path which have many directories. I need to go through each directory and get a specific file "file.log.gz" from it and read the file and do some process.
This is my current attempt:
import os
import sys
import gzip
infile = sys.argv[1]
directory = ("%s/NEW_FOLDER" % infile)
for root, dirs, files in os.walk(directory):
for file in files:
if "file.log.gz" in file:
with gzip.open(os.path.join(root, file)) as fin:
new = False
for line in fin:
if "CODE" in line.decode('utf-8'):
print("string is present")
found = True
exit()
else:
print("string is not present")
what i need is to go through each directories inside NEW_FOLDER and get file.log.gz. and do the following process for file.log.gz in each directory.
with the current code i get file.log.gz inside each directory but i'm not able to do rest of the process that is opening file.log.gz in each directory and do the rest process.
Expected Output:
/NEW_FOLDER/dir1/file.log.gz
string is present
/NEW_FOLDER/dir2/file.log.gz
string is present
/NEW_FOLDER/dir3/file.log.gz
string is not present

Because you are using os.walk(). You need to merge the root directory with the filename. You will notice it if you print (file) and see what the values you are getting.
Try print this out. You suppose to pass the entire directory to open and not just the file name.
for file in files:
print(os.path.join(root, file))

Find directories missing .csv file in Python

I have ~1000 directories, containing various .csv files within them. I am trying to check if a specific type of csv file, containing a filename that begins with PTSD_OCOTBER, exists in each directory.
If this file does not exist in the directory, I want to print out that directory into a .txt file.
Here is what I have so far.
import os,sys,time,shutil
import subprocess
#determine filetype to look for.
file_type = ".csv"
print("Running file counter for" + repr(file_type))
#for each folder in the root directory
for subdir, dirs, files in os.walk(rootdir):
if("GeneSet" in subdir):
folder_name = subdir.rsplit('/', 1)[-1] #get the folder name.
for f in files:
#unclear how to write this part.
#how to tell if no files exist in directory?
This successfully finds the .csv files of interest, but how do achieve the above?

So files is the list of files in that directory that you are currently walking. You want to know if there are no files that start with PTSD_OCOTBER (PTSD_OCTOBER ?):
for subdir, dirs, files in os.walk(rootdir):
if("GeneSet" in subdir):
folder_name = subdir.rsplit('/', 1)[-1] #get the folder name.
dir_of_interest = not any(f.startswith('PTSD_OCOTBER') for f in files)
if dir_of_interest:
# do stuff with folder_name
Now you want to save the results into a text file? If you have a Unix-style computer, then you can use output redirection on your terminal, such as
python3 fileanalysis.py > result.txt
after writing print(folder_name) instead of # do stuff with folder_name.
Or you can use Python itself to write the file, such as:
found_dirs = []
for subdir, dirs, files in os.walk(rootdir):
...
if dir_of_interest:
found_dirs.append(folder_name)
with open('result.txt', 'w') as f:
f.write('\n'.join(found_dirs))

python unzip all files to parent directory

How can I extract all the .zip files in a certain directory to its parent directory?
I tried:
import zipfile
parent_directory = '../input'
directory = '../input/zip'
for f in os.listdir(directory):
with zipfile.ZipFile(os.path.join(directory,f), "r") as z:
z.extractall(parent_directory)
However the unzipped files are not saved in '..input/zip', they are saved in nested folders

This might be a bit exaggerated.
After files are unzipped, I run this to:
move the original .zip file up one directory level. (to avoid /src_filename' already exists error)
move all files from all subdirectories into the zip parent directory.
move the original .zip file back into the parent directory.
import os
import shutil
src = r'C:\Users\Owner\Desktop\PythonZip\PyUnzip01\child_dir\unzip_test2'
dest = r'C:\Users\Owner\Desktop\PythonZip\PyUnzip01\child_dir'
pdir = '../PyUnzip01'
os.replace(r"C:\Users\Owner\Desktop\PythonZip\PyUnzip01\child_dir\unzip_test2.zip", r"C:\Users\Owner\Desktop\PythonZip\PyUnzip01\unzip_test2.zip")
for root, subdirs, files in os.walk(src):
for file in files:
path = os.path.join(root, file)
shutil.move(path, dest)
os.replace(r"C:\Users\Owner\Desktop\PythonZip\PyUnzip01\unzip_test2.zip", r"C:\Users\Owner\Desktop\PythonZip\PyUnzip01\child_dir\unzip_test2.zip")

Python: Create a zip file of all files ending with ".json" in a directory

Let's say the directory is /Home/Documents/Test_files.
I would like to create a zip file of all the files ending with ".json" and if possible delete the files so that only the zip file is left
So far I have been able to create a zip file of all the files in the given path but when I use the line zipf.write(file) it throws the error "[Errno 2] No such file or directory: sample.json". However when I use zipf.write(os.path.join(root, file)) it does write the files but also the whole directory path which I don't want.
I just want to write the files themselves. When I use print file the correct files seemed to be printed so I don't know why I get the error that the file doesn't exist
Currently my code looks like this:
def create_zip(path,zipf):
#path is the directory address (i.e. /Home/Documents/Test_files)
for root, dirs, files in os.walk(path):
for file in files:
if file.endswith(".json"):
print file
zipf.write(os.path.join(root, file))
#zipf.write(file)
I would also like to remove/delete the files after creating the zip file to save space.
Any help as to why this is happening would be appreciated!

You can chdir before adding it to zip file not to include the whole directory path and use os.remove to delete the files afterwards:
def create_zip(path,zipf):
for root, dirs, files in os.walk(path):
for file in files:
if file.endswith(".json"):
chdir(root)
zipf.write(file)
os.remove(file)

If you're using Python's ZipFile module, you can just specify the argument
arcname = archive name
in the write() method, as in:
import os
from zipfile import ZipFile
def create_zip(path,zipf):
#path is the directory address (i.e. /Home/Documents/Test_files)
for root, dirs, files in os.walk(path):
for file in files:
if file.endswith(".json"):
print file
zipf.write(os.path.join(root, file), arcname=file)
os.remove(os.path.join(root, file))

Python program to traverse directories and read file information

I'm just getting started with Python but already have found it much more productive than Bash shell scripting.
I'm trying to write a Python script that will traverse every directory that branches from the directory I launch the script in, and for each file it encounters, load an instance of this class:
class FileInfo:
def __init__(self, filename, filepath):
self.filename = filename
self.filepath = filepath
The filepath attribute would be the full absolute path from root (/). Here's the pseudocode mockup for what I'd like the main program to do:
from (current directory):
for each file in this directory,
create an instance of FileInfo and load the file name and path
switch to a nested directory, or if there is none, back out of this directory
I've been reading about os.walk() and ok.path.walk(), but I'd like some advice about what the most straightforward way to implement this in Python would be. Thanks in advance.

I'd use os.walk doing the following:
def getInfos(currentDir):
infos = []
for root, dirs, files in os.walk(currentDir): # Walk directory tree
for f in files:
infos.append(FileInfo(f,root))
return infos

Try
info = []
for path, dirs, files in os.walk("."):
info.extend(FileInfo(filename, path) for filename in files)
or
info = [FileInfo(filename, path)
for path, dirs, files in os.walk(".")
for filename in files]
to get a list of one FileInfo instance per file.

Try it
import os
for item in os.walk(".", "*"):
print(item)