We Keep Coding

Python - Multiprocessing reading files from sub directories and creating csv file list output

I'm new to python.
I'm trying to read all pdf files on sub-folders and add the name and file path of each file to a csv file. ( This part of the code works)
I would like to use multiprocesseing to read all this files faster and create the csv file output.
Can anyone give me some tips?
Thank you.
import os
import csv
# Right function using slice
def right(s, amount):
return s[-amount:]
# File Directory
search_directory ="C:/Myfolder/"
# File Name for Output CSV
output_file = right(search_directory,8) +'_pdf_files_.csv'
with open(output_file, 'w',newline='') as csvfile:
list_writer= csv.writer(csvfile,delimiter=',',quotechar=',', quoting=csv.QUOTE_MINIMAL)
for path,sub, files in os.walk(search_directory):
for filename in files:
if filename.endswith('.pdf'):
file_path = os.path.join(path,filename)
file_list =[file_path, filename[5:15]]
list_writer.writerow(file_list)
print('Reading:'+ filename)

how to write csv file into specific folder

i am trying to write several .csv file into one specific directory
here is my code
with open(f+'.csv', 'w', newline='') as csvfile:
writer = csv.writer(csvfile)
writer.writerow(["index", "B", "G", "R"])
for row in rows:
writer.writerow(row)
writer.writerow(["Mean", mean_b/total_b, mean_g/total_g, mean_r/total_r])
writer.writerow("STD", np.sqrt(var_b/total_b), np.sqrt(var_g/total_g), np.sqrt(var_r/total_r))
i have created the csv file into the directory which is same as the .py file
however i would like to create a directory and create my csv file in it
i know i need to us os.makedirs() function
but i don't know whether i have to create the directory first and designate the path for the csv file or i simply put the directory name into the open() function
please help me

Instead of using os I recommend using the pathlib module.
You can create a directory with:
path = Path('path/to/dir')
path.mkdir(parents=True)
to create the directory and all its missing parent dirs.
After doing this you can create a file in the new directory with
fpath = (path / 'filename').with_suffix('.csv')
with fpath.open(mode='w+') as csvfile:
# your csv writer code

I would simply create the directory and except directory exists error
try:
os.mkdir("./CSV")
except OSError as e:
print("Directory exists")
with open("./CSV/" + f + ".csv", newline="") as csvfile:
[...]

You can add a check for the directory like this just before open statement
dir_path = 'folder_to_save_csv_file_in'
if not os.path.isdir(dir_path):
os.makedirs(dir_path)
with open('{file_path}.csv'.format(file_path=os.path.join(dir_path, file_name), 'w+') as csv_file:
....

How open file based on extension?

I want to open any .txt file in the same directory.
In ruby I can do
File.open("*.txt").each do |line|
puts line
end
In python I can't do this it will give an error
file = open("*.txt","r")
print(file.read())
file.close()
It gives an error invalid argument.
So is there any way around it?

You can directly use the glob module for this
import glob
for file in glob.glob('*.txt'):
with open(file, 'r') as f:
print(f.read())

Use os.listdir to list all files in the current directory.
all_files = os.listdir()
Then, filter the ones which have the extension you are looking for and open each one of them in a loop.
for filename in all_files:
if filename.lower().endswith('.txt'):
with open(filename, 'rt') as f:
f.read()

Python file-IO and zipfile. Trying to loop through all the files in a folder and then loop through the texts in respective file using Python

Trying to extract all the zip files and giving the same name to the folder where all the files are gonna be.
Looping through all the files in the folder and then looping through the lines within those files to write on a different text file.
This is my code so far:
#!usr/bin/env python3
import glob
import os
import zipfile
zip_files = glob.glob('*.zip')
for zip_filename in zip_files:
dir_name = os.path.splitext(zip_filename)[0]
os.mkdir(dir_name)
zip_handler = zipfile.ZipFile(zip_filename, "r")
zip_handler.extractall(dir_name)
path = dir_name
fOut = open("Output.txt", "w")
for filename in os.listdir(path):
for line in filename.read().splitlines():
print(line)
fOut.write(line + "\n")
fOut.close()
This is the error that I encounter:
for line in filename.read().splitlines():
AttributeError: 'str' object has no attribute 'read'

You need to open the file and also join the path to the file, also using splitlines and then adding a newline to each line is a bit redundant:
path = dir_name
with open("Output.txt", "w") as fOut:
for filename in os.listdir(path):
# join filename to path to avoid file not being found
with open(os.path.join(path, filename)):
for line in filename:
fOut.write(line)
You should always use with to open your files as it will close them automatically. If the files are not large you can simply fOut.write(f.read()) and remove the loop.
You also set path = dir_name which means path will be set to whatever the last value of dir_name was in your first loop which may or may not be what you want. You can also use iglob to avoid creating a full list zip_files = glob.iglob('*.zip').

Opening a file in a folder in Python?

I want to open a file to write to.
with open(oname.text , 'w') as f:
and now I want to write the files in a folder "Playlist"
I know that I have to use os.path But I do not know how to use it
ty all

path = os.path.join('Playlist', oname.text)
with open(path, 'w') as f:
...
If you're not sure if the 'Playlist' subdir of the current directory already exists, prefix that with:
if not os.path.isdir('Playlist'):
if os.path.exists('Playlist'):
raise RuntimeError('Playlist exists and is a file, now what?!')
os.mkdir('Playlist')
This raises an exception if 'Playlist' does exist but as a file, not a directory -- handle this anomalous case as you wish, but unless you remove or rename the file, you're not going to be able to have it as a directory as well!
Use os.makedirs instead of os.mkdir if the path you desire has multiple levels of directories, e.g Play/List/Whatever (you could use it anyway just in case).

You could change the current working directory using os.chdir function.
os.chdir('Playlist')
with open(oname.text , 'w') as f:
...

Use with statement and os.path.join method
dir_path = "/home/Playlist"
file_path = os.path.join('dir_path, "oname.txt")
content = """ Some content..."""
with open(file_path, 'wb') as fp:
fp.write(content)
OR
fp = open(file_path, "wb"):
fp.write(content)
fp.close()