I have the following example dataset output from pandas.
What I would like to do in an efficient way is using glob to search the filename in the associated main folder and sub folder only and not to loop through all the main folders/ sub folders as per my current code. I need this to then match against a main folders and sub folder I have and if it matches then it copies the file. I have code that works but it is very inefficient and has to go through all folders/sub folders for each search. The code is as below;At the moment, main_folder and searchdate are lists.filenames_i_want, is also the list that I will be matching to. Any way i can make it go straight to the folder/subfolder e.g if i provided this as CSV input?
import itertools
import glob
import shutil
from pathlib import Path
filenames_i_want = Search_param
main_folder=locosearch
searchfolder= Search_date
TargetFolder = r'C:\ELK\LOGS\XX\DEST'
for directory,folder in itertools.product(main_folder, searchfolder):
files = glob.glob('Z:/{}/{}/asts_data_logger/*.bz2'.format(directory, folder))
for f in files:
current_path = Path(f)
cpn = current_path.name
if current_path.name in filenames_i_want:
print(f"found target file: {f}")
shutil.copy2(f, TargetFolder)
I created a column and used the fields to make an absolute path and then used tuples to go through each row and shutil to copy
TargetFolder = r'C:\ELK\LOGS\ATH\DEST'
for row in df.itertuples():
search = row.Search
try:
shutil.copy2(search, TargetFolder)
except Exception as e:
print(e)
Related
I have a list of files like this in the images folder.
and How can I create a new folder if there are multiple files with a similar name and move those similar files to that folder?
I am new to python.
Here is my expectation:
Try this:
import glob
from pathlib import Path
for fn in Path("Images").glob("*"):
file_base_name = "_".join(fn.stem.split("_")[:-1])
file_count = len(glob.glob1("Images", f"{file_base_name}*"))
if file_count > 1 or Path(file_base_name).is_dir():
outdir = Path("Images") / file_base_name
outdir.mkdir(exist_ok=True)
fn.rename(outdir / fn.name)
Input:
Output:
Please ignore file names extension. I create those just to test my code
In this case you don't even need re:
from pathlib import Path
for fn in Path("Images").glob("*.jpg"):
outdir = Path("Images") / "_".join(fn.stem.split("_")[:-1])
outdir.mkdir(exist_ok=True)
fn.rename(outdir / fn.name)
What's going on here?
Pathlib is how you want to think of paths if you can. It combines most of the os.path apis. Specifically:
glob gets us all the files matching the glob in the path
mkdir makes the directory (only if it doesn't exist)
rename moves the file there
I am unable to test since I don't have your files. My suggestion would be to comment out the mkdir command and the shutil.move command and replace them with print statements to see what commands would be generated before letting it run for real. But I think it should work.
import pathlib
import os
import re
from itertools import groupby
import shutil
source_dir = 'Images'
files = [os.path.basename(f) for f in pathlib.Path(source_dir).glob('*.jpg')]
def keyfunc(file):
m = re.match('^(.*?)_\d+.jpg$', file)
return m[1]
matched_files = [file for file in files if re.search(r'_\d+.jpg$', file)]
matched_files.sort()
for k, g in groupby(matched_files, keyfunc):
new_dir = os.path.join(source_dir, k)
if not os.path.exists(new_dir):
os.mkdir(new_dir)
for file in g:
shutil.move(os.path.join(source_dir, file), new_dir)
I have a folder that contains a lot of subfolders, with images saved as png in each folder :
For example :
emotion\angry
emotion\disgusted
emotion\fearful
emotion\happy
I can remove the images in one of the folder using the below code :
folder_path = (r'C:\Users\emotion\angry')
test = os.listdir(folder_path)
for images in test:
if images.endswith(".png"):
os.remove(os.path.join(folder_path, images))
How do I create a loop to loop through each subfolder in emotion/?
As I do not want to manually write out all the code to clear all the folders...
You can list files using glob patterns and delete them using your normal os.remove.
import os
import glob
fileList = glob.glob('C:\Users\emotion\*\*.png')
for filePath in fileList:
try:
os.remove(filePath)
except:
print("Error while deleting file : ", filePath)
def all_dirs(directory):
# Returning list with all directories in selected directory
dir_list = []
for dir in [x[0] for x in os.walk(directory)]:
dir_list.append(dir)
return dir_list
This solution should work for every OS.
I have a one folder, within it contains 5 sub-folders.
Each sub folder contains some 'x.txt','y.txt' and 'z.txt' files and it repeats in every sub-folders
Now I need to read and print only 'y.txt' file from all sub-folders.
My problem is I'm unable to read and print 'y.txt' files. Can you tell me how solve this problem.
Below is my code which I have written for reading y.txt file
import os, sys
import pandas as pd
file_path = ('/Users/Naga/Desktop/Python/Data')
for root, dirs, files in os.walk(file_path):
for name in files:
print(os.path.join(root, name))
pd.read_csv('TextInformation.txt',delimiter=";", names = ['Name', 'Value'])
error :File TextInformation.txt does not exist: 'TextInformation.txt'
You could also try the following approach to fetch all y.txt files from your subdirectories:
import glob
import pandas as pd
# get all y.txt files from all subdirectories
all_files = glob.glob('/Users/Naga/Desktop/Python/Data/*/y.txt')
for file in all_files:
data_from_this_file = pd.read_csv(file, sep=" ", names = ['Name', 'Value'])
# do something with the data
Subsequently, you can apply your code to all the files within the list all_files. The great thing with glob is that you can use wilcards (*). Using them you don't need the names of the subdirectories (you can even use it within the filename, e.g. *y.txt). Also see the documentation on glob.
Your issue is forgot adding the parent path of 'y.txt' file. I suggest this code for you, hope it help.
import os
pth = '/Users/Naga/Desktop/Python/Data'
list_sub = os.listdir(pth)
filename = 'TextInformation.txt'
for sub in list_sub:
TextInfo = open('{}/{}/{}'.format(pth, sub, filename), 'r').read()
print(TextInfo)
I got you a little code. you can personalize it anyway you like but the code works for you.
import os
for dirPath,foldersInDir,fileName in os.walk(path_to_main_folder):
if fileName is not []:
for file in fileName:
if file.endswith('y.txt'):
loc = os.sep.join([dirPath,file])
y_txt = open(loc)
y = y_txt.read()
print(y)
But keep in mind that {path_to_main} is the path that has the subfolders.
I attended an interview and they asked me to write a script to move from one directory to another and delete only the .html files.
Now I tried to do this at first using os.remove() . Following is the code:
def rm_files():
import os
from os import path
folder='J:\\Test\\'
for files in os.listdir(folder):
file_path=path.join(folder,files)
os.remove(file_path)
The problem I am facing here is that I cannot figure out how to delete only .html files in my directory
Then I tried using glob. Following is the code:
def rm_files1():
import os
import glob
files=glob.glob('J:\\Test\\*.html')
for f in files:
os.remove(f)
Using glob I can delete the .html files but still I cannot figure out how to implement the logic of moving from one directory to another.
And along with that can someone please help me figure out how to delete a specific file type using os.remove() ?
Thank you.
Either of these methods should work. For the first way, you could just string.endswith(suffix) like so:
def rm_files():
import os
from os import path
folder='J:\\Test\\'
for files in os.listdir(folder):
file_path=path.join(folder,files)
if file_path.endswith(".html"):
os.remove(file_path)
Or if you prefer glob, moving directories is fairly straightforward: os.chdir(path) like this:
def rm_files1():
import os
os.chdir('J:\\Test')
import glob
files=glob.glob('J:\\Test\\*.html')
for f in files:
os.remove(f)
Though it seems unnecessary since glob is taking an absolute path anyway.
Your problem can be described in the following steps.
move to specific directory. This can be done using os.chdir()
grab list of all *.html files. Use glob.glob('*.html')
remove the files. use os.remove()
Putting it all together:
import os
import glob
import sys
def remove_html_files(path_name):
# move to desired path, if it exists
if os.path.exists(path_name):
os.chdir(path_name)
else:
print('invalid path')
sys.exit(1)
# grab list of all html files in current directory
file_list = glob.glob('*.html')
#delete files
for f in file_list:
os.remove(f)
#output messaage
print('deleted '+ str(len(file_list))+' files in folder' + path_name)
# call the function
remove_html_files(path_name)
To remove all html files in a directory with os.remove() you can do like this using endswith() function
import sys
import os
from os import listdir
directory = "J:\\Test\\"
test = os.listdir( directory )
for item in test:
if item.endswith(".html"):
os.remove( os.path.join( directory, item ) )
I have been working this challenge for about a day or so. I've looked at multiple questions and answers asked on SO and tried to 'MacGyver' the code used for my purpose, but still having issues.
I have a directory (lets call it "src\") with hundreds of files (.txt and .xml). Each .txt file has an associated .xml file (let's call it a pair). Example:
src\text-001.txt
src\text-001.xml
src\text-002.txt
src\text-002.xml
src\text-003.txt
src\text-003.xml
Here's an example of how I would like it to turn out so each pair of files are placed into a single unique folder:
src\text-001\text-001.txt
src\text-001\text-001.xml
src\text-002\text-002.txt
src\text-002\text-002.xml
src\text-003\text-003.txt
src\text-003\text-003.xml
What I'd like to do is create an associated folder for each pair and then move each pair of files into its respective folder using Python. I've already tried working from code I found (thanks to a post from Nov '12 by Sethdd, but am having trouble figuring out how to use the move function to grab pairs of files. Here's where I'm at:
import os
import shutil
srcpath = "PATH_TO_SOURCE"
srcfiles = os.listdir(srcpath)
destpath = "PATH_TO_DEST"
# grabs the name of the file before extension and uses as the dest folder name
destdirs = list(set([filename[0:9] for filename in srcfiles]))
def create(dirname, destpath):
full_path = os.path.join(destpath, dirname)
os.mkdir(full_path)
return full_path
def move(filename, dirpath):
shutil.move(os.path.join(srcpath, filename)
,dirpath)
# create destination directories and store their names along with full paths
targets = [
(folder, create(folder, destpath)) for folder in destdirs
]
for dirname, full_path in targets:
for filename in srcfile:
if dirname == filename[0:9]:
move(filename, full_path)
I feel like it should be easy, but Python isn't something I work with everyday and it's been a while since my scripting days... Any help would be greatly appreciated!
Thanks,
WK2EcoD
Use the glob module to interate all of the 'txt' files. From that you can parse and create the folders and copy the files.
The process should be as simple as it appears to you as a human.
for file_name in os.listdir(srcpath):
dir = file_name[:9]
# if dir doesn't exist, create it
# move file_name to dir
You're doing a lot of intermediate work that seems to be confusing you.
Also, insert some simple print statements to track data flow and execution flow. It appears that you have no tracing output so far.
You can do it with os module. For every file in directory check if associated folder exists, create if needed and then move the file. See the code below:
import os
SRC = 'path-to-src'
for fname in os.listdir(SRC):
filename, file_extension = os.path.splitext(fname)
if file_extension not in ['xml', 'txt']:
continue
folder_path = os.path.join(SRC, filename)
if not os.path.exists(folder_path):
os.mkdir(folderpath)
os.rename(
os.path.join(SRC, fname),
os.path.join(folder_path, fname)
)
My approach would be:
Find the pairs that I want to move (do nothing with files without a pair)
Create a directory for every pair
Move the pair to the directory
#! /usr/bin/env python
# -*- coding: utf-8 -*-
import os, shutil
import re
def getPairs(files):
pairs = []
file_re = re.compile(r'^(.*)\.(.*)$')
for f in files:
match = file_re.match(f)
if match:
(name, ext) = match.groups()
if ext == 'txt' and name + '.xml' in files:
pairs.append(name)
return pairs
def movePairsToDir(pairs):
for name in pairs:
os.mkdir(name)
shutil.move(name+'.txt', name)
shutil.move(name+'.xml', name)
files = os.listdir()
pairs = getPairs(files)
movePairsToDir(pairs)
NOTE: This script works when called inside the directory with the pairs.