I'm making a program to back up files and folders to a destination.
The problem I'm currently facing is if I have a folder inside a folder and so on, with files in between them, I can't Sync them at the destination.
e.g.:
The source contains folder 1 and file 2. Folder 1 contains folder 2, folder 2 contains folder 3 and files etc...
The backup only contains folder 1 and file 2.
If the backup doesn't exist I simply use: shutil.copytree(path, path_backup), but in the case, I need to sync I can't get the files and folders or at least I'm not seeing a way to do it. I have walked the directory with for path, dir, files in os.walk(directory) and even used what someone suggest in another post:
def walk_folder(target_path, path_backup):
for files in os.scandir(target_path):
if os.path.isfile(files):
file_name = os.path.abspath(files)
print(file_name)
os.makedirs(path_backup)
elif os.path.isdir(files):
walk_folder(files, path_backup)
Is there a way to make the directories in the backup folder from the ground up and then add the info alongside or is the only way to just delete the whole folder and use shutil.copytree(path, path_backup).
With makedirs, all it does is say it can't create because the folder already exists, this is understandable as it's trying to write in the Source folder and not in the backup. Is there a way to make the path to replace Source for backup?
If any more code is needed feel free to ask!
Related
I have a folder structure where I have a few txt files in a few folders, and then I have to copy all the folders where I found txt files to another folder. My root folder is always different, so to find it I use os.path.dirname(os.path.abspath(file)) and search all the folders from there. Unfortunately I'm stuck here, because I never use any file handling in python.
From your question I'm assuming you have certain folders at a specific location and the folders contain specific txt files. Now you want to move all the folders containing text files to a specific location.
You can do this:
Walk through all the folders in the master_folder.
Walk through files in the sub-folders, and if any file has a .txt extension, it moves the folder to the target location:
import os
import shutil
for dirpath, dirs, files in os.walk('path_to_master_folder'):
for filename in files:
if filename.endswith('txt'):
shutil.move(dirpath, 'path_to_destination_folder')
I have 3 Folders
Folder A (Landing Folder)
Folder B (Deletion Folder)
Folder C (Transfer to AWS Folder)
I've written some python code to move files to the landing folder after some data transformation, from the landing folder - i need to make a decision tree for the files:
Check each filename in folder A to see if it exists in the database
if the file exists already:
move the file to Folder B
Else
Insert the file name into the Database & move the file to Folder C for transfer to AWS
I have the SQL statement inside python tested and working, but im not sure on how to move the files to their respective folders.
I'm assuming that i need something along the lines of using shutil.move to Folder B or C depending on outcome, but not 100% sure on the syntax to get there.
Any help appreciated.
You can use os.listdir to get a list of the files in a given path, and as you said, you can use shutil.move() to move the files. So, you could try something like this:
import os
import shutil
folderA='pathfolderA'
folderB='pathfolderB'
folderC='pathfolderC'
files=os.listdir(folderA)
for fil in files:
if fil in database:
shutil.move(fil,folderB)
else:
# insert(fil) into database
shutil.move(fil,folderC)
I'm ignoring the explicity of steps if fil in database and insert(fil) into database, because you said you have the SQL statement successfully tested. You can chekout the this helpful info about shutil and os.listdir: link 1:shutil, link 2:shutil, link 3:shutil, link 4:os.listdir.
I had 120 files in my source folder which I need to move to a new directory (destination). The destination is made in the function I wrote, based on the string in the filename. For example, here is the function I used.
path ='/path/to/source'
dropbox='/path/to/dropbox'
files = = [os.path.join(path,i).split('/')[-1] for i in os.listdir(path) if i.startswith("SSE")]
sam_lis =list()
for sam in files:
sam_list =sam.split('_')[5]
sam_lis.append(sam_list)
sam_lis =pd.unique(sam_lis).tolist()
# Using the above list
ID = sam_lis
def filemover(ID,files,dropbox):
"""
Function to move files from the common place to the destination folder
"""
for samples in ID:
for fs in files:
if samples in fs:
desination = dropbox + "/"+ samples + "/raw/"
if not os.path.isdir(desination):
os.makedirs(desination)
for rawfiles in fnmatch.filter(files, pat="*"):
if samples in rawfiles:
shutil.move(os.path.join(path,rawfiles),
os.path.join(desination,rawfiles))
In the function, I am creating the destination folders, based on the ID's derived from the files list. When I tried to run this for the first time it threw me FILE NOT exists error.
However, later when I checked the source all files starting with SSE were missing. In the beginning, the files were there. I want some insights here;
Whether or not os.shutil.move moves the files to somewhere like a temp folder instead of destination folder?
whether or not the os.shutil.move deletes the files from the source in any circumstance?
Is there any way I can test my script to find the potential reasons for missing files?
Any help or suggestions are much appreciated?
It is late but people don't understand the op's question. If you move a file into a non-existing folder, the file seems to become a compressed binary and get lost forever. It has happened to me twice, once in git bash and the other time using shutil.move in Python. I remember the python happens when your shutil.move destination points to a folder instead of to a copy of the full file path.
For example, if you run the code below, a similar situation to what the op described will happen:
src_folder = r'C:/Users/name'
dst_folder = r'C:/Users/name/data_images'
file_names = glob.glob(r'C:/Users/name/*.jpg')
for file in file_names:
file_name = os.path.basename(file)
shutil.move(os.path.join(src_folder, file_name), dst_folder)
Note that dst_folder in the else block is just a folder. It should be dst_folder + file_name. This will cause what the Op described in his question. I find something similar on the link here with a more detailed explanation of what went wrong: File moving mistake with Python
shutil.move does not delete your files, if for any reason your files failed to move to a given location, check the directory where your code is stored, for a '+' folder your files are most likely stored there.
I use os.renmae to rename files and move them about, but i am failing at doing the following task.
I have a main folder containing sub-folders with the structure below.
Main folder "Back", containing sub-folders named with letters and numbers e.g. A01, A02, B01, B02, etc.. inside each of those folders is a set of files, amongst them is a file called "rad" so a file path example looks something like this:
Back/A01/rad
/A02/rad
/B01/rad
.../rad
I have another sub-folder called "rads" inside the main "Back"
Back/rads
What i want to do is copy all the rad files only, from each of the folders in back and move them to the folder "rads" and name each rad file based on the folder it came from.
e.g. rad_A01, rad_A02, rad_B01, etc...
I couldnt really figure out how to increase the folder number when i move the files.
os.rename ("Back//rad, Back/rads/rad_")
I thought of making a list of all the names of the files and then do something like from x in list, do os.rename (), but i didnt know how to tell python to name the file according to the subfolder it came from as they are not a continuous series..
Any help is appreciated.
Thanks in advance
import os
for subdir, dirs, files in os.walk('./Back/'):
for file in files:
filepath = subdir+os.sep+file
if filepath.endswith("rad.txt"):
par_dir = os.path.split(os.path.dirname(filepath))[1]
os.system('cp '+filepath+' ./Back/rads/rad_'+par_dir)
save this python file beside Back directory and it should work fine!
This code iterates over each files in each subdirectory of Back, checks all files with name rad.txt, appends name of parent directory and copy to the rads folder.
Note: I saved rad files with .txt extension, change it accordingly.
I have a some directories that contain some other directories which, at the lowest level, contain bunch of csv files such as (folder) a -> b -> c -> (csv files). There is usually only one folder at each level. When I process a directory how can I follow this structure until the end to get the csv files ? I was thinking maybe a recursive solution but I think there may be better ways to do this. I am using python. Hope I was clear.
The os package has a walk function that will do exactly what you need:
for current_path, directory, files in walk("/some/path"):
# current_path is the full path of the directory we are currently in
# directory is the name of the directory
# files is a list of file names in this directory
You can use os.path's to derive the full path to each file (if you need it).
Alternately, you might find the glob module to be of more use to you:
for csv_file in glob(/some/path/*/*.csv"):
# csv_file is the full path to the csv file.