We Keep Coding

How do I copy subfolders into another location

I'm making a program to back up files and folders to a destination.
The problem I'm currently facing is if I have a folder inside a folder and so on, with files in between them, I can't Sync them at the destination.
e.g.:
The source contains folder 1 and file 2. Folder 1 contains folder 2, folder 2 contains folder 3 and files etc...
The backup only contains folder 1 and file 2.
If the backup doesn't exist I simply use: shutil.copytree(path, path_backup), but in the case, I need to sync I can't get the files and folders or at least I'm not seeing a way to do it. I have walked the directory with for path, dir, files in os.walk(directory) and even used what someone suggest in another post:
def walk_folder(target_path, path_backup):
for files in os.scandir(target_path):
if os.path.isfile(files):
file_name = os.path.abspath(files)
print(file_name)
os.makedirs(path_backup)
elif os.path.isdir(files):
walk_folder(files, path_backup)
Is there a way to make the directories in the backup folder from the ground up and then add the info alongside or is the only way to just delete the whole folder and use shutil.copytree(path, path_backup).
With makedirs, all it does is say it can't create because the folder already exists, this is understandable as it's trying to write in the Source folder and not in the backup. Is there a way to make the path to replace Source for backup?
If any more code is needed feel free to ask!

shutil.move() creates duplicates and fails on subsequent calls

I finished writing a script which creates some files so I'm making a tidy() function which sorts these files in folders. The end result should look like this:
/Scripting
- Output
- script.py
/Scripting/Output
- Folder1
- Folder2
- Folder3
Each folder contains the necessary files
I managed to create the list of folders and get the files in them without any problem so I now have in /Project: script.py, folder1, folder2, etc... I copy pasted most of the code from the first part in order to move them into the Output folder. The following code is executed with every subfolder containing their respective files is located in the same directory as the script.
try:
os.mkdir('output')
except FileExistsError:
pass
for file in os.listdir():
if '.' not in file and file != 'output':
shutil.move(file, f'{os.getcwd()}/output/{file})
The problem is that if I look into my folder after running, I find the following directory tree:
/Output
- Folder1
- Folder1
- File1
- File2
I get a duplicate folder within that folder and I don't understand where it's coming from. If I try to call the script again, I get the error: shutil.Error destination path 'Scripting/output/folder1/fodler1' already exists
What am I doing wrong?
Edit:
Here's the new code:
try:
os.mkdir('output')
except FileExistsError:
pass
obj = os.scandir()
cwd = os.getcwd()
for entry in obj:
if entry.is_dir() and not entry.name.startswith('.'):
continue
shutil.move(entry.name, f'{cwd}/'/output/'{entry.name}')
This works the first time I run it, but breaks if I keep calling the script by giving me the same mistake as above. It creates folder1 within folder1 only on subsequent calls and I can't find a reason for it.

Found the answer mostly by trial and error. I initially chose to use shutil.move() because it replaces a file if it finds another one with the same name. However, it does not do this with directory. It will instead add to that path. /Scripting/Output/Folder1/ as a destination path for Folder1 would give an error when I run the script a second time so instead of replacing the folder, it simple adds it into its path which would then become /Scripting/Output/Folder1/Folder1/ while still adding the files to the initial path (it looks like it runs the shutil.move() on everything within that path). To fix this, use obj = os.scandir() with obj.is_dir() and obj.name to parse your folders. Either os.rmdir() the extra folder every time, or add the folders before adding the files. This is the code that worked for me:
cwd = os.getcwd()
try:
os.mkdir('output')
except:
pass
os.chdir('output')
for name in folder_names:
try:
os.mkdir(name)
except:
pass
os.chdir('..')
obj = os.scandir()
cwd = os.getcwd()
for f in obj:
if f.is_file():
if True:# depends on how your files are organized
shutil.move(f.name, f'{cwd}/output/folder1/{f.name}')
# Do this for every file

Lost files while tried to move files using python shutil.move

I had 120 files in my source folder which I need to move to a new directory (destination). The destination is made in the function I wrote, based on the string in the filename. For example, here is the function I used.
path ='/path/to/source'
dropbox='/path/to/dropbox'
files = = [os.path.join(path,i).split('/')[-1] for i in os.listdir(path) if i.startswith("SSE")]
sam_lis =list()
for sam in files:
sam_list =sam.split('_')[5]
sam_lis.append(sam_list)
sam_lis =pd.unique(sam_lis).tolist()
# Using the above list
ID = sam_lis
def filemover(ID,files,dropbox):
"""
Function to move files from the common place to the destination folder
"""
for samples in ID:
for fs in files:
if samples in fs:
desination = dropbox + "/"+ samples + "/raw/"
if not os.path.isdir(desination):
os.makedirs(desination)
for rawfiles in fnmatch.filter(files, pat="*"):
if samples in rawfiles:
shutil.move(os.path.join(path,rawfiles),
os.path.join(desination,rawfiles))
In the function, I am creating the destination folders, based on the ID's derived from the files list. When I tried to run this for the first time it threw me FILE NOT exists error.
However, later when I checked the source all files starting with SSE were missing. In the beginning, the files were there. I want some insights here;
Whether or not os.shutil.move moves the files to somewhere like a temp folder instead of destination folder?
whether or not the os.shutil.move deletes the files from the source in any circumstance?
Is there any way I can test my script to find the potential reasons for missing files?
Any help or suggestions are much appreciated?

It is late but people don't understand the op's question. If you move a file into a non-existing folder, the file seems to become a compressed binary and get lost forever. It has happened to me twice, once in git bash and the other time using shutil.move in Python. I remember the python happens when your shutil.move destination points to a folder instead of to a copy of the full file path.
For example, if you run the code below, a similar situation to what the op described will happen:
src_folder = r'C:/Users/name'
dst_folder = r'C:/Users/name/data_images'
file_names = glob.glob(r'C:/Users/name/*.jpg')
for file in file_names:
file_name = os.path.basename(file)
shutil.move(os.path.join(src_folder, file_name), dst_folder)
Note that dst_folder in the else block is just a folder. It should be dst_folder + file_name. This will cause what the Op described in his question. I find something similar on the link here with a more detailed explanation of what went wrong: File moving mistake with Python

shutil.move does not delete your files, if for any reason your files failed to move to a given location, check the directory where your code is stored, for a '+' folder your files are most likely stored there.

how to `shutil.move` with readonly files across drives

At least on windows, shutil.move a folder containing readonly files to another drive will fail. It fails because move is implemented with a copy followed by a rmtree. In the end, it's the rmtree trying to delete non writable files.
Currently I work around it by first setting the stat.S_IWUSER for all (nested) files, but now I should still restore the original stat afterwards:
def make_tree_writable(source_dir):
for root, dirs, files in os.walk(source_dir):
for name in files:
make_writable(path.join(root, name))
def make_writable(path_):
os.chmod(path_, stat.S_IWUSR)
def movetree_workaround(source_dir, target_dir):
make_tree_writable(source_dir)
shutil.move(source_dir, target_dir)
So I wonder: is this the way? Is there a shutil2 in the making that I could use? Can I be of any help there?

You can do that in two steps: first, use shutil.copytree() to copy the full directory and file structure with appropriate permissions. Then you can change permissions of the source to make sure you have rights to delete stuff, and use shutil.rmtree() to remove the old source.

Error [183] when using python os.rename

This is my first time using python and I keep running into error 183. The script I created searches the network for all '.py' files and copies them to my backup drive. Please don't laugh at my script as this is my first.
Any clue to what I am doing wrong in the script?
import os
import shutil
import datetime
today = datetime.date.today()
rundate = today.strftime("%Y%m%d")
for root,dirr,filename in os.walk("p:\\"):
for files in filename:
if files.endswith(".py"):
sDir = os.path.join(root, files)
dDir = "B:\\Scripts\\20120124"
modname = rundate + '_' + files
shutil.copy(sDir, dDir)
os.rename(os.path.join(dDir, files), os.path.join(dDir, modname))
print "Renamed %s to %s in %s" % (files, modname, dDir)

I'm guessing you are running the script on windows. According to the list of windows error codes error 183 is ERROR_ALREADY_EXISTS
So I would guess the script is failing because you're attempting to rename a file over an existing file.
Perhaps you are running the script more than once per day? That would result in all the destination files already being there, so the rename is failing when the script is run additional times.
If you specifically want to overwrite the files, then you should probably delete them using os.unlink first.

Given the fact that error 183 is [Error 183] Cannot create a file when that file already exists, you're most likely finding 2 files with the same name in the os.walk() call and after the first one is renamed successfully, the second one will fail to be renamed to the same name so you'll get a file already exists error.
I suggest a try/except around the os.rename() call to treat this case (append a digit after the name or something).
[Yes, i know it's been 7 years since this question was asked but if I got here from a google search maybe others are reaching it too and this answer might help.]

I just encounter the same issue, when you trying to rename a folder with a folder that existed in the same directory has the same name, Python will raise an error.
If you trying to do that in Windows Explorer, it will ask you if you want to merge these two folders. however, Python doesn't have this feature.
Below is my codes to achieve the goal of rename a folder while a same name folder already exist, which is actually merging folders.
import os, shutil
DEST = 'D:/dest/'
SRC = 'D:/src/'
for filename in os.listdir(SRC): # move files from SRC to DEST folder.
try:
shutil.move(SRC + filename, DEST)
# In case the file you're trying to move already has a copy in DEST folder.
except shutil.Error: # shutil.Error: Destination path 'D:/DEST/xxxx.xxx' already exists
pass
# Now delete the SRC folder.
# To delete a folder, you have to empty its files first.
if os.path.exists(SRC):
for i in os.listdir(SRC):
os.remove(os.path.join(SRC, i))
# delete the empty folder
os.rmdir(SRC)