Unexpected extra ghost value in dictionary python - python

I am writing a Python script that downloads files from an ftp location using the ftplib module. The files I download are specifically named whatsnew.txt and are located inside {ftp location}/{folder}/{subfolder} and there are many such folders and I download all whatsnew.txt located in them.
The way I do it is that before downloading, I gather what the directory tree of the ftp location looks like and store it as a dictionary (I also save this dictionary, formatted as a json file, in an output file named directory_tree.txt).
Now, when I parse this dictionary to get the values for {folder} and {subfolder}, there is an unexpected extra value in the dictionary, specifically the /318/08 as seen in the image. 318 is the folder and 08 is the subfolder. I have no idea where the hell did this ghost entry came from.
It is unexpected since this is what the folder in question looks like:
There is no frickin' 08 subfolder.
Here is a snippet of my code (Python 2.7.9) where I think I went wrong. How can I resolve my error? How do I fix this extra value? I hope there is no need for exorcism. Hopefully just exorcism of my programming insufficiency. Thanks.
for folder in directory_tree["ControlPatch_CPR"]: //Loop for checking dict entries
for subfolder in folder:
print str(folder) + " --- " + str(subfolder)
for folder in directory_tree["ControlPatch_CPR"]:
for subfolder in folder:
if len(subfolder) == 1:
subfolder = "0" + subfolder
ftp.cwd("/")
print "\nretrieving {0}/ControlPatch/{1}/{2}/whatsnew.txt ...".format(ftplocation, folder, subfolder)
with open("{0}.{1}.whatsnew.txt".format(folder, subfolder), "wb") as whatsnew:
ftp.cwd("/ControlPatch/{0}/{1}".format(folder, subfolder))
print "working directory: " + ftp.pwd()
ftp.retrbinary("RETR /ControlPatch/{0}/{1}/whatsnew.txt".format(folder, subfolder), whatsnew.write)
Also, I am scared.

I have fixed(?) my problem, though I can't explain it. From the code I posted above, I changed a part from:
for folder in directory_tree["ControlPatch_CPR"]:
for subfolder in folder:
if len(subfolder) == 1:
to:
for folder in directory_tree["ControlPatch_CPR"]:
for subfolder in directory_tree["ControlPatch_CPR"][folder]:
if len(subfolder) == 1:
This part directory_tree["ControlPatch_CPR"][folder] is a list thus I thought doing subfolder in folder just refers to folder as the same iterable. Anybody care to explain? Thanks.
Anyway, problem's solved. Whew. I can now return to my computer.

Related

Adding files from other directories to zip root directory with ZipFile

This seems like a fairly trivial question but I wasn't able to figure it out when I last worked on this project months ago, and I'm still not able to figure it out.
I have the following code
z = zipfile.ZipFile(make_file_safe(chapter_name) + '.cbz', 'a')
for file in os.listdir('tmp'):
z.write('tmp/' + file)
os.remove('tmp/' + file)
What I am trying to do is take all the files in the tmp directory and add them to the root zip directory (so it would look something like File.zip/1.jpg, but no matter what I do it always end up as File.zip/tmp/1.jpg. If I try to change any of the directories in z.write, it isn't able to find the file anymore.

shutil.move() creates duplicates and fails on subsequent calls

I finished writing a script which creates some files so I'm making a tidy() function which sorts these files in folders. The end result should look like this:
/Scripting
- Output
- script.py
/Scripting/Output
- Folder1
- Folder2
- Folder3
Each folder contains the necessary files
I managed to create the list of folders and get the files in them without any problem so I now have in /Project: script.py, folder1, folder2, etc... I copy pasted most of the code from the first part in order to move them into the Output folder. The following code is executed with every subfolder containing their respective files is located in the same directory as the script.
try:
os.mkdir('output')
except FileExistsError:
pass
for file in os.listdir():
if '.' not in file and file != 'output':
shutil.move(file, f'{os.getcwd()}/output/{file})
The problem is that if I look into my folder after running, I find the following directory tree:
/Output
- Folder1
- Folder1
- File1
- File2
I get a duplicate folder within that folder and I don't understand where it's coming from. If I try to call the script again, I get the error: shutil.Error destination path 'Scripting/output/folder1/fodler1' already exists
What am I doing wrong?
Edit:
Here's the new code:
try:
os.mkdir('output')
except FileExistsError:
pass
obj = os.scandir()
cwd = os.getcwd()
for entry in obj:
if entry.is_dir() and not entry.name.startswith('.'):
continue
shutil.move(entry.name, f'{cwd}/'/output/'{entry.name}')
This works the first time I run it, but breaks if I keep calling the script by giving me the same mistake as above. It creates folder1 within folder1 only on subsequent calls and I can't find a reason for it.
Found the answer mostly by trial and error. I initially chose to use shutil.move() because it replaces a file if it finds another one with the same name. However, it does not do this with directory. It will instead add to that path. /Scripting/Output/Folder1/ as a destination path for Folder1 would give an error when I run the script a second time so instead of replacing the folder, it simple adds it into its path which would then become /Scripting/Output/Folder1/Folder1/ while still adding the files to the initial path (it looks like it runs the shutil.move() on everything within that path). To fix this, use obj = os.scandir() with obj.is_dir() and obj.name to parse your folders. Either os.rmdir() the extra folder every time, or add the folders before adding the files. This is the code that worked for me:
cwd = os.getcwd()
try:
os.mkdir('output')
except:
pass
os.chdir('output')
for name in folder_names:
try:
os.mkdir(name)
except:
pass
os.chdir('..')
obj = os.scandir()
cwd = os.getcwd()
for f in obj:
if f.is_file():
if True:# depends on how your files are organized
shutil.move(f.name, f'{cwd}/output/folder1/{f.name}')
# Do this for every file

Lost files while tried to move files using python shutil.move

I had 120 files in my source folder which I need to move to a new directory (destination). The destination is made in the function I wrote, based on the string in the filename. For example, here is the function I used.
path ='/path/to/source'
dropbox='/path/to/dropbox'
files = = [os.path.join(path,i).split('/')[-1] for i in os.listdir(path) if i.startswith("SSE")]
sam_lis =list()
for sam in files:
sam_list =sam.split('_')[5]
sam_lis.append(sam_list)
sam_lis =pd.unique(sam_lis).tolist()
# Using the above list
ID = sam_lis
def filemover(ID,files,dropbox):
"""
Function to move files from the common place to the destination folder
"""
for samples in ID:
for fs in files:
if samples in fs:
desination = dropbox + "/"+ samples + "/raw/"
if not os.path.isdir(desination):
os.makedirs(desination)
for rawfiles in fnmatch.filter(files, pat="*"):
if samples in rawfiles:
shutil.move(os.path.join(path,rawfiles),
os.path.join(desination,rawfiles))
In the function, I am creating the destination folders, based on the ID's derived from the files list. When I tried to run this for the first time it threw me FILE NOT exists error.
However, later when I checked the source all files starting with SSE were missing. In the beginning, the files were there. I want some insights here;
Whether or not os.shutil.move moves the files to somewhere like a temp folder instead of destination folder?
whether or not the os.shutil.move deletes the files from the source in any circumstance?
Is there any way I can test my script to find the potential reasons for missing files?
Any help or suggestions are much appreciated?
It is late but people don't understand the op's question. If you move a file into a non-existing folder, the file seems to become a compressed binary and get lost forever. It has happened to me twice, once in git bash and the other time using shutil.move in Python. I remember the python happens when your shutil.move destination points to a folder instead of to a copy of the full file path.
For example, if you run the code below, a similar situation to what the op described will happen:
src_folder = r'C:/Users/name'
dst_folder = r'C:/Users/name/data_images'
file_names = glob.glob(r'C:/Users/name/*.jpg')
for file in file_names:
file_name = os.path.basename(file)
shutil.move(os.path.join(src_folder, file_name), dst_folder)
Note that dst_folder in the else block is just a folder. It should be dst_folder + file_name. This will cause what the Op described in his question. I find something similar on the link here with a more detailed explanation of what went wrong: File moving mistake with Python
shutil.move does not delete your files, if for any reason your files failed to move to a given location, check the directory where your code is stored, for a '+' folder your files are most likely stored there.

Python batch-rename script sending files to root folder

Ok, this is weird and maybe awkward.
I made a script so I could change the end of subtitles files to keep consistency.
Basically it replaces A.X.str to A.Y.str. It worked flawlessly at a single folder.
I decided then to make a recursive version of it so I could do it on any folder I had, regardless if the episodes where together, separated by season or each on an individual path.
I really don't know how or why, but it sent all the files it reached to the root folder I was using until it halted raising a FileExistsError.
The code bit I'm using is:
def rewrite(folder, old, new):
for f in next(os.walk(folder))[2]:
os.rename(os.path.join(folder, f),
os.path.join(path, f.replace(old, new)))
for f in next(os.walk(folder))[1]:
x = os.path.join(folder, f)
rewrite(x, old, new)
Where 'old' is "A.X.str", 'new' is "A.Y.str" and folder is the full path of the root folder "C:\Series\Serie Name".
Why doesn't this work as recursive? The first bit of code (First FOR loop) works fine on it's own in a single folder.
Is the problem with the "next" I use to get the names of files and folders?
The code you are showing us is using a path variable in the rename destination -- that should be the folder variable instead.

Error [183] when using python os.rename

This is my first time using python and I keep running into error 183. The script I created searches the network for all '.py' files and copies them to my backup drive. Please don't laugh at my script as this is my first.
Any clue to what I am doing wrong in the script?
import os
import shutil
import datetime
today = datetime.date.today()
rundate = today.strftime("%Y%m%d")
for root,dirr,filename in os.walk("p:\\"):
for files in filename:
if files.endswith(".py"):
sDir = os.path.join(root, files)
dDir = "B:\\Scripts\\20120124"
modname = rundate + '_' + files
shutil.copy(sDir, dDir)
os.rename(os.path.join(dDir, files), os.path.join(dDir, modname))
print "Renamed %s to %s in %s" % (files, modname, dDir)
I'm guessing you are running the script on windows. According to the list of windows error codes error 183 is ERROR_ALREADY_EXISTS
So I would guess the script is failing because you're attempting to rename a file over an existing file.
Perhaps you are running the script more than once per day? That would result in all the destination files already being there, so the rename is failing when the script is run additional times.
If you specifically want to overwrite the files, then you should probably delete them using os.unlink first.
Given the fact that error 183 is [Error 183] Cannot create a file when that file already exists, you're most likely finding 2 files with the same name in the os.walk() call and after the first one is renamed successfully, the second one will fail to be renamed to the same name so you'll get a file already exists error.
I suggest a try/except around the os.rename() call to treat this case (append a digit after the name or something).
[Yes, i know it's been 7 years since this question was asked but if I got here from a google search maybe others are reaching it too and this answer might help.]
I just encounter the same issue, when you trying to rename a folder with a folder that existed in the same directory has the same name, Python will raise an error.
If you trying to do that in Windows Explorer, it will ask you if you want to merge these two folders. however, Python doesn't have this feature.
Below is my codes to achieve the goal of rename a folder while a same name folder already exist, which is actually merging folders.
import os, shutil
DEST = 'D:/dest/'
SRC = 'D:/src/'
for filename in os.listdir(SRC): # move files from SRC to DEST folder.
try:
shutil.move(SRC + filename, DEST)
# In case the file you're trying to move already has a copy in DEST folder.
except shutil.Error: # shutil.Error: Destination path 'D:/DEST/xxxx.xxx' already exists
pass
# Now delete the SRC folder.
# To delete a folder, you have to empty its files first.
if os.path.exists(SRC):
for i in os.listdir(SRC):
os.remove(os.path.join(SRC, i))
# delete the empty folder
os.rmdir(SRC)

Categories