Trying to move files from logfile_directory to tmp_directory in blocks of 7 using random.sample.
If there are less than 7 files in the folder it will just move the remainder of the files. However when I try and move less that 7 files I get an error as the tmp_folder is trying to copy into itself.
Tried using glob.glob command but can't get that to work either. Not sure what I'm doing wrong to just move the files and not folder. Any help would be appreciated.
Running the same code on a different machine and getting the below message where as before the error message was related to copying the tmp_folder into itself. Nothing special about this file it's creating the error on so don't know why I'm now getting this.
Message=[WinError 5] Access is denied: 'c:\securelog_test\bdlog.txt'
Source=C:\Users\jarra\source\repos\archive_test\archive_test.py
StackTrace:
File "\archive_test.py", line 72, in
shutil.move(path, tmp_folder)
logfile_directory = 'c:\\securelog_test\\'
archive_folder = 'c:\\securelog_archive\\'
workfiles_folder = 'c:\\securelog_workfiles\\'
tmp_folder = 'c:\\securelog_test\\temp\\'
completed_folder = 'c:\\securelog_test\\completed\\'
#count how many files are in the log file folder
onlyfiles = [f for f in listdir(logfile_directory) if isfile(join(logfile_directory, f))]
print('-----------------')
print (len(onlyfiles))
if len(onlyfiles) > 7:
#move 7 random files to the temp folder for archiving
files = os.listdir(logfile_directory)
for fileName in random.sample(files, min(len(files), 8)):
path = os.path.join(logfile_directory, fileName)
shutil.move(path, tmp_folder)
else:
#if there are less than 7 files move them
#for file in glob.glob(logfile_directory):
# shutil.move(file, tmp_folder)
for fileName in os.listdir(logfile_directory):
path = os.path.join(logfile_directory, fileName)
shutil.move(path, tmp_folder)
Well, seems that file existed in the destination folder and couldn't write over it. Not sure why so will look into that eventually.
In order to make my life a bit easier I put the temp and completed folders into their own root folders like below and this seems to work with out any issues. Feel this is cheating a bit and would love to try and figure this out but I need to move onto other elements of the program.
logfile_directory = 'c:\\securelog\\securelog_test\\'
archive_folder = 'c:\\securelog\\securelog_archive\\'
workfiles_folder = 'c:\\securelog\\securelog_workfiles\\'
tmp_folder = 'c:\\securelog\\temp\\'
completed_folder = 'c:\\securelog\\completed\\'
check the os current directory path try to print it for each iteration you get where its getting wrong.
Related
Attempting to write a function that walks a file system and returns the absolute path and filename for use in another function.
Example "/testdir/folderA/222/filename.ext".
Having tried multiple versions of this I cannot seem to get it to work properly.
filesCheck=[]
def findFiles(filepath):
files=[]
for root, dirs, files in os.walk(filepath):
for file in files:
currentFile = os.path.realpath(file)
print (currentFile)
if os.path.exists(currentFile):
files.append(currentFile)
return files
filesCheck = findFiles(/testdir)
This returns
"filename.ext" (only one).
Substitute in currentFile = os.path.join(root, file) for os.path.realpath(file) and it goes into a loop in the first directory. Tried os.path.join(dir, file) and it fails as one of my folders is named 222.
I have gone round in circles and get somewhat close but haven't been able to get it to work.
Running on Linux with Python 3.6
There's a several things wrong with your code.
There are multiple values are being assigned to the variable name files.
You're not adding the root directory to each filename os.walk() returns which can be done with os.path.join().
You're not passing a string to the findFiles() function.
If you fix those things there's no longer a need to call os.path.exists() because you can be sure it does.
Here's a working version:
import os
def findFiles(filepath):
found = []
for root, dirs, files in os.walk(filepath):
for file in files:
currentFile = os.path.realpath(os.path.join(root, file))
found.append(currentFile)
return found
filesCheck = findFiles('/testdir')
print(filesCheck)
Hi I think this is what you need. Perhaps you could give it a try :)
from os import walk
path = "C:/Users/SK/Desktop/New folder"
files = []
for (directoryPath, directoryNames, allFiles) in walk(path):
for file in allFiles:
files.append([file, f"{directoryPath}/{file}"])
print(files)
Output:
[ ['index.html', 'C:/Users/SK/Desktop/New folder/index.html'], ['test.py', 'C:/Users/SK/Desktop/New folder/test.py'] ]
Hi everyone I´m trying now to move some files from one folder to another. The files that i have are
results_1_1
results_2_1
results_3_1
results_1_2
results_2_2
results_3_2
What i´d like to get is to move files
results_1_1
results_2_1
results_3_1 to one folder let´s call it A
other
results_1_2
results_2_2
results_3_2 to another let´s call it B
I don´t want to do it manually because, in reality, I have much more files
I have created by python a new folder that I want to have
and so far about moving the files to another directory
import shutil
source_folder = r"C:/Users/...."
destination_folder= r"C:/Users/...."
files_to_move = ['results_1_3.dat']
for file in files_to_move:
source = source_folder + file
destination = destination_folder + file
shutil.move(source,destination)
print("Moved!")
But with this program I can only move one file at a time I tried writing results_.+3.dat', results*_3.dat' but I´m seeing an error all the time
You can use os.listdir to find all the files in your directory.
This code finds all the files which names end with '1'.
import os
files_to_move = [f for f in os.listdir() if f.split('.')[0][-1] == '1']
You can also add an extension check if necessary:
files_to_move = [f for f in os.listdir() if f.split('.')[0][-1] == '1' and f.split('.')[1] == 'dat']
You can list all files in your directory using os.listdir(source_folder)
If you have mixed files in your directory, you can select your results files with something like this
files_to_move = [x for x in os.listdir(source_folder) if all(j in x for j in ['results', '.dat'])]
I would then create a dictionary containing the destination folders associated with the numbers in your results file.
destination_A = r'/path/to/destinationA'
destination_B = r'/path/to/destinationB'
moving_dict = {
'1': destination_A,
'2': destination_B
}
You can then move your files based on the filename
for file in files_to_move:
destination_folder = moving_dict[file[:-4].split('_')[-1]]
source = source_folder + file
destination = destination_folder + file
shutil.move(source,destination)
I see many answers on here to similar questions but I cannot seem to adapt it quite yet due to my budding Python skill. I'd like to save the time of individually grabbing the data sets that contain what I need for analysis in R, but my scripts either don't run or seem to do what I need.
I need to 1) loop through a sea of subfolders in a parent folder, 2) loop through the bagillion .csv files in those subs and pick out the 1 that matters (matching text below) and 3) copy it over to a new clean folder with only what I want.
What I have tried:
1)
import os, shutil, glob
src_fldr = 'C:/project_folder_withsubfolders';
dst_fldr = 'C:/project_folder_withsubfolders/subfolder_to_dump_into';
try:
os.makedirs(dst_fldr); ## it creates the destination folder
except:
print ("Folder already exist or some error");
for csv_file in glob.glob(src_fldr+'*statistics_Intensity_Sum_Ch=3_Img=1.csv*'):
shutil.copy2(csv_file, dst_fldr);
where the text statistics_Intensity_Sum etc is the exact pattern I need for the file to copy over
this didn't actually copy anything over
Making a function that will do this:
srcDir = 'C:/project_folder_withsubfolders'
dstDir = 'C:/project_folder_withsubfolders/subfolder_to_dump_into'
def moveAllFilesinDir(srcDir, dstDir):
files = os.listdir(srcDir)
for f in files:
if f.find("statistics_Intensity_Sum_Ch=3_Img=1"):
shutil.move(f, dstDir)
else:
shutil.move(f, srcDir)
moveAlllFilesinDir(srcDir, dstDir)
This returned the following error:
File "C:\Users\jbla12\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 806, in move
os.rename(src, real_dst)
FileNotFoundError: [WinError 2] The system cannot find the file specified: 'F1 converted' -> 'C:/Users/jbla12/Desktop/R Analyses/p65_project/sum_files\\F1 converted'
That's because that's a sub-folder I want it to go through! I've tried other methods but don't have record of them in my scripts.
SOLVED:
Special thanks to "Automate the Boring Stuff"
import shutil
import os
dest = 'C:/Users/jbla12/Desktop/R Analyses/p65_project/sum_files/'
src = 'C:/Users/jbla12/Desktop/R Analyses/p65_project/'
txt_ID = 'statistics_Intensity_Sum_Ch=3_Img=1.csv'
def moveSpecFiles(txt_ID, src, dest):
#src is the original file(s) destination
#dest is the destination for the files to end up in
#spec_txt is what the files end with that you want to ID
for foldername, subfolders, filenames in os.walk(src):
for file in filenames:
if file.endswith(txt_ID):
shutil.copy(os.path.join(foldername, file), dest)
print('Your files are ready sir/madam!')
moveSpecFiles(txt_ID, src, dest)
I've created a simple script to rename my media files that have lots of weird periods and stuff in them that I have obtained and want to organize further. My script kinda works, and I will be editing it to edit the filenames further but my os.rename line throws this error:
[Windows Error: Error 2: The system cannot find the file specified.]
import os
for filename in os.listdir(directory):
fcount = filename.count('.') - 1 #to keep the period for the file extension
newname = filename.replace('.', ' ', fcount)
os.rename(filename, newname)
Does anyone know why this might be? I have a feeling that it doesn't like me trying to rename the file without including the file path?
try
os.rename(filename, directory + '/' + newname);
Triton Man has already answered your question. If his answer doesn't work I would try using absolute paths instead of relative paths.
I've done something similar before, but in order to keep any name clashes from happening I temporarily moved all the files to a subfolder. The entire process happened so fast that in Windows Explorer I never saw the subfolder get created.
Anyhow if you're interested in looking at my script It's shown below. You run the script on the command line and you should pass in as a command-line argument the directory of the jpg files you want renamed.
Here's a script I used to rename .jpg files to multiples of 10. It might be useful to look at.
'''renames pictures to multiples of ten'''
import sys, os
debug=False
try:
path = sys.argv[1]
except IndexError:
path = os.getcwd()
def toint(string):
'''changes a string to a numerical representation
string must only characters with an ordianal value between 0 and 899'''
string = str(string)
ret=''
for i in string:
ret += str(ord(i)+100) #we add 101 to make all the numbers 3 digits making it easy to seperate the numbers back out when we need to undo this operation
assert len(ret) == 3 * len(string), 'recieved an invalid character. Characters must have a ordinal value between 0-899'
return int(ret)
def compare_key(file):
file = file.lower().replace('.jpg', '').replace('dscf', '')
try:
return int(file)
except ValueError:
return toint(file)
#files are temporarily placed in a folder
#to prevent clashing filenames
i = 0
files = os.listdir(path)
files = (f for f in files if f.lower().endswith('.jpg'))
files = sorted(files, key=compare_key)
for file in files:
i += 10
if debug: print('renaming %s to %s.jpg' % (file, i))
os.renames(file, 'renaming/%s.jpg' % i)
for root, __, files in os.walk(path + '/renaming'):
for file in files:
if debug: print('moving %s to %s' % (root+'/'+file, path+'/'+file))
os.renames(root+'/'+file, path+'/'+file)
Edit: I got rid of all the jpg fluff. You could use this code to rename your files. Just change the rename_file function to get rid of the extra dots. I haven't tested this code so there is a possibility that it might not work.
import sys, os
path = sys.argv[1]
def rename_file(file):
return file
#files are temporarily placed in a folder
#to prevent clashing filenames
files = os.listdir(path)
for file in files:
os.renames(file, 'renaming/' + rename_file(file))
for root, __, files in os.walk(path + '/renaming'):
for file in files:
os.renames(root+'/'+file, path+'/'+file)
Looks like I just needed to set the default directory and it worked just fine.
folder = r"blah\blah\blah"
os.chdir(folder)
for filename in os.listdir(folder):
fcount = filename.count('.') - 1
newname = filename.replace('.', ' ', fcount)
os.rename(filename, newname)
I have written a script to move video files from one directory to another, it will also search sub directories using os.walk. however if the script finds a video file it will only move the file and not the containing folder. i have added an if statement to check if the containing folder is different to the original search folder.
i cant find the code to actually move(or rename?) the folder and file to a different directory. i have read/watch a lot on moving files and there is a lot of information on that, but i cant find anything for moving folders.
i have tried using shutil.move and os.rename and i get an error both times. when i try and search for the problem i get a lot of results about how to move files, or how to change the current working directory of python.
any advice(even how to phrase the google search to accuratly describe how to find a tutorial on the subject) would be really appreciated. it's my first real world python program and ive learnt a lot but this last step is wearing me down!
EDIT: when trying to use os.rename(src_file, dst_file) i get the error WindowsError: error 3 The system cannot find the path specified.
when trying shutil.move(src_file, dst_file) i get ioerror errno 2 no such file or directory "H:\\Moviesfrom download...\OneOfTheVideoFilesNotInParentFolder ie the folder and the file needs to move.
thanks.
ps like i said it's my first script outside of code academy so any random suggestions would also be appreciated.
import os
import shutil
import time
movietypes = ('.3gp', '.wmv', '.asf', '.avi', '.flv', '.mov', '.mp4', '.ogm', '.mkv',
'. mpg', '.mpg', '.nsc', '.nsv', '.nut', '.a52', '.tta', '.wav', '.ram', '.asf',
'.wmv', '. ogg', '.mka', '.vid', '.lac', '.aac', '.dts', '.tac',
'.dts', '.mbv')
filewrite = open('H:\\Movies from download folder\\Logs\\logstest.txt', 'w')
dir_src = "C:\\Users\\Jeremy\\Downloads\\"
dir_dst = "H:\\Movies from download folder\\"
for root, dirs, files in os.walk(dir_src):
for file in files:
if file.endswith(movietypes) == True:
filestr = str(file)
locationoffoundfile = os.path.realpath(os.path.join(root,filestr))
folderitwasin = locationoffoundfile.replace(dir_src,'')
folderitwasin = folderitwasin.replace(filestr,'')
pathofdir = os.path.realpath(root) + "\\"
if pathofdir != dir_src:
src_file = locationoffoundfile
dst_file = dir_dst + folderitwasin + filestr
os.rename(src_file, dst_file) #****This line is the line im having issues with***
print src_file
print dst_file
filewrite.write(file + " " + "needs to have dir and file moved Moved!" + '\n')
else:
src_file = os.path.join(dir_src, file)
dst_file = os.path.join(dir_dst, file)
print src_file
print dst_file
shutil.move(src_file, dst_file)
filewrite.write(os.path.dirname(file) + '\n')
filewrite.write(file + " " + "needs to have file moved Moved!" + '\n')
filewrite.close()
Looks like you're only moving files, without doing anything about the folders. So if you try to move
C:\Users\Jeremy\Downloads\anime\pokemon.avi
to
H:\Movies from download folder\anime\pokemon.avi
it will fail because there's no anime directory on H:\ yet.
Before iterating through files, iterate through dirs to ensure that the directory exists at your destination, creating it if necessary.
for root, dirs, files in os.walk(dir_src):
for dir in dirs:
dest_dir = os.path.join(dir_dst, dir)
if not os.path.isdir(dest_dir):
os.mkdir(dest_dir)
for file in files:
#rest of code goes here as usual...
As these are MS Windows paths use forward slashes instead and declare path as a string literal; e.g.
dir_dst = r"H:/Movies from download folder/"