We Keep Coding

How to wait for a folder/file to be complete to finally process it in Python?

I'm trying to code a little script that watches a defined directory with a while-loop. Every file or directory that is in this directory is compressed to RAR and moved to another directory after the process is completed.
My problem: everytime I copy a file or folder to this directory, the script doesn't wait and startes the process the second it sees a new file or folder. But when the files or folders are bigger than a few kilobytes the loop breaks with a permission error.
Since I'm a Python beginner I don't know which module to use. Is there a checking module to see if the file or folder that the tool wants to process is used by another process? Or am I going in the wrong direction?
Edit: added the code for directory-only listening:
watchDir = "L:\\PythonTest\\testfolder\\"
finishedDir = "L:\\PythonTest\\finishedfolders\\"
rarfilesDir = "L:\\PythonTest\\rarfiles\\"
rarExe = "L:\\PythonTest\\rar.exe"
rarExtension = ".rar"
rarCommand = "a"
while True:
dirList = [name for name in os.listdir(watchDir) if os.path.isdir(os.path.join(watchDir,name))]
for entryName in dirList:
if not os.path.exists((os.path.join(finishedDir,entryName))):
sourcePath = os.path.join(watchDir,entryName)
entryNameStripped = entryName.replace(" ", "")
os.chdir(watchDir)
archiveName = rarfilesDir+entryNameStripped+rarExtension
subprocesscall = [rarExe, rarCommand, archiveName, entryName]
subprocess.call(subprocesscall, shell=True)
shutil.move(sourcePath,finishedDir)
When I run the script and try to add a file of several GB (named #filename# in the following lines) these errors occur:
Creating archive L:\PythonTest\rarfiles\#filename#.rar
Cannot open #filename#
The process cannot access the file, since it's used by another process.
Adding #filename# OK
WARNING: Cannot open 1 file
Done
Traceback (most recent call last):
File "C:\Python34\lib\shutil.py", line 522, in move
os.rename(src, real_dst)
PermissionError: [WinError 5] Access denied: #filepath#
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "L:/Python Test/test.py", line 35, in <module>
shutil.move(sourcePath,finishedDir)
File "C:\Python34\lib\shutil.py", line 531, in move
copytree(src, real_dst, symlinks=True)
File "C:\Python34\lib\shutil.py", line 342, in copytree
raise Error(errors)
shutil.Error: #filepath#

instead of using os.listdir, you can use os.walk, os.walk yields 3 tuple dirpath(path of directory,filenames(all files in that dirpath),dirnames(all the sub directories in dirpath)
for x,y,z in os.walk('path-of-directory'):
do you stuff with x,y,z the three tuples

Error copying files

I'm trying to write a short Python script that will copy all files from a directory with a certain extension and place them in a different folder. Here is the script:
import os, shutil
source = "C:\\TCR_Calgary\\projects\\"
destination = "C:\\TCR_Calgary\\reference\\my_code\\city_of_calgary_scripts\\"
extension = ".py"
for root, dirs, files in os.walk(source):
for file in files:
if file.lower().endswith(extension):
shutil.copy2(file, destination)
This is the exception I get:
Traceback (most recent call last):
File "C:/TCR_Calgary/reference/my_code/city_of_calgary_scripts/python_script_copier.py", line 13, in <module>
shutil.copy2(file, destination)
File "C:\Program Files (x86)\Python26\lib\shutil.py", line 99, in copy2
copyfile(src, dst)
File "C:\Program Files (x86)\Python26\lib\shutil.py", line 47, in copyfile
raise Error, "`%s` and `%s` are the same file" % (src, dst)
shutil.Error: `AnnoMover.py` and `C:\TCR_Calgary\reference\my_code\city_of_calgary_scripts\AnnoMover.py` are the same file
I'm confused because AnnoMover.py is not in the destination folder. In my mind, the original file and its copy would be "the same" although their paths would not. There's clearly something I'm not understanding here. Any help is much appreciated!
Version: Python 2.6
Interpeter: PyCharm Community Edition 3.4
OS: Windows 7

The exception message could be a little clearer, but the last line should be shutil.copy2(os.path.join(source, file), destination) so that the source is fully qualified.

I got it to work:
import os, shutil
source = r"C:\TCR_Calgary\projects"
destination = r"C:\TCR_Calgary\reference\my_code\city_of_calgary_scripts"
extension = ".py"
for root, dirs, files in os.walk(source):
for file in files:
if file.lower().endswith(extension):
file_path = os.path.realpath(os.path.join(root, file))
shutil.copy2(file_path, destination)

Removing a tree on a USB device in Python

I've used the following code to remove a tree on a USB device however I'm receiving an OSError:
I also ran the code with sudo python.
import shutil
import os
src = "/media/device/my_folder"
if os.path.exists(dst):
shutil.rmtree(dst)
I've just used shutil.copytree(src, dst) in another script to write the files to the device in the first place. However the USB device was removed during the copy, this is probably causing the issue I'm having as all other files except the one that was half copied have been removed okay.
I'm getting the following traceback:
Traceback (most recent call last):
File "writetousb/tests/deleteTest.py", line 32, in <module>
shutil.rmtree(src)
File "/usr/lib/python2.7/shutil.py", line 252, in rmtree
onerror(os.remove, fullname, sys.exc_info())
File "/usr/lib/python2.7/shutil.py", line 250, in rmtree
os.remove(fullname)
OSError: [Errno 30] Read-only file system: '/media/device/21823/21916.jpg'
So I'm guessing I'll need to change the permissions of the folder and it's files before I remove them?

If I use chmod to set the permissions correctly before I try to use shutil.rmtree then it should work. I'm going to test this and provide an update when I know it works.
I can confirm the solution works.
import shutil
import os
src = "/media/device/my_folder"
if os.path.exists(dst):
os.chmod(dst, 0o777)
for root,dirs,_ in os.walk(dst):
for d in dirs :
os.chmod(os.path.join(root,d) , 0o777)
shutil.rmtree(dst)

error says \\\\Ref\\builds/out exists but it doesnt exist

I am trying to copy a source code tree using the below code and running into an error,am not sure why I am getting this?error says \\Ref\builds/out exists but it doesnt exist,"out" is the directory the source location that the script is trying to copy to destination,any other ways to do copy if shutil is not suited for this type of copy?
//local/mnt/workspace/04.01_HY11/out
\\Ref\builds/out
copying
Traceback (most recent call last):
File "test.py", line 21, in <module>
main()
File "test.py", line 18, in main
copytree(src,dst)
File "test.py", line 11, in copytree
shutil.copytree(s, d)
File "/pkg/qct/software/python/2.5.2/.amd64_linux26/lib/python2.5/shutil.py", line 110, in copytree
os.makedirs(dst)
File "/pkg/qct/software/python/2.5.2/.amd64_linux26/lib/python2.5/os.py", line 171, in makedirs
mkdir(name, mode)
OSError: [Errno 17] File exists: '\\\\Ref\\builds/out'
Python code
import os,shutil
def copytree(src, dst, symlinks=False, ignore=None):
for item in os.listdir(src):
s = os.path.join(src, item)
print s
d = os.path.join(dst, item)
print d
if os.path.isdir(s):
print "copying"
shutil.copytree(s, d, symlinks, ignore)
else:
shutil.copy2(s, d)
def main ():
src="//local/mnt/workspace/04.01_HY11"
dst="\\\\Ref\\builds"
copytree(src,dst)
if __name__ == '__main__':
main()

The documentation for shutil.copytree clearly says:
The destination directory, named by dst, must not already exist; it will be created as well as missing parent directories.
But \\ref\builds\out already exists—you can see from the exception's stack trace that it's trying to mkdir that path, but that's failing with an error indicating that that path already exists (which can happen when the path exists as either a regular file or a directory).
You need to copy to path that doesn't already exist, by either choosing a different path, or by deleting the existing tree at that location first. The latter can be done with shutil.rmtree.

Python script to copy files

I am trying to write a Python script that will copy files from one user’s home directory into another user’s home directory. I want it to copy the permissions, as well. I read the Python API and I think that the copy2 method does just that. However, when I run the following code, I get errors.
import shutil
src = raw_input("Please enter a source: ")
dst = raw_input("Please enter a destination: ")
shutil.copy2(src, dst)
The error says:
Traceback (most recent call last):
File "copyfiles.py", line 5, in <module>
shutil.copy2(src, dst)
File "/usr/lib/python2.6/shutil.py", line 99, in copy2
copyfile(src, dst)
File "/usr/lib/python2.6/shutil.py", line 52, in copyfile
fsrc = open(src, 'rb')
IOError: [Errno 2] No such file or directory: '../../../../Desktop/byteswap.c'

Check your current directory using os.getcwd().