We Keep Coding

Watchdog ignore pattern

I tried working with other posts on here about this, but couldn't get it to work. I'm a newbie with Python.
I need help with ignore_pattern.
I'm uploading images to a folder and temporarily images are being added with __, so the actual image added while file is uploading is __image-name.jpg. After it's done uploading it gets added again as image-name.jpg (and deletes the __image-name.jpg.
I want to ignore all the __image-name.jpg files with watchdog.
Here's my code:
class Watcher:
DIRECTORY_TO_WATCH = "director/where/images/are/uploaded"
def __init__(self):
self.observer = Observer()
def run(self):
event_handler = Handler()
self.observer.schedule(event_handler, self.DIRECTORY_TO_WATCH, recursive=True)
self.observer.start()
try:
while True:
time.sleep(5)
except:
self.observer.stop()
print("Error")
self.observer.join()
class Handler(FileSystemEventHandler):
#staticmethod
def on_any_event(event):
if event.is_directory:
return None
elif event.event_type == 'created':
# Take any action here when a file is first created.
print(event.src_path)
img = Image.open(event.src_path)
for result in engine.classify_with_image(img, top_k=3):
print('---------------------------')
print(labels[result[0]])
print('Score : ', result[1])
# elif event.event_type == 'modified':
# Taken any action here when a file is modified.
# print("Received modified event - %s." % event.src_path)
elif event.event_type == 'deleted':
# Taken any action here when a file is deleted.
print("Received deleted event - %s." % event.src_path)
if __name__ == '__main__':
w = Watcher()
w.run()
Thank you so much.

Does event.src_path returns a string? If so you can use the startswith method of the string class to skip over the images you don't want.
For example:
elif event.event_type == 'created':
# Take any action here when a file is first created.
print(event.src_path)
# Check if this filename starts with "__" and execute the next block
if not event.src_path.startswith('__'):
img = Image.open(event.src_path)
for result in engine.classify_with_image(img, top_k=3):
print('---------------------------')
print(labels[result[0]])
print('Score : ', result[1])
# else do nothing

File Monitoring using watchdog library

I have written one python script to monitor one local folder which is having only .txt files and i want to start this script automatically if some changes happened to the folder(created,deleted or updated)
I tried to run this script and also tried to make changes in the directory, but i couldn't see any output and no error messages. It always says "Process finished with exit code 0" can any one review my code and give me some tips where to correct to get the expected out put.
import os
import sys
import time
from watchdog.observers import Observer
from watchdog.events import PatternMatchingEventHandler
#Step 1 Create the event handler
if __name__ == "__main__":
patterns = ".txt"
ignore_patterns = None
ignore_directories = False
case_sensitive = True
event_handler = PatternMatchingEventHandler(patterns, ignore_patterns, ignore_directories, case_sensitive)
#step 2 Handle all the events
def on_created(event):
print("new files has been created!")
def on_deleted(event):
print("Some files has been Deleted")
def on_modified(event):
print("Some files has been modified")
def on_moved(event):
print("Some files has been moved")
#step 3 specify to the handler that we want these functions to be called
event_handler.on_created = on_created
event_handler.on_deleted = on_deleted
event_handler.on_modified = on_modified
event_handler.on_moved = on_moved
#step 4 create an observer
path = "T:\Laboratory\Instruments\Worklists\TrackMateRacks\old"
go_recursively = True
my_observer = Observer()
my_observer.path(event_handler, path, recursive=go_recursively)
# start the observer
my_observer.start()
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
my_observer.stop()
my_observer.join()

you have to move the lines after " # start the observer" to the very left.
otherwise there will be executed nothing. that lines are part of on_moved(). but you want them to be executed if you start the script.
or
for most programs it's useful to add this line:
if __name__ == '__main__':
bevore line "# start the observer"
than your my_observer.start() will be executed, if you call your script. but if you import your script in another script, this will not be executed, but the other script can use all the functions, you created.

It seems, that you're really new to Python. You've to watch the indents, they're part of the syntax.
Exceptionally I reformat the complete code for you:
import os
import sys
import time
from watchdog.observers import Observer
from watchdog.events import PatternMatchingEventHandler
#Step 1 Create the event handler
patterns = ".txt"
ignore_patterns = None
ignore_directories = False
case_sensitive = True
event_handler = PatternMatchingEventHandler(patterns, ignore_patterns, ignore_directories, case_sensitive)
#step 2 Handle all the events
def on_created(event):
print("new files has been created!")
def on_deleted(event):
print("Some files has been Deleted")
def on_modified(event):
print("Some files has been modified")
def on_moved(event):
print("Some files has been moved")
#step 3 specify to the handler that we want these functions to be called
event_handler.on_created = on_created
event_handler.on_deleted = on_deleted
event_handler.on_modified = on_modified
event_handler.on_moved = on_moved
#step 4 create an observer
def main():
path = "T:\Laboratory\Instruments\Worklists\TrackMateRacks\old"
go_recursively = True
my_observer = Observer()
my_observer.path(event_handler, path, recursive=go_recursively)
# start the observer
my_observer.start()
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
my_observer.stop()
my_observer.join()
if __name__ == "__main__":
main()
#end of file
Good luck!

Python watchdog for files from two different directories

I am trying to listen to filesystem changes using the watchdog module of Python. I want to monitor the files from two different directories. For a single file watch, I used PatternMatchingEventHandler from watchdog.events. I want to use the same for multiple directories.
code:
import time
from watchdog.observers import Observer
from watchdog.events import PatternMatchingEventHandler
class EventHandler(PatternMatchingEventHandler):
def on_modified(self, event):
super(EventHandler, self).on_modified(event)
print event
if __name__ == "__main__":
dir_name = ["/home/user1/first", "/home/user1/second"]
observer = Observer()
patterns = ["/home/user1/first/first.log","/home/user1/second/second.log")]
for i in xrange(len(dir_name)):
event_handler = EventHandler(patterns = patterns[i])
observer.schedule(event_handler, dir_name[i], recursive=True)
observer.start()
try:
while True:
time.sleep(0.1)
except KeyboardInterrupt:
observer.stop()
In the above code, I tried to do multiple directory observing format and create an event handler for each of the files. It's not working for me. Is there anything that I am missing here?? What is the way to do this??
Thanks.

some wrong in here
EventHandler(patterns = patterns[i])
arg patterns is a type of list, so you can use like this
patterns = [["/home/user1/first/first.log"], ["/home/user1/second/second.log"]]
EventHandler(patterns = patterns[i])

Though it does not use the watchdog library , this will the easy way just to check if the specific type of files are added or removed
if u want to check which files u can append them using any variable and store them in array
import os
import fnmatch
import threading
import time
initial_count = 0
flag = 0
files = []
path = ["/home/kirti/workspace/pythonproject6/img", "/home/kirti/workspace/pythonproject6/copy"]
def taskcount(path, flag, initial_count):
while 1:
time.sleep(3)
new_count = len(fnmatch.filter(os.listdir(path), "*.jpg"))
if new_count > initial_count:
if flag != 0:
print("Added \nCount :", new_count, "=", path)
else:
print(new_count)
if new_count < initial_count:
print("Removed \nCount :", new_count, "=", path)
initial_count = new_count
flag = 1
for j in range(len(path)):
t = threading.Thread(target=taskcount, args=(path[j], flag, initial_count))
t.start()

I am using python3, LINUX OS
With a minor modification as suggested above and some additions from myside too this is working now
import time
from watchdog.observers import Observer
from watchdog.events import PatternMatchingEventHandler
class EventHandler(PatternMatchingEventHandler):
def on_modified(self, event):
super(EventHandler, self).on_modified(event)
print(event)
if __name__ == "__main__":
dir_name = ["/home/don/test1", "/home/don/test2"]
observer = Observer()
threads=[]
patterns = [['*.log'],['*.ok']]
for i in range(len(dir_name)):
event_handler = EventHandler(patterns = patterns[i],ignore_directories=True,case_sensitive=False)
observer.schedule(event_handler, dir_name[i], recursive=True)
threads.append(observer)
observer.start()
try:
while True:
time.sleep(0.1)
except KeyboardInterrupt:
observer.stop()
observer.join()

Comparing with mine, you are lacking of observer.join() at the EOF. Try with that.
EDIT
Try this code below:
import time
from watchdog.observers import Observer
from watchdog.events import PatternMatchingEventHandler
class EventHandler(PatternMatchingEventHandler):
def on_modified(self, event):
super(EventHandler, self).on_modified(event)
print event
if __name__ == "__main__":
observer = Observer()
patterns = ["/home/user1/first/first.log","/home/user1/second/second.log"]
for pattern in patterns:
event_handler = EventHandler(patterns=pattern)
observer.schedule(event_handler, dir_name[i], recursive=True)
observer.start()
try:
while True:
time.sleep(0.1)
except KeyboardInterrupt:
observer.stop()
observer.join()

re-using the variable in a function in a class in python

I will try to explain the problem I am facing with a small piece of code:
class MyHandler(PatternMatchingEventHandler):
patterns = ["*.csv","*.processing", "*.transforming","*.loading"]
def process(self, event):
eventFileName = event.src_path
eventType = event.event_type
if eventType == 'moved':
eventFileName = event.dest_path
fileNameWithPath, fileExtension = os.path.splitext(eventFileName)
if fileExtension == '.processing':
# Here some function is called to do something, and then appends ".loading" to the file name
testVariable = 75.3
if fileExtension == '.loading':
print testVariable
def on_moved(self, event):
self.process(event)
def on_created(self, event):
self.process(event)
if __name__ == '__main__':
observer = Observer()
observer.schedule(MyHandler(), path='.')
observer.start()
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
observer.stop()
observer.join()
When I try to do the above I am getting this error: global name 'testVariable' is not defined which kinda makes sense but how do I make the above work? I tried to define "testVariable" globally and initiated to 0 and tried using it the way I showed in above code but it did not work as well.
I also tried initiating testVariable as testVariable=0 inside the class (right after "patterns = ....." line but I got this error: local variable "testVariable" referenced before assignment pointing towards print testVariable. So that didnot work as well.

"(...) how do I make the above work?"
By defining testVariable outside your conditional statements. E.g. here:
def process(self, event):
eventFileName = event.src_path
testVariable = 0
...
This will make it available within the process function. If you want it to be available throughout the class, you can define it here:
class MyHandler(PatternMatchingEventHandler):
patterns = ["*.csv","*.processing", "*.transforming","*.loading"]
testVariable = 0
But then you have to access it via the self object within functions like so:
def process(self, event):
...
if fileExtension == '.processing':
# Here some function is called to do something, and then appends ".loading" to the file name
self.testVariable = 75.3

testVariable only exists if you have the extension ".processing". If it's ".loading", the program tries to print a variable that hasn't been made to exist.
If statements do not create a garbage collecting scope in Python, so you don't have to "declare" it outside, so long as somewhere in your if-tree, tesVariable gets a value.

def process(self, event):
def extension():
eventFileName = event.src_path
eventType = event.event_type
if eventType == 'moved':
eventFileName = event.dest_path
return os.path.splitext(eventFileName)[1]
if extension() == '.processing':
...
if extension() == '.loading':
...

How to pass an argument to my Python watchdog.events.PatternMatchingEventHandler

I'm quite new to Python and hope the answer to this is obvious to most of you.
I'm creating a class in Python that represents a ScanFolder.
In the __init__ of that class, I start a watchdog.observers
This observer will fire a watchdog.events.PatternMatchingEventHandler whenever a file is changed under the watched directory.
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import sys
import time
from watchdog.observers import Observer
import watchdog.events
path = sys.argv[1] if len(sys.argv) > 1 else '.'
class MyEventHandler(watchdog.events.PatternMatchingEventHandler):
def on_any_event(self, event):
print(event.src_path, event.event_type)
class ScanFolder:
'Class defining a scan folder'
def __init__(self, path):
self.path = path
self.documents = dict() # key = document label value = Document reference
self.event_handler = MyEventHandler(patterns=["*.jpg", "*.jpeg", "*.png", "*.bmp", "*.pdf"],
ignore_patterns=[],
ignore_directories=True)
self.observer = Observer()
self.observer.schedule(self.event_handler, self.path, recursive=False)
self.observer.start()
def stop(self):
self.observer.stop()
self.observer.join()
scan_folder = ScanFolder(path)
try:
while True:
time.sleep(1)
"""Here, I'll act on my scan_folder object that lists the discovered files"""
except KeyboardInterrupt:
log.warning("Ouch !!! Keyboard interrupt received.")
scan_folder.stop()
My problem is the following:
How can I have my scan_folder object modified by my scan_folder.event_handler() ?
Actually, I would like to populate the scan_folder.documents dictionary wherever a file is detected in the scan folder.
Thank you very much and sorry for my ignorance.

There are a lot of way to do it: but the simplest way is set a bound method of ScanFolder as on_any_event callback function of watchdog.events.PatternMatchingEventHandler. So your code become:
class ScanFolder:
'Class defining a scan folder'
def __init__(self, path):
self.path = path
self.documents = dict() # key = document label value = Document reference
self.event_handler = watchdog.events.PatternMatchingEventHandler(patterns=["*.jpg", "*.jpeg", "*.png", "*.bmp", "*.pdf"],
ignore_patterns=[],
ignore_directories=True)
self.event_handler.on_any_event = self.on_any_event
self.observer = Observer()
self.observer.schedule(self.event_handler, self.path, recursive=False)
self.observer.start()
def on_any_event(self, event):
print(event.src_path, event.event_type)
print("Complete ScanFolder() access")
def stop(self):
self.observer.stop()
self.observer.join()
Other way could be derive ScanFolder from watchdog.events.PatternMatchingEventHandler .... But injecting function is one of the power of python