I want linux machine(Raspberry pi) to monitor a shared folder by AFP(Apple file protocol, macbook is host).
I can mount shared folder by mount_afp, and installed watchdog python library to monitor a shared folder. The problem is that watchdog can monitor only modifications from linux machine itself.
If a monitoring folder has been modified by the host machine(Apple macbook) or other pc, linux machine couldn't find out the change. No logs came out.
After I tested the same watchdog python file in host machine(Apple macbook), I can get every logs of modifications by other machine.
Host machine can get every modifications of file or folder. But other machine(guest machine) can not monitor modifications of file from host or others.
Is it normal status of watchdog? Or is there any problem in account and permission?
Here is watchdog sample.
import time
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler
class Watcher:
DIRECTORY_TO_WATCH = "/path/to/my/directory"
def __init__(self):
self.observer = Observer()
def run(self):
event_handler = Handler()
self.observer.schedule(event_handler, self.DIRECTORY_TO_WATCH, recursive=True)
self.observer.start()
try:
while True:
time.sleep(5)
except:
self.observer.stop()
print "Error"
self.observer.join()
class Handler(FileSystemEventHandler):
#staticmethod
def on_any_event(event):
if event.is_directory:
return None
elif event.event_type == 'created':
# Take any action here when a file is first created.
print "Received created event - %s." % event.src_path
elif event.event_type == 'modified':
# Taken any action here when a file is modified.
print "Received modified event - %s." % event.src_path
if __name__ == '__main__':
w = Watcher()
w.run()
For network mounts, the usual filesystem events don't always get emitted. In such cases, instead of using Observer, try using PollingObserver -- i.e., change from:
self.observer = Observer()
to
from watchdog.observers.polling import PollingObserver
...
self.observer = PollingObserver()
There is also a PollingObserverVFS class you could experiment with.
Documentation: https://pythonhosted.org/watchdog/api.html#module-watchdog.observers.polling
Related
The python code below is my first attempt for notification when a file is modified, however nothing happens when the file is changed.
What have I missed here?
#!/usr/bin/python3
import time
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler
class MyHandler(FileSystemEventHandler):
def on_modified(self, event):
print('file changed')
if __name__ == "__main__":
event_handler = MyHandler()
observer = Observer()
observer.schedule(event_handler, path='/Users/jeff/smb/storage/wsjt-x/wsjtx_log.adi', recursive=False)
observer.start()
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
observer.stop()
observer.join()
NOTE: I found that this code works well using Linux. It's on macOS Big Sur where I'm having issues.
After clarifying your environment in comments, one of following 3 approaches might resolve your issue.
Reproduction: Watching a local file on Linux
Running in one shell A and meanwhile using another shell B to add to the observed file so.md like:
echo "---" >> so.md
results in following output:
python3 so_watchdog.py so.md
file changed <FileModifiedEvent: event_type=modified, src_path='so.md', is_directory=False>
file changed <DirModifiedEvent: event_type=modified, src_path='', is_directory=True>
Here is the slightly modified version of your script (see comments):
#!/usr/bin/python3
import time
import sys
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler
class MyHandler(FileSystemEventHandler):
def on_modified(self, event):
print('file changed', event) # added event to see the difference of two events for a single modification
if __name__ == "__main__":
if len(sys.argv) < 2:
file = '/Users/jeff/smb/storage/wsjt-x/wsjtx_log.adi'
else:
file = sys.argv[1]
# below only added the file as path
event_handler = MyHandler()
observer = Observer()
observer.schedule(event_handler, path=file, recursive=False)
observer.start()
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
observer.stop()
observer.join()
Note: This was tested on Linux, with a local file.
Watching shared files (CIFS/SMB)
Since your path /Users/jeff/smb/storage/wsjt-x/wsjtx_log.adi contains smb I assume, you are watching a file on a mounted shared network location using the SMB (Server Message Blocks) protocol, also known as CIFS (Common Internet File System).
Then please follow the official documentation, About using watchdog with CIFS:
When you want to watch changes in CIFS, you need to explicitly tell watchdog to use PollingObserver, that is, instead of letting watchdog decide an appropriate observer like in the example above, do:
from watchdog.observers.polling import PollingObserver as Observe
So your import statement needs to be modified.
Watching files locally on MacOS
For MacOS the underlying file-system event API used is FSEvents or FreeBSD's kqueue, instead of Linux's inotify.
Thus the behavior might also deviate.
I have a simple Watchdog and Queue process to monitor files in a directory.
Code taken from https://camcairns.github.io/python/2017/09/06/python_watchdog_jobs_queue.html
import time
from watchdog.events import PatternMatchingEventHandler
from watchdog.observers import Observer
from queue import Queue
from threading import Thread
dir_path = "/data"
def process_queue(q):
while True:
if not q.empty():
event = q.get()
print("New event %s" % event)
time.sleep(5)
class FileWatchdog(PatternMatchingEventHandler):
def __init__(self, queue, patterns):
PatternMatchingEventHandler.__init__(self, patterns=patterns)
self.queue = queue
def process(self, event):
self.queue.put(event)
def on_created(self, event):
self.process(event)
if __name__ == '__main__':
watchdog_queue = Queue()
worker = Thread(target=process_queue, args=(watchdog_queue,))
worker.setDaemon(True)
worker.start()
event_handler = FileWatchdog(watchdog_queue, patterns="*.ini")
observer = Observer()
observer.schedule(event_handler, path=dir_path)
observer.start()
try:
while True:
time.sleep(2)
except KeyboardInterrupt:
observer.stop()
observer.join()
Once the process is running new files are processed correctly.
However if I restart the process and a file already exists in the directory it is ignored.
I have tried to create a dict to add to the queue
for file in os.listdir(dir_path):
if file.endswith(".ini"):
file_path = os.path.join(dir_path, file)
event = {'event_type' : 'on_created', 'is_directory' : 'False', 'src_path' : file_path}
watchdog_queue.put(event)
but it's expecting an object of type (class 'watchdog.events.FileCreatedEvent') and I can't work out how to create this.
Alternatively I can see in the Watchdog documentation (class watchdog.utils.dirsnapshot.DirectorySnapshot) but I cannot work out how to run this and add it to the queue.
Any suggestions on how I can add existing files to the queue on startup ?
This code should do what you are trying to achieve.
from watchdog.events import FileCreatedEvent
# Loop to get all files; dir_path is your lookup folder.
for file in os.listdir(dir_path):
filename = os.path.join(dir_path, file)
event = FileCreatedEvent(filename)
watchdog_queue.put(event)
I stumbled over the same problem and maybe this solution is for you too. At least on linux this works like a charm.
Add the "on_modified" method
class FileWatchdog(PatternMatchingEventHandler):
def __init__(self, queue, patterns):
PatternMatchingEventHandler.__init__(self, patterns=patterns)
self.queue = queue
...
def on_modified(self, event):
self.process(event)
Now after starting the observer, loop through all files in directory and "touch" them, so they will be "modified".
# Loop to get all files; dir_path is your lookup folder.
for file in os.listdir(dir_path):
filename = os.path.join(dir_path, file)
os.popen(f'touch {filename}')
No need to add special filters as your FileHandler will handle that.
I try to monitor text log file for changes and want to see lines that was added
I try with watchdog it seems to work with some manually created/edited files for testing, but my log file (in the same directory) isn't detected as "changed", but it was changed. It's probably because this file is already open (maybe in a specific mode). If I close the app that uses this log file and change it manually then watchdog works ok.
How do I check if the file was changed and if the changes were written to
the console?
#!/usr/bin/python
import time
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler
class MyHandler(FileSystemEventHandler):
def on_modified(self, event):
print(f'event type: {event.event_type} path : {event.src_path}')
if __name__ == "__main__":
event_handler = MyHandler()
observer = Observer()
observer.schedule(event_handler, path='C:\gory\parcienaszklo\logs', recursive=False)
observer.start()
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
observer.stop()
observer.join()
I am trying to start a python script everytime is a certain file modified. To be accurate, I have a device on Raspberry Pi's serial ports which writes data into a text file (test.txt) . I have tried both tools - Watchdog / Pyinotify. Everytime the file is modified (triggers event Watchdog: on_modified / Pyinotify: IN_MODIFY), it makes duplicate trigger. I have tried every other method, even IN_CLOSE_WRITE as some people suggest, but this doesn't work at all.
Does someone know, how can just a single event be triggered on one file update?
My code using Pyinotify (a bit edited tutorial file):
import pyinotify,subprocess
def onChange(ev):
cmd = ['/usr/bin/env', 'python', 'doThisFile.py', ev.pathname]
subprocess.Popen(cmd).communicate()
wm = pyinotify.WatchManager()
wm.add_watch('/home/pi/test.txt', pyinotify.IN_MODIFY, onChange)
notifier = pyinotify.Notifier(wm)
notifier.loop()
or Watchdog:
#!/usr/bin/python
import time
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler
import subprocess
class MyHandler(FileSystemEventHandler):
def on_modified(self, event):
subprocess.call("/home/pi/doThisFile.py")
print("Code started")
if __name__ == "__main__":
event_handler = MyHandler()
observer = Observer()
observer.schedule(event_handler, path='.', recursive=False)
observer.start()
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
observer.stop()
observer.join()
I was facing the same issue while using pyinotify. But changing IN_MODIFY to IN_CLOSE_WRITE solved the issue. You can get more info from this answer
This probably because of the text editor which was used for editing your source code.
solution: calculate the time between 2 events, for example, If the events trigger too frequently, exit the handler from one of the events.
class SubEventHandler(FileSystemEventHandler):
def __init__(self):
self._time0 = self.getTime()
def on_modified(self, event):
time1 = self.getTime()
if (time1 - self._time0) < 5:
exit(0) # not sure it will exit properly
The 2 events that you are getting are:
The file was modified
The directory was modified
If you run the demo code and then in a touch a file in the directory being watched, you'll see the following output:
2022-11-04 10:28:45 - Modified file: ./test
2022-11-04 10:28:45 - Modified directory: .
I am trying to learn python-watchdog, but I am sort of confused why the job I set up runs more than once. So, here is my set up:
#handler.py
import os
from watchdog.events import FileSystemEventHandler
from actions import run_something
def getext(filename):
return os.path.splitext(filename)[-1].lower()
class ChangeHandler(FileSystemEventHandler):
def on_any_event(self, event):
if event.is_directory:
return
if getext(event.src_path) == '.done':
run_something()
else:
print "event not directory.. exiting..."
pass
the observer is set up like so:
#observer.py
import os
import time
from watchdog.observers import Observer
from handler import ChangeHandler
BASEDIR = "/path/to/some/directory/bin"
def main():
while 1:
event_handler = ChangeHandler()
observer = Observer()
observer.schedule(event_handler, BASEDIR, recursive=True)
observer.start()
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
observer.stop()
observer.join()
if __name__ == '__main__':
main()
and finally, the actions like so:
#actions.py
import os
import subprocess
def run_something():
output = subprocess.check_output(['./run.sh'])
print output
return None
..where ./run.sh is just a shell script I would like to run when a file with an extension .done is found on /path/to/some/directory/bin
#run.sh
#!/bin/bash
echo "Job Start: $(date)"
rm -rf /path/to/some/directory/bin/job.done # remove the .done file
echo "Job Done: $(date)"
However, when I issue a python observer.py and then do a touch job.done on /path/to/some/directory/bin, I see that my shell script ./run.sh runs three times and not one..
I am confused why this runs thrice and not just once (I do delete the job.done file on my bash script)
To debug watchdog scripts, it is useful to print what watchdog is seeing as events. One file edit or CLI command, such as touch, can result in multiple watchdog events. For example, if you insert a print statement:
class ChangeHandler(FileSystemEventHandler):
def on_any_event(self, event):
print(event)
to log every event, running
% touch job.done
generates
2014-12-24 13:11:02 - <FileCreatedEvent: src_path='/home/unutbu/tmp/job.done'>
2014-12-24 13:11:02 - <DirModifiedEvent: src_path='/home/unutbu/tmp'>
2014-12-24 13:11:02 - <FileModifiedEvent: src_path='/home/unutbu/tmp/job.done'>
Above there were two events with src_path ending in job.done. Thus,
if getext(event.src_path) == '.done':
run_something()
runs twice because there is a FileCreatedEvent and a FileModifiedEvent.
You might be better off only monitoring FileModifiedEvents.