I am trying to learn python-watchdog, but I am sort of confused why the job I set up runs more than once. So, here is my set up:
#handler.py
import os
from watchdog.events import FileSystemEventHandler
from actions import run_something
def getext(filename):
return os.path.splitext(filename)[-1].lower()
class ChangeHandler(FileSystemEventHandler):
def on_any_event(self, event):
if event.is_directory:
return
if getext(event.src_path) == '.done':
run_something()
else:
print "event not directory.. exiting..."
pass
the observer is set up like so:
#observer.py
import os
import time
from watchdog.observers import Observer
from handler import ChangeHandler
BASEDIR = "/path/to/some/directory/bin"
def main():
while 1:
event_handler = ChangeHandler()
observer = Observer()
observer.schedule(event_handler, BASEDIR, recursive=True)
observer.start()
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
observer.stop()
observer.join()
if __name__ == '__main__':
main()
and finally, the actions like so:
#actions.py
import os
import subprocess
def run_something():
output = subprocess.check_output(['./run.sh'])
print output
return None
..where ./run.sh is just a shell script I would like to run when a file with an extension .done is found on /path/to/some/directory/bin
#run.sh
#!/bin/bash
echo "Job Start: $(date)"
rm -rf /path/to/some/directory/bin/job.done # remove the .done file
echo "Job Done: $(date)"
However, when I issue a python observer.py and then do a touch job.done on /path/to/some/directory/bin, I see that my shell script ./run.sh runs three times and not one..
I am confused why this runs thrice and not just once (I do delete the job.done file on my bash script)
To debug watchdog scripts, it is useful to print what watchdog is seeing as events. One file edit or CLI command, such as touch, can result in multiple watchdog events. For example, if you insert a print statement:
class ChangeHandler(FileSystemEventHandler):
def on_any_event(self, event):
print(event)
to log every event, running
% touch job.done
generates
2014-12-24 13:11:02 - <FileCreatedEvent: src_path='/home/unutbu/tmp/job.done'>
2014-12-24 13:11:02 - <DirModifiedEvent: src_path='/home/unutbu/tmp'>
2014-12-24 13:11:02 - <FileModifiedEvent: src_path='/home/unutbu/tmp/job.done'>
Above there were two events with src_path ending in job.done. Thus,
if getext(event.src_path) == '.done':
run_something()
runs twice because there is a FileCreatedEvent and a FileModifiedEvent.
You might be better off only monitoring FileModifiedEvents.
Related
The python code below is my first attempt for notification when a file is modified, however nothing happens when the file is changed.
What have I missed here?
#!/usr/bin/python3
import time
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler
class MyHandler(FileSystemEventHandler):
def on_modified(self, event):
print('file changed')
if __name__ == "__main__":
event_handler = MyHandler()
observer = Observer()
observer.schedule(event_handler, path='/Users/jeff/smb/storage/wsjt-x/wsjtx_log.adi', recursive=False)
observer.start()
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
observer.stop()
observer.join()
NOTE: I found that this code works well using Linux. It's on macOS Big Sur where I'm having issues.
After clarifying your environment in comments, one of following 3 approaches might resolve your issue.
Reproduction: Watching a local file on Linux
Running in one shell A and meanwhile using another shell B to add to the observed file so.md like:
echo "---" >> so.md
results in following output:
python3 so_watchdog.py so.md
file changed <FileModifiedEvent: event_type=modified, src_path='so.md', is_directory=False>
file changed <DirModifiedEvent: event_type=modified, src_path='', is_directory=True>
Here is the slightly modified version of your script (see comments):
#!/usr/bin/python3
import time
import sys
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler
class MyHandler(FileSystemEventHandler):
def on_modified(self, event):
print('file changed', event) # added event to see the difference of two events for a single modification
if __name__ == "__main__":
if len(sys.argv) < 2:
file = '/Users/jeff/smb/storage/wsjt-x/wsjtx_log.adi'
else:
file = sys.argv[1]
# below only added the file as path
event_handler = MyHandler()
observer = Observer()
observer.schedule(event_handler, path=file, recursive=False)
observer.start()
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
observer.stop()
observer.join()
Note: This was tested on Linux, with a local file.
Watching shared files (CIFS/SMB)
Since your path /Users/jeff/smb/storage/wsjt-x/wsjtx_log.adi contains smb I assume, you are watching a file on a mounted shared network location using the SMB (Server Message Blocks) protocol, also known as CIFS (Common Internet File System).
Then please follow the official documentation, About using watchdog with CIFS:
When you want to watch changes in CIFS, you need to explicitly tell watchdog to use PollingObserver, that is, instead of letting watchdog decide an appropriate observer like in the example above, do:
from watchdog.observers.polling import PollingObserver as Observe
So your import statement needs to be modified.
Watching files locally on MacOS
For MacOS the underlying file-system event API used is FSEvents or FreeBSD's kqueue, instead of Linux's inotify.
Thus the behavior might also deviate.
I'm relatively new to python so please forgive early level understanding!
I am working to create a kind of flag file. Its job is to monitor a Python executable, the flag file is constantly running and prints "Start" when the executable started, "Running" while it runs and "Stop" when its stopped or crashed, if a crash occurs i want it to be able to restart the script. so far i have this down for the Restart:
from subprocess import run
from time import sleep
# Path and name to the script you are trying to start
file_path = "py"
restart_timer = 2
def start_script():
try:
# Make sure 'python' command is available
run("python "+file_path, check=True)
except:
# Script crashed, lets restart it!
handle_crash()
def handle_crash():
sleep(restart_timer) # Restarts the script after 2 seconds
start_script()
start_script()
how can i implement this along with a flag file?
Not sure what you mean with "flag", but this minimally achieves what you want.
Main file main.py:
import subprocess
import sys
from time import sleep
restart_timer = 2
file_path = 'sub.py' # file name of the other process
def start():
try:
# sys.executable -> same python executable
subprocess.run([sys.executable, file_path], check=True)
except subprocess.CalledProcessError:
sleep(restart_timer)
return True
else:
return False
def main():
print("starting...")
monitor = True
while monitor:
monitor = start()
if __name__ == '__main__':
main()
Then the process that gets spawned, called sub.py:
from time import sleep
sleep(1)
print("doing stuff...")
# comment out to see change
raise ValueError("sub.py is throwing error...")
Put those files into the same directory and run it with python main.py
You can comment out the throwing of the random error to see the main script terminate normally.
On a larger note, this example is not saying it is a good way to achieve the quality you need...
I try to monitor text log file for changes and want to see lines that was added
I try with watchdog it seems to work with some manually created/edited files for testing, but my log file (in the same directory) isn't detected as "changed", but it was changed. It's probably because this file is already open (maybe in a specific mode). If I close the app that uses this log file and change it manually then watchdog works ok.
How do I check if the file was changed and if the changes were written to
the console?
#!/usr/bin/python
import time
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler
class MyHandler(FileSystemEventHandler):
def on_modified(self, event):
print(f'event type: {event.event_type} path : {event.src_path}')
if __name__ == "__main__":
event_handler = MyHandler()
observer = Observer()
observer.schedule(event_handler, path='C:\gory\parcienaszklo\logs', recursive=False)
observer.start()
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
observer.stop()
observer.join()
I want linux machine(Raspberry pi) to monitor a shared folder by AFP(Apple file protocol, macbook is host).
I can mount shared folder by mount_afp, and installed watchdog python library to monitor a shared folder. The problem is that watchdog can monitor only modifications from linux machine itself.
If a monitoring folder has been modified by the host machine(Apple macbook) or other pc, linux machine couldn't find out the change. No logs came out.
After I tested the same watchdog python file in host machine(Apple macbook), I can get every logs of modifications by other machine.
Host machine can get every modifications of file or folder. But other machine(guest machine) can not monitor modifications of file from host or others.
Is it normal status of watchdog? Or is there any problem in account and permission?
Here is watchdog sample.
import time
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler
class Watcher:
DIRECTORY_TO_WATCH = "/path/to/my/directory"
def __init__(self):
self.observer = Observer()
def run(self):
event_handler = Handler()
self.observer.schedule(event_handler, self.DIRECTORY_TO_WATCH, recursive=True)
self.observer.start()
try:
while True:
time.sleep(5)
except:
self.observer.stop()
print "Error"
self.observer.join()
class Handler(FileSystemEventHandler):
#staticmethod
def on_any_event(event):
if event.is_directory:
return None
elif event.event_type == 'created':
# Take any action here when a file is first created.
print "Received created event - %s." % event.src_path
elif event.event_type == 'modified':
# Taken any action here when a file is modified.
print "Received modified event - %s." % event.src_path
if __name__ == '__main__':
w = Watcher()
w.run()
For network mounts, the usual filesystem events don't always get emitted. In such cases, instead of using Observer, try using PollingObserver -- i.e., change from:
self.observer = Observer()
to
from watchdog.observers.polling import PollingObserver
...
self.observer = PollingObserver()
There is also a PollingObserverVFS class you could experiment with.
Documentation: https://pythonhosted.org/watchdog/api.html#module-watchdog.observers.polling
I am trying to start a python script everytime is a certain file modified. To be accurate, I have a device on Raspberry Pi's serial ports which writes data into a text file (test.txt) . I have tried both tools - Watchdog / Pyinotify. Everytime the file is modified (triggers event Watchdog: on_modified / Pyinotify: IN_MODIFY), it makes duplicate trigger. I have tried every other method, even IN_CLOSE_WRITE as some people suggest, but this doesn't work at all.
Does someone know, how can just a single event be triggered on one file update?
My code using Pyinotify (a bit edited tutorial file):
import pyinotify,subprocess
def onChange(ev):
cmd = ['/usr/bin/env', 'python', 'doThisFile.py', ev.pathname]
subprocess.Popen(cmd).communicate()
wm = pyinotify.WatchManager()
wm.add_watch('/home/pi/test.txt', pyinotify.IN_MODIFY, onChange)
notifier = pyinotify.Notifier(wm)
notifier.loop()
or Watchdog:
#!/usr/bin/python
import time
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler
import subprocess
class MyHandler(FileSystemEventHandler):
def on_modified(self, event):
subprocess.call("/home/pi/doThisFile.py")
print("Code started")
if __name__ == "__main__":
event_handler = MyHandler()
observer = Observer()
observer.schedule(event_handler, path='.', recursive=False)
observer.start()
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
observer.stop()
observer.join()
I was facing the same issue while using pyinotify. But changing IN_MODIFY to IN_CLOSE_WRITE solved the issue. You can get more info from this answer
This probably because of the text editor which was used for editing your source code.
solution: calculate the time between 2 events, for example, If the events trigger too frequently, exit the handler from one of the events.
class SubEventHandler(FileSystemEventHandler):
def __init__(self):
self._time0 = self.getTime()
def on_modified(self, event):
time1 = self.getTime()
if (time1 - self._time0) < 5:
exit(0) # not sure it will exit properly
The 2 events that you are getting are:
The file was modified
The directory was modified
If you run the demo code and then in a touch a file in the directory being watched, you'll see the following output:
2022-11-04 10:28:45 - Modified file: ./test
2022-11-04 10:28:45 - Modified directory: .