We Keep Coding

Python: How to resume the python script as soon as vpn network is up?

I have a python script (xyz.py) that I run through the command prompt. My question is that don't we have any method which helps to resume the python code automatically from where it was lost the VPN connection, without any manual intervention. This will help to avoid monitoring the code frequently. Below is my code but it reads from the start if there is any disconnection. Please suggest.
filename = 'xyz.py'
while True:
p = subprocess.Popen('python '+filename, shell=True).wait()
""" #if your there is an error from running 'xyz.py',
the while loop will be repeated,
otherwise the program will break from the loop"""
if p != 0:
continue
else:
break

If me, time.sleep will be used:
import os
import time
from datetime import datetime
import requests
script = 'xyz.py'
def main():
network_check_url = 'http://8.8.8.8'
while True:
try:
requests.get(network_check_url)
except Exception as e:
print(datetime.now(), e)
time.sleep(1)
else:
print('Network is ok. {datetime.now():%Y-%m-%d_%H:%M:%S}')
os.system(f'python {script}')
return
if __name__ == '__main__':
main()

How do I correctly manage the serial resource? [Python]

I'm working on an automation project with python and an Arduino. I need to run a test that ensures that the Arduino is connected (and not some other device). I have a simple test case that works, but doesn't manage the serial object's memory at all. I've got a solution that I feel should work, however it causes an issue with the serial connection. The result is that the first read returns an empty string. Subsequent reads return the correct value. For my application I need it to read correctly on the first try.
Here is the test code I used. This code works just fine (it's how I know the issue isn't on the Arduino side of things)
ser = serial.Serial('/dev/tty.usbmodem14101', 9600, timeout=1)
ser.flush()
while True:
text = input("Input: ")
ser.write(bytes(text, 'utf-8'))
time.sleep(0.2)
line = ser.readline().decode('utf-8').rstrip()
print(line)
This is my attempt at creating a better way to manage the serial resource using the context manager and the with statement. I tried 2 things. The ArduinoWith class only returns false (and prints and empty string) with no indication of any input on the Arduino. The ArduinoSelf has the same result the first time, but returns True the second time.
import serial
import time
from contextlib import contextmanager
#contextmanager
def arduino_serial_connection(path, rate, timeout=1):
connection = serial.Serial(path, rate, timeout=timeout)
yield connection
connection.close()
class ArduinoWith:
def __init__(self):
pass
def connection_test(self):
try:
with arduino_serial_connection('/dev/tty.usbmodem14101', 9600) as connection:
connection.write(b"S\n")
time.sleep(0.2)
answer = connection.readline().decode('utf-8').rstrip()#connection.read(8)
if answer == "H":
return True
else:
print("Answer: \"" + answer +"\"")
except serial.serialutil.SerialTimeoutException:
print("Timeout")
except Exception:
print("Execption")
return False
class ArduinoSelf:
def __init__(self):
self.ser = serial.Serial('/dev/tty.usbmodem14101', 9600, timeout=1)
def connection_test(self):
try:
self.ser.write(b"S\n")
time.sleep(0.2)
answer = self.ser.readline().decode('utf-8').rstrip()#connection.read(8)
if answer == "H":
return True
else:
print("Answer: \"" + answer +"\"")
except serial.serialutil.SerialTimeoutException:
print("Timeout")
except Exception:
print("Execption")
return False
a1 = ArduinoWith()
print(a1.connection_test())
time.sleep(1)
a2 = ArduinoSelf()
print(a2.connection_test())
time.sleep(1)
print(a2.connection_test())
I'm not exactly sure why running the method a second time on the ArduinoSelf class worked, but it did and that made me think there must be something that I'm not initializing correctly.

Correctly handle errors using watchdog

I'm trying to understand how to use the error handling correct in Python.
I'm usting Watchdog to look in my folers in a network connected disc. Somethimes the disc dissconnects shortly and then connects again and an error pops up. "Exception in thread Thread-2:"
I have an error handeler but I'm not sure I'm doing it correctly.
Should I put an other try at the observer.schedule step?
Python 3.6, Windows 10
if __name__ == '__main__':
path = "P:\\03_auto\\Indata"
observer = Observer()
observer.schedule(MyHandler(), path, recursive=True)
observer.start()
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
observer.stop()
observer.join()

Well, the sys.excepthook approach I mentioned in my comment is not feasible at the moment. There's a more than a decade old bug that makes threads derived from threading.Thread ignore sys.excepthook. There are some workarounds outlined in the message thread of that bug, but I am hestiant to post an answer employing a workaround, especially since this bug finally seems to get a fix for Python 3.8.
The other option would be to derive a custom Observer from Watchdog's Observer.
The most basic way I can think of is a wrapper around the parent's run() method:
class CustomObserver(Observer):
def run(self):
while self.should_keep_running():
try:
# Tweak the super call if you require compatibility with Python 2
super().run()
except OSError:
# You did not mention the excpetion class in your post.
# Be specific about what you want to handle here
# give the file system some time to recover
time.sleep(.5)
Judging by a quick glance at the source, all Observers seem to be inheriting their run from EventDispatcher.run(), so you probably could even omit the wrapper and reimplement that method directly
class CustomObserver(Observer):
def run(self):
while self.should_keep_running():
try:
self.dispatch_events(self.event_queue, self.timeout)
except queue.Empty:
continue
except OSError:
time.sleep(.5)
However, I don't have that package installed on my box, so these things are untested; you might have to fiddle around a bit to get this going.
Oh, and make sure to replace the OSError* by whatever exception class actually raises in your case :)
Edit:
* As per #HenryYik's comment below, a network drive disconnecting, appears to be raising an OSError (WinError 64: ERROR_NETNAME_DELETED) on Windows systems. I find it likely that UNIX style OSs raise the same type of exception in that situation, hence I updated the code snippets to now use OSError instead of the FileNotFoundError I used originally.

I think super().run() is not a good answer.
I've tried many things with shmee's answer, but it was not concrete and meaningless.
Below is my answer.
I've finished the test.
from watchdog.observers import Observer
from watchdog.events import PatternMatchingEventHandler
class Watcher :
def __init__(self):
self.observer = Observer()
print('observer init...')
def run(self):
global osv_status
try :
event_handler = MyHandler(patterns=["*.txt"])
self.observer.schedule(event_handler, Directory, recursive=True)
self.observer.start()
osv_status = 1
except OSError as ex:
print("OSError")
time.sleep(.5)
if self.observer.is_alive() is True:
self.observer.stop()
self.observer.join()
print("Observer is removed [ ",osv_status," ]")
osv_status = 2
except Exception as ex:
self.logger.exception("Exception ex :{0} ".format(ex))
class MyHandler(PatternMatchingEventHandler):
def __init__(self,*args, **kwargs):
super(MyHandler, self).__init__(*args, **kwargs)
print("MyHandler Init")
def on_created(self, event):
print('New file is created ',event.src_path)
except Exception as ex:
self.logger.exception("Exception ex :{0} ".format(ex))
def on_modified(self,event):
print('File is modified ',event.src_path)
try :
doing()
except Exception as ex:
self.logger.exception("Exception ex :{0} ".format(ex))
if __name__ == "__main__":
.....
osv_status = 0 # 0: ready, 1:start 2: halt
wch =Watcher()
wch.run()
try :
while(1):
if is_Connect == False:
sock_connect()
print('sock_connect')
time.sleep(1)
if os.path.exists(Directory) is False and osv_status == 1:
wch.observer.stop()
wch.observer.join()
print("Observer is removed [ ",osv_status," ]")
osv_status = 0
elif os.path.exists(Directory) is True and (osv_status == 0 or osv_status == 2):
if wch.observer.is_alive() is False:
wch =Watcher()
wch.run()

I've also posted this in a related thread here
This is how i solved this problem:
from watchdog import observers
from watchdog.observers.api import DEFAULT_OBSERVER_TIMEOUT, BaseObserver
class MyEmitter(observers.read_directory_changes.WindowsApiEmitter):
def queue_events(self, timeout):
try:
super().queue_events(timeout)
except OSError as e:
print(e)
connected = False
while not connected:
try:
self.on_thread_start() # need to re-set the directory handle.
connected = True
print('reconnected')
except OSError:
print('attempting to reconnect...')
time.sleep(10)
observer = BaseObserver(emitter_class=MyEmitter, timeout=DEFAULT_OBSERVER_TIMEOUT)
...
Subclassing WindowsApiEmitter to catch the exception in queue_events. In order to continue after reconnecting, watchdog needs to re-set the directory handle, which we can do with self.on_thread_start().
Then use MyEmitter with BaseObserver, we can now handle losing and regaining connection to shared drives.

how to close python script without just activating "except" in try/except statement

I have a python script running over some 50,000 items in a database, it takes about 10 seconds per item and I want to stop it.
But, if i just press ctrl + C while the code is in the try except part, then my program enter the expect statement and delete that item then continue. The only way to close the program is to repeatedly delete items this way until I, just be sheer luck, catch it not in the try statement.
How do I exit the script without just killing a try statement?
EDIT 1
I have a statement that looks something like this:
for item in items:
try:
if is_item_gone(content) == 1:
data = (item['id'])
db.update('UPDATE items SET empty = 0 WHERE id = (%s);', data)
else:
do alot of stuff
except:
data = (item['id'])
db.delete('...')
EDIT 2
The top of my connect to db code looks like:
#!/usr/bin/python
import MySQLdb
import sys
class Database:
....

The issue is because you are using a blanket except which is always a bad idea, if you catch specific exceptions then when you KeyboardInterrupt your script will stop:
for item in items:
try:
if is_item_gone(content) == 1:
data = (item['id'])
db.update('UPDATE items SET empty = 0 WHERE id = (%s);', data)
else:
do alot of stuff
except MySQLdb.Error as e:
print(e)
data = (item['id'])
db.delete('...')
If you have other exceptions to catch you can use multiple in the except:
except (KeyError, MySQLdb.Error) as e
At the very least you could catch Exception as e and print the error.
for item in items:
try:
if is_item_gone(content) == 1:
data = (item['id'])
db.update('UPDATE items SET empty = 0 WHERE id = (%s);', data)
else:
do alot of stuff
except Exception as e:
print(e)
data = (item['id'])
db.delete('...')
The moral of the story is don't use a blanket except, catch what you expect and logging the errors might also be a good idea. The exception-hierarchy is also worth reading to see exactly what you are catching.

I would rewrite it like this:
try:
# ...
except KeyboardInterrupt:
sys.exit(1)
except Exception: # I would use a more specific exception here if you can.
data = (item['id'])
db.delete('...')
Or just
try:
# ...
except Exception: # I would use an even more specific exception here if you can.
data = (item['id'])
db.delete('...')
Python 2's exception hiearchy can be found here: https://docs.python.org/2/library/exceptions.html#exception-hierarchy
Also, many scripts are written with something like this boiler plate at the bottom:
def _main(args):
...
if __name__ == '__main__':
try:
sys.exit(_main(sys.argv[1:]))
except KeyboardInterrupt:
print >> sys.stderr, '%s: interrupted' % _SCRIPT_NAME
sys.exit(1)
When you use except: instead of except Exception: in your main body, you are not allowing this top level exception block to catch the KeyboardInterrupt. (You don't have to actually catch it for Ctrl-C to work -- this just makes it prettier when KeyboardInterrupt is raised.)

import signal
import MySQLdb
import sys
## Handle process signals:
def sig_handler(signal, frame):
global connection
global cursor
cursor.close()
connection.close()
exit(0)
## Register a signal to the handler:
signal.signal(signal.SIGINT, sig_handler)
class Database:
# Your class stuf here.
# Note that in sig_handler() i close the cursor and connection,
# perhaps your class has a .close call? If so use that.
for item in items:
try:
if is_item_gone(content) == 1:
data = (item['id'])
db.update('UPDATE items SET empty = 0 WHERE id = (%s);', data)
else:
do alot of stuff
except MySQLdb.Error as e:
data = (item['id'])
db.delete('...')
This would register and catch for instance Ctrl+c at any given time. The handler would terminate a socket accordingly, then exit with exit code 0 which is a good exit code.
This is a more "global" approach to catching keyboard interrupts.

OSError: [Errno 11] Resource temporarily unavailable. What causes this?

Background
I have two python processes that need to communicate with each other. The comminication is handled by a class named Pipe. I made a seperate class for this because most of the information that needs to be communicated comes in the form of dictionaries so Pipe implements a pretty simple protocol for doing this.
Here is the Pipe constructor:
def __init__(self,sPath):
"""
create the fifo. if it already exists just associate with it
"""
self.sPath = sPath
if not os.path.exists(sPath):
try:
os.mkfifo(sPath)
except:
raise Exception('cannot mkfifo at path \n {0}'.format(sPath))
self.iFH = os.open(sPath,os.O_RDWR | os.O_NONBLOCK)
self.iFHBlocking = os.open(sPath,os.O_RDWR)
So ideally I would just construct a Pipe in each process with the same path and they would be able to talk nice.
I'm going to skip out stuff about the protocol because I think it is largely unnecessary here.
All read and write operations make use of the following 'base' functions:
def base_read_blocking(self,iLen):
self.lock()
lBytes = os.read(self.iFHBlocking,iLen)
self.unlock()
return lBytes
def base_read(self,iLen):
print('entering base read')
self.lock()
lBytes = os.read(self.iFH,iLen)
self.unlock()
print('exiting base read')
return lBytes
def base_write_blocking(self,lBytes):
self.lock()
safe_write(self.iFHBlocking,lBytes)
self.unlock()
def base_write(self,lBytes):
print('entering base write')
self.lock()
safe_write(self.iFH,lBytes)
self.unlock()
print('exiting base write')
safe_write was suggested in another post
def safe_write(*args, **kwargs):
while True:
try:
return os.write(*args, **kwargs)
except OSError as e:
if e.errno == 35:
import time
print(".")
time.sleep(0.5)
else:
raise
locking and unlocking is handled like this:
def lock(self):
print('locking...')
while True:
try:
os.mkdir(self.get_lock_dir())
print('...locked')
return
except OSError as e:
if e.errno != 17:
raise e
def unlock(self):
try:
os.rmdir(self.get_lock_dir())
except OSError as e:
if e.errno != 2:
raise e
print('unlocked')
The Problem
This sometimes happens:
....in base_read
lBytes = os.read(self.iFH,iLen)
OSError: [Errno 11] Resource temporarily unavailable
Sometimes it's fine.
The Magical Solution
I seem to have stopped the problem from happening. Please note this is not me answering my own question. My question is explained in the next section.
I changed the read functions to look more like this and it sorted stuff out:
def base_read(self,iLen):
while not self.ready_for_reading():
import time
print('.')
time.sleep(0.5)
lBytes = ''.encode('utf-8')
while len(lBytes)<iLen:
self.lock()
try:
lBytes += os.read(self.iFH,iLen)
except OSError as e:
if e.errno == 11:
import time
print('.')
time.sleep(0.5)
finally:
self.unlock()
return lBytes
def ready_for_reading(self):
lR,lW,lX = select.select([self.iFH,],[],[],self.iTimeout)
if not lR:
return False
lR,lW,lX = select.select([self.iFHBlocking],[],[],self.iTimeout)
if not lR:
return False
return True
The Question
I'm struggling to find out exactly why it is temporarily unavailable. The two processes cannot access the actual named pipe at the same time due to the locking mechanism (unless I am mistaken?) so is this due to something more fundamental to fifos that my program is not taking into account?
All I really want is an explanation... The solution I found works but it looks like magic. Can anyone offer an explanation?
System
Ubuntu 12.04,
Python3.2.3

I had a similar problem with Java before. Have a look at the accepted answer--- the problem was that I was creating new threads in a loop. I suggest you have a look at the code creating the pipe and make sure you are not creating multiple pipes.