EDIT #2: So guess what, I'm almost there! I am facing what seems to be the last of my problems, well, as long as programming is concerned.
This is actually very interesting, I never ran into such a thing before.
The matter is in the following code, my javascript function. I usually never post for problems that seem very simple to solve, but I really have no idea what's going on here.
The issue seems to be in the first condition of the update function. See the line that says alert('hey'); ? Well, if I erase that line, for some unknown reason, nothing is sent to the action function. Nor to the Arduino, to the console... Just nothing happens. It's absolutely fascinating, as I like to call it. I have no idea. I thought maybe the alert() created some kind of delay that was necessary to read the arduino output, but when i create a delay with setTimeout, nothing happens either. It's incredible.
Just one more time: without the alert, the action function is not called, I checked by making the function print something if it's called. Nothing is printed, nothing. It's just not called. But with the alert, the function is called and the arduino turns on the LED.
Do you have any explanation? Here is my code:
function update(command=0) {
// if command send it
if (command!=0) {
$.getJSON('/action?command='+command);
alert('hey');
}
// read no matter what
$.getJSON('/read', {}, function(data) {
if (data.state != 'failure' && data.content != '') {
$('.notice').text(data.content);
$('.notice').hide().fadeIn('slow');
setTimeout(function () { $('.notice').fadeOut(1000); }, 1500);
}
setTimeout(update, 5000);
});
}
update();
I am attempting to create a web interface accessible from any computer to control my Arduino.
I'm getting closer. One of my problems is that, using the following code, when I press a button to send a command to the Arduino, the Arduino does get it (the LED blinks as configured), then does send a message, and the Python script does retrieve the data, but it does NOT display it properly. The string misses some characters, and index.html is not returned as desired.
Basically, a function is called when a button is pressed, and I need to return the result of the function in a different function than the one the result was generated from.
Here is the code:
# -*- coding: utf-8 -*-
import cherrypy, functools, json, uuid, serial, threading, webbrowser, time
try:
ser = serial.Serial('COM4', 9600)
time.sleep(2)
ser.write('1')
except:
print('Arduino not detected. Moving on')
INDEX_HTML = open('index.html', 'r').read()
def timeout(func, args = (), kwargs = {}, timeout_duration = 10, default = None):
class InterruptableThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.result = default
def run(self):
self.result = func(*args, **kwargs)
it = InterruptableThread()
it.start()
it.join(timeout_duration)
if it.isAlive():
return it.result
else:
return it.result
def get_byte(useless):
return ser.read().encode('Utf-8')
def json_yield(command):
#functools.wraps(command)
def _(self, command):
if (command == 'Turn the LED on'):
ser.write('2')
time.sleep(2)
print('wrote to port')
print('ok, ok')
try:
m = ''
while 1:
print('reading...')
byte = timeout(get_byte, ('none',), timeout_duration = 5)
if byte == '*' or byte == None: break
m = m + byte
content = m
time.sleep(1)
return json.dumps({'state': 'ready', 'content':content})
except StopIteration:
return json.dumps({'state': 'done', 'content': None})
return _
class DemoServer(object):
#cherrypy.expose
def index(self):
return INDEX_HTML
#cherrypy.expose
#json_yield
def yell(self):
yield 'nothing'
#cherrypy.expose
#json_yield
def command(self):
yield 'nothing'
if __name__ == '__main__':
t = threading.Timer(0.5, webbrowser.open, args=('http://localhost:8080',))
t.daemon = True
t.start()
cherrypy.quickstart(DemoServer(), config='config.conf')
First of all, I did'nt think about telling you about that in your previous question, but a while back I wrote a software called pyaler that does exactly what you want (except it did not support long polling requests, as it is (was?) a wsgi limitation).
To answer your question, why don't you make your form a javascript query that sends the action, and gets the result as JSON that you can parse and update your current page with the result? That's way more elegant, simple and 2013...
Sadly I can't really tell, given your code, why you're not getting the result. It's too much over-sophisticated to understand if it really does what you want it to do... KISS!
Avoid doing anything at the module scope, except if you put it in if __name__ == "__main__", or the day you'll want to extend your module by importing it, you'll execute incidentally some code, and it forces making a better design for your code.
You can take the InterruptableThread() class out of your timeout function, and give it the default as a parameter. InterruptableThread(default) and def __init__(self, default): self.result = default. But talking about that part, why do you do such over-sophisticated thing whereas you got a timeout argument you can use in when you create your serial connection?
Here is a slight modification I'd make to your code:
# -*- coding: utf-8 -*-
import cherrypy, functools, json, uuid, serial, threading, webbrowser, time
def arduino_connect(timeout=0):
try:
ser=serial.Serial('COM4', 9600, timeout=timeout)
time.sleep(2)
ser.write('1')
return ser
except:
raise Exception('Arduino not detected. Moving on')
class ArduinoActions(object):
def __init__(self, ser):
self.ser = ser
def get_data(self):
content = ""
while True:
print('reading...')
data = self.ser.read().encode('utf-8')
if not data or data == '*':
return content
content += data
def turn_led_on(self):
ser.write('2')
time.sleep(2)
print('wrote led on to port')
def turn_led_off(self):
ser.write('2') # Replace with the value to tur the led off
time.sleep(2)
print('wrote to led off port')
class DemoServer(ArduinoActions):
def __init__(self, ser):
ArduinoActions.__init__(self, ser)
with open('index.html', 'r') as f:
self.index_template = f.read()
#cherrypy.expose
def index(self):
return self.index_template
#cherrypy.expose
def action(self, command):
state = 'ready'
if command == "on":
content = self.turn_led_on()
elif command == "off":
content = self.turn_led_off()
else:
content = 'unknown action'
state = 'failure'
return {'state': state, 'content': content}
#cherrypy.expose
def read(self):
content = self.get_data()
time.sleep(1)
# set content-type to 'application/javascript'
return content
if __name__ == '__main__':
ser = arduino_connect(5)
# t = threading.Timer(0.5, webbrowser.open, args=('http://localhost:8080',))
# t.daemon = True
# t.start()
cherrypy.quickstart(DemoServer(), config='config.conf')
In your javascript code, you you're calling the yell resource that returns literally nothing.
You'd better make an action method (as I modified the given python code) and a separate read() method.
So you the action method will act on the arduino by writing bytes, thus commands to the arduino, and the
read method will read the output.
As the webserver may create parallel calls to the read/write method on the serial object, and
you can't read in parallel the same object, you may want to make an independant thread, by creating
a new class that inherits from threading.Thread, you implement an infinite loop reading the output
of the serial (by giving the serial object as parameter to that class' __init__ function).
Then you push each new content data to a list, if you want to keep a log of all previous data, or
a Queue.Queue if you want only the last reading returned. Then from ArduinoActions, in the read()
method you just need to return that list that will grow with any new reading from the arduino, and
thus make a log of all the data (or get the last data if you have a queue).
$(function() {
function update(command) {
// if you give a command argument to the function, it will send a command
if (command) {
$.getJSON('/action?command='+command);
}
// then it reads the output
$.getJSON('/read, {}, function(data) {
if (data.state !== 'failure' && data.content !== '') {
$('.notice').text(data.content);
$('.notice').hide().fadeIn('fast');
setTimeout(function () { $('.notice').fadeOut('fast'); }, 1500);
}
// and rearms the current function so it refreshes the value
setTimeout(update(), 2); // you can make the updates less often, you don't need to flood your webserver and anyway the arduino reading are blocking
});
}
update();
});
Always use === or !== in javascript so type don't get coerced. You can call again the function less often
If you do not valuate a javascript argument, it is set to undefined per default.
That's only a little update on what you wrote, it's getting late now, so I hope you'll make something nice out of that!
HTH
Related
For example the first script:
from secondScript import Second
---
""
""
""
while True:
lastResult = <a list> --> I need to send this result to other script
---
My other script
class Second:
def __init__(self):
""
""
""
self.dum = Thread(target=self.func1)
self.dum.deamon = True
self.dum.start()
self.tis = Thread(target=self.func2, args= <a list>)
self.tis.deamon = True
self.tis.start()
def func1(self):
while True:
""
""
""
def func2(self, lastResult):
while True:
print(lastResult)
As a result, I want to send the value which I found in the first script to a infinity thread function in script 2. I can't import first script to second because I am also getting another values from script 2 to script 1.
Edit:
We can think of it like: There is a part of my program that is already running. We can say that I am getting real time images from the camera. While the whole code is running, it also generates a number value continuously and uninterruptedly. All of these operations are done in the 1st file. While the 1st file continues to work, it needs to continuously send this number to the 2nd file. In the second code, 2 different infinite loop functions are running at the same time. In the 1st function, the data from the arduino is constantly being read continuously and uninterruptedly. The 2nd function should print the number which coming from the 1st code. So actually there is nothing I can change in code 1. I am generating the number value. I need to send it to code 2 somehow. I'm not sure how to edit the code you wrote. Any sleep etc. I can't use any interrupt method because in code 1 the camera should work without interruption.
Ok this answers your question, but I changed a bit the structure. I would do it as follows:
EDIT: If you have streams of continuous data like the CameraFeed you mentioned, you can use a Queue with a pattern like this (You don't really need the Second class in this case, you can implement the CameraFeed and CameraDataConsumer in different classes).
If the dum thread does not send data to the tis thread, you can use the send_data method of tis to send data to it through main function and remove the queue from CameraFeed.
from threading import Event, Thread
from queue import Queue, Full
import time
class CameraFeed(Thread):
def __init__(self, queue):
super().__init__()
# This event is used to stop the thread
# Initially it is unset (False)
self._stopped_event = Event()
self._queue = queue
self._data = []
def run(self):
# sample_data is for demo purposes remove it and
# fetch data however you do it
sample_data = iter(range(10**10))
# Loop as long as the stopped event is not set
while not self._stopped_event.is_set():
# Get data from camera this is mock data
data = next(sample_data)
# Put data in the list for data to be sent
self._data.append(data)
# Get the next available item from the list
data = self._data.pop(0)
try:
# This tries to puts in the queue
# if queue is at max capacity raises a Full Exception
self._queue.put_nowait(data)
print('CameraFeed, sent data:', data)
except Full:
# If exception occures, put the data back to list
self._data.insert(0, data)
def stop(self):
# Sets the stopped event, so the thread exits the run loop
self._stopped_event.set()
class CameraDataConsumer(Thread):
def __init__(self, queue):
super().__init__()
self._stopped_event = Event()
self._queue = queue
def run(self):
while not self._stopped_event.is_set():
# Waits for data from queue
data = self._queue.get(block=True)
# If data is None then do nothing
if data is None:
continue
print('CameraConsumer, got data:', data)
def send_data(self, data):
"""Method to send data to this thread from main probably"""
self._queue.put(data, block=True)
def stop(self):
# Set the stopped event flag
self._stopped_event.set()
# Try to put data to queue, to wake up the thread
try:
self._queue.put_nowait(Data(None, EventType.OPERATION))
except Full:
# If queue is full, don't do anything it is probably
# safe to assume that setting the stop flag is sufficient
print('Queue is full')
# Create a Queue with capacity 1000
queue = Queue(maxsize=1000)
dum = CameraFeed(queue)
dum.start()
tis = CameraDataConsumer(queue)
tis.start()
# time.sleep is for demo purposes
time.sleep(1)
tis.stop()
dum.stop()
tis.join()
dum.join()
In first script:
print(lastResult, end='\n', file=sys.stdout, flush=True)
In other script and other thread:
second = Second()
...
for lastResult in sys.stdin:
lastResult = lastResult[:-1]
second.func2(lastResult)
...
I'm working on an automation project with python and an Arduino. I need to run a test that ensures that the Arduino is connected (and not some other device). I have a simple test case that works, but doesn't manage the serial object's memory at all. I've got a solution that I feel should work, however it causes an issue with the serial connection. The result is that the first read returns an empty string. Subsequent reads return the correct value. For my application I need it to read correctly on the first try.
Here is the test code I used. This code works just fine (it's how I know the issue isn't on the Arduino side of things)
ser = serial.Serial('/dev/tty.usbmodem14101', 9600, timeout=1)
ser.flush()
while True:
text = input("Input: ")
ser.write(bytes(text, 'utf-8'))
time.sleep(0.2)
line = ser.readline().decode('utf-8').rstrip()
print(line)
This is my attempt at creating a better way to manage the serial resource using the context manager and the with statement. I tried 2 things. The ArduinoWith class only returns false (and prints and empty string) with no indication of any input on the Arduino. The ArduinoSelf has the same result the first time, but returns True the second time.
import serial
import time
from contextlib import contextmanager
#contextmanager
def arduino_serial_connection(path, rate, timeout=1):
connection = serial.Serial(path, rate, timeout=timeout)
yield connection
connection.close()
class ArduinoWith:
def __init__(self):
pass
def connection_test(self):
try:
with arduino_serial_connection('/dev/tty.usbmodem14101', 9600) as connection:
connection.write(b"S\n")
time.sleep(0.2)
answer = connection.readline().decode('utf-8').rstrip()#connection.read(8)
if answer == "H":
return True
else:
print("Answer: \"" + answer +"\"")
except serial.serialutil.SerialTimeoutException:
print("Timeout")
except Exception:
print("Execption")
return False
class ArduinoSelf:
def __init__(self):
self.ser = serial.Serial('/dev/tty.usbmodem14101', 9600, timeout=1)
def connection_test(self):
try:
self.ser.write(b"S\n")
time.sleep(0.2)
answer = self.ser.readline().decode('utf-8').rstrip()#connection.read(8)
if answer == "H":
return True
else:
print("Answer: \"" + answer +"\"")
except serial.serialutil.SerialTimeoutException:
print("Timeout")
except Exception:
print("Execption")
return False
a1 = ArduinoWith()
print(a1.connection_test())
time.sleep(1)
a2 = ArduinoSelf()
print(a2.connection_test())
time.sleep(1)
print(a2.connection_test())
I'm not exactly sure why running the method a second time on the ArduinoSelf class worked, but it did and that made me think there must be something that I'm not initializing correctly.
I'm writing a game that uses python and webkit, and a webpage is the front-end/GUI. The PC is connected to an Arduino that controls a coin hopper and other i/o. When the Arduino sends 'coinin' over serial, I capture this in a serial-watching thread, then run some javascript on the webpage to 'add' a coin to the game.
For simplicity in troubleshooting, I set up an example that runs a test thread instead of reading serial, but the problem is the same. The thread tries to add a coin every second by running 'addcoin()' on the webpage. If you uncomment the run_javascript() line, the program core dumps.
I came up with a keyboard hack workaround. The test thread, instead of trying to run_javascript() directly, does an os.system call to xdotool to type the letters 'conn' to the program window. That window has a key-event listener, and when it gets the letters 'conn' in keybuffer[], it then runs the desired run_javascript() call to the webpage. If you copy the two files to a folder, and run the python program, you'll see the coin text count up every second (Hit BackSpace to end the program). If you try to run the javascript from the thread instead, you'll see the program core dump.
The question is, is there a better way to do this, without having to use the keyboard hack to run the javascript? Although the hack gets around the problem, it introduces a weakness in the game. You can cheat a coin in by typing 'conn' on the keyboard. I'd like to find some other way to trigger the event, without having to use the keyboard event.
Sample webpage index.htm
<html>
<script language="JavaScript" type="text/javascript">
var mycoins=0;
document.onkeydown = function(evt) {
evt = evt || window.event;
cancelKeypress = (evt.ctrlKey && evt.keyCode == 84);
return false;
};
function addcoin()
{
mycoins+=1;
id('mycoins').innerHTML="You Have "+mycoins.toString()+" coins"
}
function id(myID){return document.getElementById(myID)}
</script>
<html>
<body>
<div id=mycoins>You Have 0 Coins</div>
</body>
</html>
sample python
#!/usr/bin/python
import gi
gi.require_version('Gtk', '3.0')
from gi.repository import Gtk, Gdk
gi.require_version('WebKit2', '4.0')
from gi.repository import WebKit2
import os,time,sys,threading,serial
defaultpath = os.path.dirname(os.path.realpath(__file__))
killthread=False
keybuffer=[]
buffkeys=['c','o','n','h','p','e']
myname=os.path.basename(__file__)
serial_ports=['/dev/ttyUSB0','/dev/ttyUSB1','/dev/ttyACM0','/dev/ttyACM1']
checkserial=True;
class BrowserView:
def __init__(self):
global checkserial
window = Gtk.Window()
window.connect("key-press-event", self._key_pressed, window)
self.view = WebKit2.WebView()
self.view.load_uri('file:///'+defaultpath+'/index.htm')
self.view.connect("notify::title", self.window_title_change)
window.add(self.view)
window.fullscreen()
window.show_all()
'''
######not used for this example#######################################
serial_port=""
for x in serial_ports:
#print 'trying ',x
if os.popen('ls '+x+' >/dev/null 2>&1 ; echo $?').read().strip()=='0':
serial_port=x
break;
baud=9600
if len(serial_port)>1:
self.ser = serial.Serial(serial_port, baud, timeout=0)
else:
self.view.load_uri('file:///'+defaultpath+'/signDOWN.htm?Serial%20Port%20Error|Keno%20will%20auto%20close')
checkserial=False;
if checkserial:
thread = threading.Thread(target=self.read_from_port)
thread.start()
#######################################################################
'''
#####thread test#############
thread = threading.Thread(target=self.testthread)
thread.start()
def testthread(self):
while True:
os.system('xdotool search --name '+myname+' type conn')
#self.view.run_javascript('addcoin()') #causes core dump
if killthread==True:
break;
time.sleep(1)
def read_from_port(self):
while True:
if self.ser.inWaiting()>0:
response=self.ser.readline()
print(response)
if 'coinin' in response:
os.system('xdotool search --name '+myname+' type conn')
#self.view.run_javascript('addcoin()') #causes core dump
if killthread==True:
break;
time.sleep(1)
def checkbuffer(self):
global keybuffer
if 'conn' in ''.join(str(x) for x in keybuffer):
self.view.run_javascript('addcoin()')
keybuffer=[]
def window_title_change(self, widget, param):
if not self.view.get_title():
return
os.chdir(defaultpath)
if self.view.get_title().startswith("pythondiag:::"):
message = self.view.get_title().split(":::",1)[1]
os.system('zenity --notification --text='+message+' --timeout=2')
def _key_pressed(self, widget, event, window):
global keybuffer
mykey=Gdk.keyval_name(event.keyval)
isakey=False
for x in buffkeys:
if mykey==x:
isakey=True;
if isakey:
keybuffer.append(Gdk.keyval_name(event.keyval))
else:
keybuffer=[]
self.checkbuffer()
if mykey == 'BackSpace':
self.myquit()
def myquit(self):
global killthread
killthread=True
try:
self.ser.write('clear\n')
except:
pass
Gtk.main_quit()
if __name__ == "__main__":
BrowserView()
Gtk.main()
Update: This answer is updated for general case, original answer below.
Although GIL allows only one python thread being run at given time,
we know nothing about other thread state at the moment of context switch
(it's just like executing multithreaded programm on a single-core machine.)
That's why you should call any not MT-safe methods from the thread they "belong" to (that includes GTK calls, which "belong" to main event loop).
If you want to call such a function, you should schedule it's execution in the main loop. Probably the easiest approach is to use idle_add. Also note, that idle_add'ed function should
return True or False whether it should be called again later or not, respectively.
Your code shoul look like this:
from gi.repository import GLib
...
class ThreadedWork:
def function(self, arg):
''' function to be called in mainloop'''
if arg:
return GLib.SOURCE_REMOVE
return GLib.SOURCE_CONTINUE
def scheduler(self, function, arg):
''' scheduler (purely for readability issues) '''
GLib.idle_add(function, arg)
def thread_func(self):
''' long long thread function '''
while True:
# Do some long work
# After it is done, schedule execution of mainloop functions.
self.scheduler(self.function, True)
time.sleep(1)
Original answer:
Looks like it's due to run_javascript
being not MT-safe (unlike this method, for example).
from gi.repository import GLib
...
class BrowserView:
def javascript_runner(self, script_name):
GLib.idle_add(self.view.run_javascript, script_name)
def testthread(self):
while True:
os.system('xdotool search --name '+myname+' type conn')
# After long work is done, schedule execution of mainloop functions.
self.javascript_runner('addcoin()')
if killthread: # btw, there is no need to check ==True explicitly
break
time.sleep(1)
I wanted to post the complete test code for anyone looking for a way to run webkit while also running a thread to get information back from the serial port (or any thread), and then do something useful with it. I searched for a solution to this for about week before asking the question, but couldn't find anything that specifically worked with webkit.
If you want to use the serial part, uncomment that section, and comment the testthread section. If you have any questions on how to use this, please ask and I'll do my best to answer them.
run_javascript('your_js_function()') is how python directs the webpage to do something.
The def window_title_change(self, widget, param): function is how you communicate from the webpage back to python. You also have to have the
'self.view.connect("notify::title", self.window_title_change)' line in the BrowserView Class, as shown in the sample code, so python will detect the change, and act on it.
For example, on your webpage, include this function:
function python(x)
{
document.title=""
document.title=x
}
Then to call python from your webpage to do something for you, simply call python like this:
python('pythondiag:::'hello python');
On the python side, you can write any function you need, to do anything you need to do to interact with the system. Webkit is a great solution to use HTML and javascript as a front-end/GUI, and then interact with your PC through python.
Thanks to Alexander Dmitriev's excellent solution to the original problem, here is the complete code with NO CORE DUMP...WOOHOO! I hope this can help others that have had this problem.
Sample webpage index.htm
<html>
<script language="JavaScript" type="text/javascript">
var mycoins=0;
document.onkeydown = function(evt) {
evt = evt || window.event;
cancelKeypress = (evt.ctrlKey && evt.keyCode == 84);
return false;
};
function addcoin()
{
mycoins+=1;
id('mycoins').innerHTML="You Have "+mycoins.toString()+" coins"
}
function id(myID){return document.getElementById(myID)}
</script>
<html>
<body>
<div id=mycoins>You Have 0 Coins</div>
</body>
</html>
sample python
#!/usr/bin/python
import gi
gi.require_version('Gtk', '3.0')
from gi.repository import Gtk, Gdk, GLib
gi.require_version('WebKit2', '4.0')
from gi.repository import WebKit2
import os,time,sys,threading,serial
defaultpath = os.path.dirname(os.path.realpath(__file__))
killthread=False
myname=os.path.basename(__file__)
serial_ports=['/dev/ttyUSB0','/dev/ttyUSB1','/dev/ttyACM0','/dev/ttyACM1']
checkserial=True;
class BrowserView:
def __init__(self):
global checkserial
window = Gtk.Window()
window.connect("key-press-event", self._key_pressed, window)
self.view = WebKit2.WebView()
self.view.load_uri('file:///'+defaultpath+'/index.htm')
self.view.connect("notify::title", self.window_title_change)
window.add(self.view)
window.fullscreen()
window.show_all()
'''
######Uncomment this to use the serial port watcher#####################
serial_port=""
for x in serial_ports:
#print 'trying ',x
if os.popen('ls '+x+' >/dev/null 2>&1 ; echo $?').read().strip()=='0':
serial_port=x
break;
baud=9600
if len(serial_port)>1:
self.ser = serial.Serial(serial_port, baud, timeout=0)
else:
self.view.load_uri('file:///'+defaultpath+'/signDOWN.htm?Serial%20Port%20Error|Keno%20will%20auto%20close')
checkserial=False;
if checkserial:
thread = threading.Thread(target=self.read_from_port)
thread.start()
#########################################################################
'''
#####thread test--comment out to use the serial port watcher#############
thread = threading.Thread(target=self.testthread)
thread.start()
#########################################################################
def javascript_runner(self, script_name):
GLib.idle_add(self.view.run_javascript, script_name)
def testthread(self):
while True:
self.javascript_runner('addcoin()')
if killthread:
break
time.sleep(1)
def read_from_port(self):
while True:
if self.ser.inWaiting()>0:
response=self.ser.readline()
print(response)
if 'coinin' in response:
self.javascript_runner('addcoin()')
if killthread:
break;
time.sleep(1)
def window_title_change(self, widget, param):
if not self.view.get_title():
return
os.chdir(defaultpath)
if self.view.get_title().startswith("pythondiag:::"):
message = self.view.get_title().split(":::",1)[1]
os.system('zenity --notification --text='+message+' --timeout=2')
def _key_pressed(self, widget, event, window):
mykey=Gdk.keyval_name(event.keyval)
print mykey
if mykey == 'BackSpace':
self.myquit()
def myquit(self):
global killthread
killthread=True
try:
self.ser.write('clear\n')
except:
pass
Gtk.main_quit()
if __name__ == "__main__":
BrowserView()
Gtk.main()
I'm working with cherrypy in a server that implements a RESTful like API.
The responses imply some heavy computation that takes about 2 seconds for
request. To do this computations, some data is used that is updated three
times a day.
The data is updated in the background (takes about half hour),
and once it is updated, the references of the new data are passed to
the functions that respond the requests. This takes just a milisecond.
What I need is to be sure that each request is answered either with the
old data or with the new data, but none request processing can take place while the data references are being changed. Ideally, I would like to find a way of buffering incoming request while the data references are changed, and also to ensure that the references are changed after all in-process requests finished.
My current (not) working minimal example is as follows:
import time
import cherrypy
from cherrypy.process import plugins
theData = 0
def processData():
"""Backround task works for half hour three times a day,
and when finishes it publish it in the engine buffer."""
global theData # using global variables to simplify the example
theData += 1
cherrypy.engine.publish("doChangeData", theData)
class DataPublisher(object):
def __init__(self):
self.data = 'initData'
cherrypy.engine.subscribe('doChangeData', self.changeData)
def changeData(self, newData):
cherrypy.engine.log("Changing data, buffering should start!")
self.data = newData
time.sleep(1) #exageration of the 1 milisec of the references update to visualize the problem
cherrypy.engine.log("Continue serving buffered and new requests.")
#cherrypy.expose
def index(self):
result = "I get "+str(self.data)
cherrypy.engine.log(result)
time.sleep(3)
return result
if __name__ == '__main__':
conf = {
'/': { 'server.socket_host': '127.0.0.1',
'server.socket_port': 8080}
}
cherrypy.config.update(conf)
btask = plugins.BackgroundTask(5, processData) #5 secs for the example
btask.start()
cherrypy.quickstart(DataPublisher())
If I run this script, and also open a browser, put localhost:8080 and refresh
the page a lot, I get:
...
[17/Sep/2015:21:32:41] ENGINE Changing data, buffering should start!
127.0.0.1 - - [17/Sep/2015:21:32:41] "GET / HTTP/1.1" 200 7 "...
[17/Sep/2015:21:32:42] ENGINE I get 3
[17/Sep/2015:21:32:42] ENGINE Continue serving buffered and new requests.
127.0.0.1 - - [17/Sep/2015:21:24:44] "GET / HTTP/1.1" 200 7 "...
...
Which means that some requests processing started before and ends after the
data references start or end to being changed. I want to avoid both cases.
Something like:
...
127.0.0.1 - - [17/Sep/2015:21:32:41] "GET / HTTP/1.1" 200 7 "...
[17/Sep/2015:21:32:41] ENGINE Changing data, buffering should start!
[17/Sep/2015:21:32:42] ENGINE Continue serving buffered and new requests.
[17/Sep/2015:21:32:42] ENGINE I get 3
127.0.0.1 - - [17/Sep/2015:21:24:44] "GET / HTTP/1.1" 200 7 "...
...
I searched documentation and the web and find these references that do not completely cover this case:
http://www.defuze.org/archives/198-managing-your-process-with-the-cherrypy-bus.html
How to execute asynchronous post-processing in CherryPy?
http://tools.cherrypy.org/wiki/BackgroundTaskQueue
Cherrypy : which solutions for pages with large processing time
How to stop request processing in Cherrypy?
Update (with a simple solution):
After giving more thought, I think that the question is misleading since it includes some implementation requirements in the question itself, namely: to stop processing and start buffering. While for the problem the requirement can be simplified to: be sure that each request is processed either with the old data or with the new data.
For the later, it is enough to store a temporal local reference of the used data. This reference can be used in all the request processing, and it will be no problem if another thread changes self.data. For python objects, the garbage collector will take care of the old data.
Specifically, it is enough to change the index function by:
#cherrypy.expose
def index(self):
tempData = self.data
result = "I started with %s"%str(tempData)
time.sleep(3) # Heavy use of tempData
result += " that changed to %s"%str(self.data)
result += " but I am still using %s"%str(tempData)
cherrypy.engine.log(result)
return result
And as a result we will see:
[21/Sep/2015:10:06:00] ENGINE I started with 1 that changed to 2 but I am still using 1
I still want to keep the original (more restrictive) question and cyraxjoe answer too, since I find those solutions very useful.
I'll explain two one approaches that will solve the issue.
The first one is Plugin based.
Plugin based Still needs a kind of synchronization. It only works because there is only one BackgroundTask making the modifications (also is just an atomic operation).
import time
import threading
import cherrypy
from cherrypy.process import plugins
UPDATE_INTERVAL = 0.5
REQUEST_DELAY = 0.1
UPDATE_DELAY = 0.1
THREAD_POOL_SIZE = 20
next_data = 1
class DataGateway(plugins.SimplePlugin):
def __init__(self, bus):
super(DataGateway, self).__init__(bus)
self.data = next_data
def start(self):
self.bus.log("Starting DataGateway")
self.bus.subscribe('dg:get', self._get_data)
self.bus.subscribe('dg:update', self._update_data)
self.bus.log("DataGateway has been started")
def stop(self):
self.bus.log("Stopping DataGateway")
self.bus.unsubscribe('dg:get', self._get_data)
self.bus.unsubscribe('dg:update', self._update_data)
self.bus.log("DataGateway has been stopped")
def _update_data(self, new_val):
self.bus.log("Changing data, buffering should start!")
self.data = new_val
time.sleep(UPDATE_DELAY)
self.bus.log("Continue serving buffered and new requests.")
def _get_data(self):
return self.data
def processData():
"""Backround task works for half hour three times a day,
and when finishes it publish it in the engine buffer."""
global next_data
cherrypy.engine.publish("dg:update", next_data)
next_data += 1
class DataPublisher(object):
#property
def data(self):
return cherrypy.engine.publish('dg:get').pop()
#cherrypy.expose
def index(self):
result = "I get " + str(self.data)
cherrypy.engine.log(result)
time.sleep(REQUEST_DELAY)
return result
if __name__ == '__main__':
conf = {
'global': {
'server.thread_pool': THREAD_POOL_SIZE,
'server.socket_host': '127.0.0.1',
'server.socket_port': 8080,
}
}
cherrypy.config.update(conf)
DataGateway(cherrypy.engine).subscribe()
plugins.BackgroundTask(UPDATE_DELAY, processData).start()
cherrypy.quickstart(DataPublisher())
In this version the synchronizations comes by the fact that both read & write operations are executed on the cherrypy.engine thread. Everything is abstracted on the plugin DataGateway you just operated publishing into the engine.
The second approach is by using an Event a threading.Event object. This is a more manual approach with the added benefit that it's probably going to be faster given that the reads are faster because it's doesn't execute over the cherrypy.engine thread.
threading.Event based (a.k.a. manual)
import time
import cherrypy
import threading
from cherrypy.process import plugins
UPDATE_INTERVAL = 0.5
REQUEST_DELAY = 0.1
UPDATE_DELAY = 0.1
THREAD_POOL_SIZE = 20
next_data = 1
def processData():
"""Backround task works for half hour three times a day,
and when finishes it publish it in the engine buffer."""
global next_data
cherrypy.engine.publish("doChangeData", next_data)
next_data += 1
class DataPublisher(object):
def __init__(self):
self._data = next_data
self._data_readable = threading.Event()
cherrypy.engine.subscribe('doChangeData', self.changeData)
#property
def data(self):
if self._data_readable.is_set():
return self._data
else:
self._data_readable.wait()
return self.data
#data.setter
def data(self, value):
self._data_readable.clear()
time.sleep(UPDATE_DELAY)
self._data = value
self._data_readable.set()
def changeData(self, newData):
cherrypy.engine.log("Changing data, buffering should start!")
self.data = newData
cherrypy.engine.log("Continue serving buffered and new requests.")
#cherrypy.expose
def index(self):
result = "I get " + str(self.data)
cherrypy.engine.log(result)
time.sleep(REQUEST_DELAY)
return result
if __name__ == '__main__':
conf = {
'global': {
'server.thread_pool': THREAD_POOL_SIZE,
'server.socket_host': '127.0.0.1',
'server.socket_port': 8080,
}
}
cherrypy.config.update(conf)
plugins.BackgroundTask(UPDATE_INTERVAL, processData).start()
cherrypy.quickstart(DataPublisher())
I've added some niceties with the #property decorator but the real gist is on the threading.Event and the fact that the DataPublisher object is shared among the worker threads.
I also added the thread pool configuration required to increase the thread pool size in both examples. The default is 10.
As a way to test what I just said you can execute this Python 3 script (if you don't have python3 now you have a pretext to install it) it will do a 100 requests more or less concurrently given the thread pool.
Test script
import time
import urllib.request
import concurrent.futures
URL = 'http://localhost:8080/'
TIMEOUT = 60
DELAY = 0.05
MAX_WORKERS = 20
REQ_RANGE = range(1, 101)
def load_url():
with urllib.request.urlopen(URL, timeout=TIMEOUT) as conn:
return conn.read()
with concurrent.futures.ThreadPoolExecutor(max_workers=MAX_WORKERS) as executor:
futures = {}
for i in REQ_RANGE:
print("Sending req {}".format(i))
futures[executor.submit(load_url)] = i
time.sleep(DELAY)
results = []
for future in concurrent.futures.as_completed(futures):
try:
data = future.result().decode()
except Exception as exc:
print(exc)
else:
results.append((futures[future], data))
curr_max = 0
for i, data in sorted(results, key=lambda r: r[0]):
new_max = int(data.split()[-1])
assert new_max >= curr_max, "The data was not updated correctly"
print("Req {}: {}".format(i, data))
curr_max = new_max
The way that you determined that you have a problem based on the log, it's not trust worthy for this kind of problems. Specially given that you don't have control over the time on which the request gets logged on the "access" log. I couldn't make it fail your code with my test code but there is indeed a race condition in the general case, in this example it should work all the time because the code is just making an atomic operation. Just one attribute assignment periodically from a central point.
I hope the code is self explanatory in case that you have a question leave a comment.
EDIT: I edited the Plugin based approach because it only works because there is just one place that is executing the plugin if you create another background task that updates the data then it could have problems when you do something more than just an assignment. Regardless the code could be what you are looking for if you will update from one BackgroundTask.
The next script I'm using is used to listen to IMAP connection using IMAP IDLE and depends heavily on threads. What's the easiest way for me to eliminate the treads call and just use the main thread?
As a new python developer I tried editing def __init__(self, conn): method but just got more and more errors
A code sample would help me a lot
#!/usr/local/bin/python2.7
print "Content-type: text/html\r\n\r\n";
import socket, ssl, json, struct, re
import imaplib2, time
from threading import *
# enter gmail login details here
USER="username#gmail.com"
PASSWORD="password"
# enter device token here
deviceToken = 'my device token x x x x x'
deviceToken = deviceToken.replace(' ','').decode('hex')
currentBadgeNum = -1
def getUnseen():
(resp, data) = M.status("INBOX", '(UNSEEN)')
print data
return int(re.findall("UNSEEN (\d)*\)", data[0])[0])
def sendPushNotification(badgeNum):
global currentBadgeNum, deviceToken
if badgeNum != currentBadgeNum:
currentBadgeNum = badgeNum
thePayLoad = {
'aps': {
'alert':'Hello world!',
'sound':'',
'badge': badgeNum,
},
'test_data': { 'foo': 'bar' },
}
theCertfile = 'certif.pem'
theHost = ('gateway.push.apple.com', 2195)
data = json.dumps(thePayLoad)
theFormat = '!BH32sH%ds' % len(data)
theNotification = struct.pack(theFormat, 0, 32,
deviceToken, len(data), data)
ssl_sock = ssl.wrap_socket(socket.socket(socket.AF_INET,
socket.SOCK_STREAM), certfile=theCertfile)
ssl_sock.connect(theHost)
ssl_sock.write(theNotification)
ssl_sock.close()
print "Sent Push alert."
# This is the threading object that does all the waiting on
# the event
class Idler(object):
def __init__(self, conn):
self.thread = Thread(target=self.idle)
self.M = conn
self.event = Event()
def start(self):
self.thread.start()
def stop(self):
# This is a neat trick to make thread end. Took me a
# while to figure that one out!
self.event.set()
def join(self):
self.thread.join()
def idle(self):
# Starting an unending loop here
while True:
# This is part of the trick to make the loop stop
# when the stop() command is given
if self.event.isSet():
return
self.needsync = False
# A callback method that gets called when a new
# email arrives. Very basic, but that's good.
def callback(args):
if not self.event.isSet():
self.needsync = True
self.event.set()
# Do the actual idle call. This returns immediately,
# since it's asynchronous.
self.M.idle(callback=callback)
# This waits until the event is set. The event is
# set by the callback, when the server 'answers'
# the idle call and the callback function gets
# called.
self.event.wait()
# Because the function sets the needsync variable,
# this helps escape the loop without doing
# anything if the stop() is called. Kinda neat
# solution.
if self.needsync:
self.event.clear()
self.dosync()
# The method that gets called when a new email arrives.
# Replace it with something better.
def dosync(self):
print "Got an event!"
numUnseen = getUnseen()
sendPushNotification(numUnseen)
# Had to do this stuff in a try-finally, since some testing
# went a little wrong.....
while True:
try:
# Set the following two lines to your creds and server
M = imaplib2.IMAP4_SSL("imap.gmail.com")
M.login(USER, PASSWORD)
M.debug = 4
# We need to get out of the AUTH state, so we just select
# the INBOX.
M.select("INBOX")
numUnseen = getUnseen()
sendPushNotification(numUnseen)
typ, data = M.fetch(1, '(RFC822)')
raw_email = data[0][1]
import email
email_message = email.message_from_string(raw_email)
print email_message['Subject']
#print M.status("INBOX", '(UNSEEN)')
# Start the Idler thread
idler = Idler(M)
idler.start()
# Sleep forever, one minute at a time
while True:
time.sleep(60)
except imaplib2.IMAP4.abort:
print("Disconnected. Trying again.")
finally:
# Clean up.
#idler.stop() #Commented out to see the real error
#idler.join() #Commented out to see the real error
#M.close() #Commented out to see the real error
# This is important!
M.logout()
As far as I can tell, this code is hopelessly confused because the author used the "imaplib2" project library which forces a threading model which this code then never uses.
Only one thread is ever created, which wouldn't need to be a thread but for the choice of imaplib2. However, as the imaplib2 documentation notes:
This module presents an almost identical API as that provided by the standard python library module imaplib, the main difference being that this version allows parallel execution of commands on the IMAP4 server, and implements the IMAP4rev1 IDLE extension. (imaplib2 can be substituted for imaplib in existing clients with no changes in the code, but see the caveat below.)
Which makes it appear that you should be able to throw out much of class Idler and just use the connection M. I recommend that you look at Doug Hellman's excellent Python Module Of The Week for module imaplib prior to looking at the official documentation. You'll need to reverse engineer the code to find out its intent, but it looks to me like:
Open a connection to GMail
check for unseen messages in Inbox
count unseen messages from (2)
send a dummy message to some service at gateway.push.apple.com
Wait for notice, goto (2)
Perhaps the most interesting thing about the code is that it doesn't appear to do anything, although what sendPushNotification (step 4) does is a mystery, and the one line that uses an imaplib2 specific service:
self.M.idle(callback=callback)
uses a named argument that I don't see in the module documentation. Do you know if this code ever actually ran?
Aside from unneeded complexity, there's another reason to drop imaplib2: it exists independently on sourceforge and PyPi which one maintainer claimed two years ago "An attempt will be made to keep it up-to-date with the original". Which one do you have? Which would you install?
Don't do it
Since you are trying to remove the Thread usage solely because you didn't find how to handle the exceptions from the server, I don't recommend removing the Thread usage, because of the async nature of the library itself - the Idler handles it more smoothly than a one thread could.
Solution
You need to wrap the self.M.idle(callback=callback) with try-except and then re-raise it in the main thread. Then you handle the exception by re-running the code in the main thread to restart the connection.
You can find more details of the solution and possible reasons in this answer: https://stackoverflow.com/a/50163971/1544154
Complete solution is here: https://www.github.com/Elijas/email-notifier