Related
Current in my project, I have a layer of models as such:
Sensor class: which stores different sensors
Entry class: which stores whenever a new data entry happens
Data class: which stores data pairs for each entry.
Generally, that additional class was used since I have different fields for different sensors and I wanted to store them in one database, so how I was doing that was through two methods, one for entry to get the data for it, and one in the sensor as follows:
class Data(models.Model):
key = models.CharField(max_length=50)
value = models.CharField(max_length=50)
entry = models.ForeignKey(Entry, on_delete=CASCADE, related_name="dataFields")
def __str__(self):
return str(self.id) + "-" + self.key + ":" + self.value
class Entry(models.Model):
time = models.DateTimeField()
sensor = models.ForeignKey(Sensor, on_delete=CASCADE, related_name="entriesList")
def generateFields(self, field=None):
"""
Generate the fields by merging the Entry and data fields.
Outputs a dictionary which contains all fields.
"""
if field is None:
field_set = self.dataFields.all()
else:
field_set = self.dataFields.filter(key=field)
output = {"timestamp": self.time.timestamp()}
for key, value in field_set.values_list("key", "value"):
try:
floated = float(value)
isnan = math.isnan(floated)
except (TypeError, ValueError):
continue
if isnan:
return None
else:
output[key] = floated
return output
class Sensor(models.Model):
.
.
.
def generateData(self, fromTime, toTime, field=None):
fromTime = datetime.fromtimestamp(fromTime)
toTime = datetime.fromtimestamp(toTime)
entries = self.entriesList.filter(time__range=(toTime, fromTime)).order_by(
"time"
)
output = []
for entry in entries:
value = entry.generateFields(field)
if value is not None:
output.append(value)
return output
After trying to troubleshoot the issues of time (as running this query for a ~5000-10000 entries took too long, almost 10 seconds!), i found that most of the time (about 95%) was spent on the method for generateFields(), I've been looking at options for caching it (by using cached_property), using different methods but none have really worked so far.
Is there a method to store the results of generateFields() in the database automatically upon saving the model perhaps? Or possibly just saving the results of the reverse-query self.dataFields.all()? I can figure out that's the main culprit since for 5000 entries, there are 25000 data fields on average at least.
(Thanks to Jacinator for the notes and improvements) above is the code after Jacinator's changes, but the issue (and the question) still stands at large, as caching the fields would speed up the process by almost 25-50 times(!!) which is critical when my actual dataset can be much larger (1~ minute to run a query isnt really acceptable)
I think this is how I would consider writing generateFields.
class Entry(models.Model):
...
def generateFields(self, field=None):
"""
Generate the fields by merging the Entry and data fields.
Outputs a dictionary which contains all fields.
"""
if field is None:
field_set = self.dataFields.all()
else:
field_set = self.dataFields.filter(key=field)
output = {"timestamp": self.time.timestamp()}
for key, value in field_set.values_list("key", "value"):
try:
floated = float(value)
isnan = math.isnan(floated)
except (TypeError, ValueError):
continue
if isnan:
return None
else:
output[key] = floated
return output
First, I'm going to avoid comparing the field (if it's provided) in Python. I can use the queryset .filter to pass that off to the SQL.
if field is None:
field_set = self.dataFields.all()
else:
field_set = self.dataFields.filter(key=field)
Second, I'm using QuerySet.values_list to retrieve the values from the entries. I could be wrong (please correct me if so), but I think this also passes off the attribute retrieval to the SQL. I don't actually know if it's faster, but I suspect it might be.
for key, value in field_set.values_list("key", "value"):
I've restructured the try/except block, but that has less to do with increasing the speed and more to do with making it explicit what errors are being caught and what lines are raising them.
try:
floated = float(value)
isnan = math.isnan(floated)
except (TypeError, ValueError):
continue
The lines outside of the try/except now should be problem free.
if isnan:
return None
else:
output[key] = floated
I'm a little unfamiliar with QuerySet.prefetch_related, but I think adding it to this line would also help.
entries = self.entriesList.filter(time__range=(toTime, fromTime)).order_by(
"time").prefetch_related("dataFields")
While it is not the perfect solution, below is the approach I am using at the moment (I am still looking for a better solution, but for this case I believe this might be useful to others who face the exact same situation as me here.)
I am using django-picklefield to store the cachedData, declaring it as:
from picklefield.fields import PickledObjectField
.
.
.
class Entry(models.Model):
time = models.DateTimeField()
sensor = models.ForeignKey(Sensor, on_delete=CASCADE, related_name="entriesList")
cachedValues = PickledObjectField(null=True)
Next, I am adding a property to both generate the value, and return it as:
#property
def data(self):
if self.cachedValues is None:
fields = self.generateFields()
self.cachedValues = fields
self.save()
return fields
else:
return self.cachedValues
Normaly, this field will be set automatically when new data is added, however since I have a large amount of data already present, while I can wait until its accessed (as future access will be much faster), i decided to quick-index it by running this:
def mass_set(request):
clear = lambda: os.system("clear")
entries = Entry.objects.all()
length = len(entries)
for count, entry in enumerate(entries):
_ = entry.data
clear()
print(f"{count}/{length} done")
Finally, below are the benchmarks for a set of 2230 fields running on my local development machine, measuring the main loop as such:
def generateData(self, fromTime, toTime, field=None):
fromTime = datetime.fromtimestamp(fromTime)
toTime = datetime.fromtimestamp(toTime)
# entries = self.entriesList.filter().order_by('time')
entries = self.entriesList.filter(time__range=(toTime, fromTime)).order_by(
"time"
)
output = []
totalTimeLoop = 0
totalTimeGenerate = 0
stage1 = time.time()
stage2 = time.time()
for entry in entries:
stage1 = time.time()
totalTimeLoop += stage1 - stage2
# Value = entry.generateFields(field)
value = entry.data
stage2 = time.time()
totalTimeGenerate += stage2 - stage1
if value is not None:
output.append(value)
print(f"Total time spent generating the loop: {totalTimeLoop}")
print(f"Total time spent creating the fields: {totalTimeGenerate}")
return output
Before:
Loop Generation Time: 0.1659650
Fields Generation Time: 3.1726377
After:
Loop Generation Time: 0.1614456
Fields Generation Time: 0.0032608
About a thousand-fold decrease in fields generation time, and in overall time a 20x speed increase
As for the cons, there are two mainly:
While applying it to my dataset currently (with 167 thousand fields), it takes a considerable amount of time to update, it took around 23 minutes on my local machine, and is expected to take about an hour or two on live servers. However this is a one-time process as all future entries will be added automatically with minimum performance effect with adding the following lines:
_ = newEntry.data
newEntry.save()
The other is database size, while went from 100.8 MB to 132.9 MB, a significant increase of 31% that can be problematic for a database a few scales larger than mine, but a good enough solution small/medium DBs where storage is less important than speed
I've been trying to create a simple linked list implementation in Python as a code exercise and, although I have most of the stuff working (inserting, removing, pretty print, swapping the content of two nodes), I've been stuck on swapping two nodes for a few days.
I've looked around on the internet and most people seem to recommend deleting/inserting the nodes or swapping the data. Both are very fine and functional options but I wanted to challenge myself and see if I could swap the nodes the "correct" way.
Ideally I would like to have a generic function that can handle all edge cases (moving to the begin, end and swapping random nodes). This has proven to be way more challenging than I expected.
I've experimented a bit with pen and paper and search around and I found the following discussion and example implementation:
Discussion about swapping in C
Example implementation in C
The issue I run into is that my node1.next and my node2.prev are swapped and that my node2.next refers to itself and not to the next node in the list.
The comment on the page of the example implementation specifically addresses this problem and mentions that it should not happen with his implementation.
I can't seem to figure out what I've done wrong. I guess I could "cheat" and force them to take the correct values at the end but that gives a lot of problems when the node is the first/last.
__author__ = 'laurens'
from django.core.exceptions import ObjectDoesNotExist
import logging
import copy
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
class DoublyLinkedList(object):
"""This class implements a basic doubly linked list in Django, it depends
on a Django model with the following field:
id : PK
prev: integer previous node
data_field: Foreign key
next: integer next node
the prev and next fields don't need to be self-referencing
When instantiating the class you have to link this class to a Django model
and specify a data field. The data field can link to a foreign key
or contain data
"""
def __init__(self, doubly_linked_list_model, data_field):
self.doubly_linked_list_model = doubly_linked_list_model
self.data_field = data_field
def get_node_from_node_id(self, node_id=None):
"""This function returns the node associated with a certain node_id"""
if node_id is None:
node = None
else:
try:
node = self.doubly_linked_list_model.get(id=node_id)
except ObjectDoesNotExist:
node = None
return node
#staticmethod
def _update_node(node, prev=None, next=None):
node.prev = prev
node.next = next
logger.debug('updating node: %s', node.id)
logger.debug('node.prev = %s', node.prev)
logger.debug('node.next = %s', node.next)
try:
node.save()
except Exception as e: #Todo: specify this
logger.debug('Error saving node: %s', node.id)
def move_node(self, node1=None, node2=None):
"""
This function swaps the position of node1 with the position of node2
"""
#swapping two nodes!
logger.debug('Swapping two random nodes!: %s, %s', node1.id, node2.id)
# Swapping next nodes
logger.debug('Swapping next node')
tmp = copy.deepcopy(node1.next)
self._update_node(node=node1,
prev=node1.prev,
next=node2.next)
#Todo: Check if tmp changes or is stored as a copy
self._update_node(node=node2,
prev=node2.prev,
next=tmp)
if node1.next is not None:
logger.debug('Connect the next node to node 1')
node_next = self.get_node_from_node_id(node1.next)
self._update_node(node=node_next,
prev=node1.id,
next=node_next.next)
if node2.next is not None:
logger.debug('Connect the next node to node 2')
node_next = self.get_node_from_node_id(node2.next)
self._update_node(node=node_next,
prev=node2.id,
next=node_next.next)
logger.debug('Swap prev nodes')
tmp = copy.deepcopy(node1.prev)
self._update_node(node=node1,
prev=node2.prev,
next=node1.next)
self._update_node(node=node2,
prev=tmp,
next=node2.next)
# Connect the node before node1 to node 1
if node1.prev is not None:
logger.debug('Connect the prev to node 1')
node_prev = self.get_node_from_node_id(node1.prev)
self._update_node(node=node_prev,
prev=node_prev.prev,
next=node1.id)
# Connect the node before node2 to node 2
if node2.prev is not None:
logger.debug('Connect the prev to node 2')
node_prev = self.get_node_from_node_id(node2.prev)
self._update_node(node=node_prev,
prev=node_prev.prev,
next=node2.id)
The _update_node function does nothing more than taking my input and committing it to the database; it can handle None values.
get_node_from_node_id takes an integer as input and returns the node object associated with it. I use it so that I don't have to work with self-referencing foreign keys (is that the correct term?) in the database, for now I would like to continue working this way. Once I have this working I'll move on to fixing it in the database in the correct way.
tip: I get answers much more quickly when I provide a minimal, complete, verifiable example (MCVE), also known as a short, self-contained, compilable example (SSCCE).
Your example code fails to demonstrate the problem, making it impossible for us to help you.
Please make it easy for us to help you.
I ran your example code, but I didn't see any output.
The issue I run into is that ... that my node2.next refers to itself and
not to the next node in the list.
Why is node2.next referring to node2 a problem?
As far as I can tell, the part of the code you gave us so far works fine.
Some of the most difficult debugging sessions I've ever had ended only when I realized that everything was actually working correctly, that the "bug" I thought I was hunting didn't actually exist.
Yes, there is an intermediate step halfway through the algorithm where a node refers to itself, which may seem obviously wrong.
But then the second half of the algorithm makes further changes.
By the time the algorithm finishes,
the "next" chain correctly runs all the way from the beginning to the end, and
the "forward" chain correctly runs in the reverse order as the "next" chain, from the end to the beginning, and
the order of those chains is similar to the original order, except that node1 and node2 have swapped logical position (but are still in the same physical position).
Every node points to 2 other nodes (or to the NULL node), never to itself.
Isn't that what you wanted?
When I ran the following test script, I get lots of output --
but I don't see any problems in the output.
(I used a simple array with integer indexes rather than true pointers or references to make the code shorter and easier to debug compared to a SQL database).
Would you mind pointing out which particular line of output is not what you expected, and spelling out what you expected that line of output to actually say?
#!/usr/bin/env python
# https://stackoverflow.com/questions/24610889/trying-to-write-a-swap-position-function-for-a-doubly-linked-list
# Is this the shortest possible program that exhibits the bug?
# Before running this probram, you may need to install
# sudo apt-get install python-django
# 2015-03-12: David added some test scaffolding
# 2014-07-07: Laurens posted to StackOverflow
__author__ = 'laurens'
from django.core.exceptions import ObjectDoesNotExist
import logging
import copy
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
class DoublyLinkedList(object):
"""This class implements a basic doubly linked list in Django, it depends
on a Django model with the following field:
id : PK
prev: integer previous node
data_field: Foreign key
next: integer next node
the prev and next fields don't need to be self-referencing
When instantiating the class you have to link this class to a Django model
and specify a data field. The data field can link to a foreign key
or contain data
"""
def __init__(self, doubly_linked_list_model, data_field):
self.doubly_linked_list_model = doubly_linked_list_model
self.data_field = data_field
def get_node_from_node_id(self, node_id=None):
"""This function returns the node associated with a certain node_id"""
if node_id is None:
node = None
else:
try:
node = self.doubly_linked_list_model.get(id=node_id)
except ObjectDoesNotExist:
node = None
return node
#staticmethod
def _update_node(node, prev=None, next=None):
node.prev = prev
node.next = next
logger.debug('updating node: %s', node.id)
logger.debug('node.prev = %s', node.prev)
logger.debug('node.next = %s', node.next)
try:
node.save()
except Exception as e: #Todo: specify this
logger.debug('Error saving node: %s', node.id)
def move_node(self, node1=None, node2=None):
"""
This function swaps the position of node1 with the position of node2
"""
#swapping two nodes!
logger.debug('Swapping two random nodes!: %s, %s', node1.id, node2.id)
# Swapping next nodes
logger.debug('Swapping next node')
tmp = copy.deepcopy(node1.next)
self._update_node(node=node1,
prev=node1.prev,
next=node2.next)
#Todo: Check if tmp changes or is stored as a copy
self._update_node(node=node2,
prev=node2.prev,
next=tmp)
if node1.next is not None:
logger.debug('Connect the next node to node 1')
node_next = self.get_node_from_node_id(node1.next)
self._update_node(node=node_next,
prev=node1.id,
next=node_next.next)
if node2.next is not None:
logger.debug('Connect the next node to node 2')
node_next = self.get_node_from_node_id(node2.next)
self._update_node(node=node_next,
prev=node2.id,
next=node_next.next)
logger.debug('Swap prev nodes')
tmp = copy.deepcopy(node1.prev)
self._update_node(node=node1,
prev=node2.prev,
next=node1.next)
self._update_node(node=node2,
prev=tmp,
next=node2.next)
# Connect the node before node1 to node 1
if node1.prev is not None:
logger.debug('Connect the prev to node 1')
node_prev = self.get_node_from_node_id(node1.prev)
self._update_node(node=node_prev,
prev=node_prev.prev,
next=node1.id)
# Connect the node before node2 to node 2
if node2.prev is not None:
logger.debug('Connect the prev to node 2')
node_prev = self.get_node_from_node_id(node2.prev)
self._update_node(node=node_prev,
prev=node_prev.prev,
next=node2.id)
global_test_array = []
obfuscation = 0xaa
class trivial_test_node_class(object):
def __init__(self, prev, id, next, data):
print "initializing test class."
# self.stuff = [ ["first", 0, 1], ["second", 0, 1] ]
self.id = id
self.prev = prev
self.next = next
self.data = data
def something(self):
print "something"
def save(self):
id = self.id
global_test_array[id] = self
def __repr__(self):
# print self.prev, self.id, self.next, self.data
the_string = "%s(%r)\n" % (self.__class__, self.__dict__)
return the_string
class trivial_test_list_model_class(object):
def __init__(self):
print "initializing test class."
#self.stuff = [ ["first", 0, 1], ["second", 0, 1] ]
self.stuff = [ 0 for i in xrange(0,10) ]
data = 'a'
for i in xrange(1,10):
self.stuff[i] = trivial_test_node_class(i-1,i,i+1,data);
data += 'r' # talk like a pirate day
self.stuff[-1].next = 0 # use 0 as NULL id.
global global_test_array
global_test_array = self.stuff
def something(self):
print "something"
def get(self,id):
return self.stuff[id]
def __repr__(self):
# for i in xrange(1,10):
# print self.stuff[i]
the_string = "%s(%r)" % (self.__class__, self.__dict__)
return the_string
if __name__ == '__main__':
# test code that only gets run when this file is run directly,
# not when this file is imported from some other python script.
print "Hello, world."
trivial_test_model = trivial_test_list_model_class()
print trivial_test_model
testll = DoublyLinkedList( trivial_test_model, "data" )
left_node = trivial_test_model.get(3)
right_node = trivial_test_model.get(4)
testll.move_node( left_node, right_node )
print trivial_test_model
# recommended by http://wiki.python.org/moin/vim
# vim: tabstop=8 expandtab shiftwidth=4 softtabstop=4 :
i'm having trouble creating an insert function with the following parameters. The insert function should take in a priority queue, and an element and inserts it using the priority rules -
The priority queue will take a series of tasks and order them
based on their importance. Each task has an integer priority from 10 (highest priority) to 1
(lowest priority). If two tasks have the same priority, the order should be based on the order
they were inserted into the priority queue (earlier first).
So, as of right now i've created the following code to initialize some of the things needed...
class Tasks():
__slots__ = ('name', 'priority')
def __init__(bval):
bval.name = myName
bval.priority = myPriority
return bval
class PriorityQueue():
__slots__ = ('queue', 'element')
def __init__(aval):
aval.queue = queue
aval.element = element
return aval
The code i'm trying to write is insert(element, queue): which should insert the elements using the priority queue. Similarly, myPriorty is an integer from 1 to 10.
Similarly can I do the following to insure that I create a priority from 1 to 10...
def __init__(bval , myPriority = 10):
bval.priority = myPriority
bval.pq = [[] for priority in range(bval.myPriority)]
so that I can replace myPriority in the insert task with bval.pq
Why are you trying to re-invent the wheel?
from Queue import PriorityQueue
http://docs.python.org/2/library/queue.html?highlight=priorityqueue#Queue.PriorityQueue
The lowest valued entries are retrieved first (the lowest valued entry is the one returned by sorted(list(entries))[0]). A typical pattern for entries is a tuple in the form:
(priority_number, data).
I use such a module to communicate between the UI and a background polling thread.
READ_LOOP = 5
LOW_PRI = 3
MED_PRI = 2
HI_PRI = 1
X_HI_PRI = 0
and then something like this:
CoreGUI.TX_queue.put((X_HI_PRI,'STOP',[]))
Note that there is a Queue. If you are okay with it being synchronized, I would use that.
Otherwise, you should use a heap to maintain your queue. See Python documentation with an example of that.
From a great book "Modern Python Standard Library Cookbook" by Alessandro Molina
Heaps are a perfect match for everything that has priorities, such as
a priority queue:
import time
import heapq
class PriorityQueue:
def __init__(self):
self._q = []
def add(self, value, priority=0):
heapq.heappush(self._q, (priority, time.time(), value))
def pop(self):
return heapq.heappop(self._q)[-1]
Example:
>>> def f1(): print('hello')
>>> def f2(): print('world')
>>>
>>> pq = PriorityQueue()
>>> pq.add(f2, priority=1)
>>> pq.add(f1, priority=0)
>>> pq.pop()()
hello
>>> pq.pop()()
world
A deque (from collections import deque) is the python implementation of a single queue. You can add items to one end and remove them from the other. If you have a deque for each priority level, you can add to the priority level you want.
Together, it looks a bit like this:
from collections import deque
class PriorityQueue:
def __init__(self, priorities=10):
self.subqueues = [deque() for _ in range(levels)]
def enqueue(self, priorty, value):
self.subqueues[priority].append(value)
def dequeue(self):
for queue in self.subqueues:
try:
return queue.popleft()
except IndexError:
continue
I am using the following method to iterate over all the nodes of a wxpython treectrl.
def get_desired_parent(self, name, selectednode = None):
if selectednode == None:
selectednode = self.treeCtrl.RootItem
# First perform the action on the first object separately
childcount = self.treeCtrl.GetChildrenCount(selectednode, False)
if childcount == 0:
return None
(item,cookie) = self.treeCtrl.GetFirstChild(selectednode)
if self.treeCtrl.GetItemText(item) == name:
return item
while childcount > 1:
childcount = childcount - 1
# Then iterate over the rest of objects
(item,cookie) = self.treeCtrl.GetNextChild(item,cookie)
if self.treeCtrl.GetItemText(item) == name:
return item
return None
This problem of excess code becomes even more apparent when I am iterating inside the structure recursively.
Is there another way of performing the same actions in more compact manner, to make my code more concise / pythonic.
You could use a function that is inside this one (in its namespace only) that will check if it matches the conditiin or not. If it does return the item if it doesn't, continue.
Otherwise you could check your condition just after the while line. This way the item variable will be defined by the first child before the loop and evaluated like any other.
Still another way: (or a mix of the two)
(child, cookie) = self.GetFirstChild(item)
while child.IsOk():
do_something(child)
(child, cookie) = self.GetNextChild(item, cookie)
Here a full example that traverses the tree going depth first. The function was bound to the right button.
def OnRightDown(self, event):
def showChildren(item,cookie):
# functions goes recursively down the tree
if item.IsOk():
child, cookie = self.tree.GetFirstChild(item)
while child.IsOk():
child, cookie = self.tree.GetNextChild(child, cookie)
if child:
print(self.tree.GetItemText(child)) #show child label name
showChildren(child,cookie)
pt = event.GetPosition()
item, flags = self.tree.HitTest(pt)
if item:
print(self.tree.GetItemText(item)) #show parent label name
showChildren(item,0) #iterate depth first into the tree
The best way to make your code highly readable here is to make it short and highly functional.
If you need to iterate through all the tree items and do so through depth first. Here's that as a single quick function. Hand it a function that gets each item, and where you start (usually self.root). It's also quite reusable since you might be doing this a lot.
def depth_first_tree(self, funct, item):
(child, cookie) = self.tree.GetFirstChild(item)
while child.IsOk():
self.depth_first_tree(funct, child)
funct(child)
(child, cookie) = self.tree.GetNextChild(item, cookie)
I am new to Django and I am trying to build a blog myself. I'm trying to create a feature I've seen implemented in Drupal using the nodequeue module.
What I want to do is to be able to create queues of objects, for example, queues of blog posts. Below, I describe how I imagine the queues to work:
the size of each queue should be user-defined.
the date an object is added to a queue should be recorded.
I would like to be able to define the order of the items that belong to each queue (but I think this would be very difficult).
if the queue is full, the addition of an extra item should discard the oldest item of the queue.
An example of how such a feature would be useful is the creation of a featured posts queue.
My current knowledge does not allow me to figure out what would be the right way to do it. I would appreciate any pointers.
Thanks in advance
Here's one approach:
import collections, datetime, itertools
class nodequeue(object):
def __init__(self, N):
self.data = collections.deque(N * [(None, None)])
def add(self, anobj):
self.data.popleft()
self.data.push((anobj, datetime.datetime.now())
def __iter__(self):
it = iter(self.data)
return it.dropwhile(lambda x: x[1] is None, self.data)
This ignores the "ordering" desiderata, but that wouldn't be too hard to add, e.g.:
class nodequeueprio(object):
def __init__(self, N):
self.data = collections.deque(N * [(None, None, None)])
def add(self, anobj, prio):
self.data.popleft()
self.data.push((anobj, datetime.datetime.now(), prio)
def __iter__(self):
it = iter(self.data)
return sorted(it.dropwhile(lambda x: x[1] is None, self.data),
key=operator.itemgetter(2))
I think that prefilling the queue with placeholder Nones simplifies the code because add can always drop the leftmost (oldest or None) item before adding the new thingy -- even though __iter__ must then remove the placeholders, that's not too bad.
Alex's approach is great. I won't pretend to compete with his level of expertise, but for sake of completeness, here is another approach using the fantastic Queue.Queue class (bonus: thread-safe, but that's kinda useless to you based on your description). This might be easier to understand for you, as you expressed some concern on that point:
myqueue.py
#!/usr/bin/python
# Renamed in Python 3.0
try: from Queue import Queue, Full, Empty
except: from queue import Queue, Full, Empty
from datetime import datetime
# Spec 1: Size of each queue should be user-defined.
# - maxsize on __init__
# Spec 2: Date an object is added should be recorded.
# - datetime.now() is first member of tuple, data is second
# Spec 3: I would like to be able to define the order of the items that
# belong to each queue.
# - Order cannot be rearranged with this queue.
# Spec 4: If the queue is full, the addition of an extra item should discard
# the oldest item of the queue.
# - implemented in put()
class MyQueue(Queue):
"Wrapper around Queue that discards old items instead of blocking."
def __init__(self, maxsize=10):
assert type(maxsize) is int, "maxsize should be an integer"
Queue.__init__(self, maxsize)
def put(self, item):
"Put an item into the queue, possibly discarding an old item."
try:
Queue.put(self, (datetime.now(), item), False)
except Full:
# If we're full, pop an item off and add on the end.
Queue.get(self, False)
Queue.put(self, (datetime.now(), item), False)
def put_nowait(self, item):
"Put an item into the queue, possibly discarding an old item."
self.put(item)
def get(self):
"Get a tuple containing an item and the datetime it was entered."
try:
return Queue.get(self, False)
except Empty:
return None
def get_nowait(self):
"Get a tuple containing an item and the datetime it was entered."
return self.get()
def main():
"Simple test method, showing at least spec #4 working."
queue = MyQueue(5)
for i in range(1, 7):
queue.put("Test item number %u" % i)
while not queue.empty():
time_and_data = queue.get()
print "%s => %s" % time_and_data
if __name__ == "__main__":
main()
expected output
2009-11-02 23:18:37.518586 => Test item number 2
2009-11-02 23:18:37.518600 => Test item number 3
2009-11-02 23:18:37.518612 => Test item number 4
2009-11-02 23:18:37.518625 => Test item number 5
2009-11-02 23:18:37.518655 => Test item number 6
You can use django-activity-stream. It doesn't have UI as Nodequeue has, but it can be used to create different queues of the objects.
https://github.com/justquick/django-activity-stream
http://www.slideshare.net/steveivy/activity-streams-lightning-talk-djangocon-2011-day-3