This concerns the _Status.getOps method. I'm very new to multithreading and I'm not sure if I need to lock the thread for this method. Is list comprehension like this atomic or not? I can't imagine it's a single bytecode, but it seems clumsy.
class _Status:
'''A container object for formatting what we are doing in a
nice way. This way, the information about what we're doing
is retained even after we're long-done doing it. '''
def __init__(self):
self.nextid = 0
self.items = {}
self.lock = threading.Rlock()
def add(self, opStr)
''' Pass this method a string describing an operation and
we'll put it into the data struture, then return to you an
id you can use to do things to this operation later. '''
with self.lock:
id = self._getid()
# Store the operation
self.items[id] = {
'name': opStr
'status': 'Initalizing',
'errors': []
}
return id
def getOps(self):
''' This method returns a list of strings describing what's
going on right now. Also comes with some suggested colors
for displaying them. '''
with self.lock:
return ['{name}: {status}'.format(**op) for op in
[self.items[id] for id in sorted(self.items.keys()]]
def clear(self):
''' This removes all stored operations--use with caution. '''
self.items = {}
def _updateOp(self, opId, opStatus):
''' This method needs an id to work with and a string which
descibes the status of that id's operation. I'll update it
if it can. This is a private method, we'll give aliases for
it, however. '''
with threading.Lock():
# Do nothing if we can't find the item
if id in self.items:
return
self.items[id]['status'] = opStatus
def _getid(self):
''' Internal method to increment the next id we hand out. '''
id = hex(self.nextid)
self.nextid += 1
return id
I'm having a bit of trouble implementing a priority queue in python. Essentially I copied all the code from the heapq implementation documentation, however in my application to recreate the data structure for Dijkstra's algorithm, I would like to update the priorities for each item.
class PriorityQueue():
REMOVED = '<removed-task>' # placeholder for a removed task
def __init__(self, arr=None):
if arr is None:
self.data = []
else:
self.data = []
self.entry_finder = {} # mapping of tasks to entries
self.counter = itertools.count() # unique sequence count
def add_task(self, task, priority=0):
'Add a new task or update the priority of an existing task'
if task in self.entry_finder:
self.remove_task(task)
count = next(self.counter) # Increments count each time item is added ( Saves index ).
entry = [priority, count, task]
self.entry_finder[task] = entry
heapq.heappush(self.data, entry)
def remove_task(self, task):
'Mark an existing task as REMOVED. Raise KeyError if not found.'
entry = self.entry_finder.pop(task)
entry[-1] = self.REMOVED
def pop_task(self):
'Remove and return the lowest priority task. Raise KeyError if empty.'
while self.data:
priority, count, task = heapq.heappop(self.data)
if task is not self.REMOVED:
del self.entry_finder[task]
return task
raise KeyError('pop from an empty priority queue')
list = PriorityQueue()
list.add_task("A", 12)
list.add_task("B", 6)
list.add_task("C", 8)
list.add_task("D", 2)
list.add_task("E", 1)
list.add_task("A", 5)
The following code works fine, except it will add a new task "A", and then the old task "A" will be renamed to 'removed-task' and keep it's index in the heap. I would prefer to outright delete the original task instead. I was confused as to how calling '''add_task"" which calls remove_task to change the value of '''entry'' was implicitly changing the value of items in self.data. I then realized that in add_task:
entry = [priority, count, task]
self.entry_finder[task] = entry
we have these two names interning and referencing the same object. If this is the case, how may I be able to delete an object by it's ID? I think this would solve my problem, unless anyone has another solution to find the item in self.data and remove it in O(1). Thanks!
I would like to be able to "send" and "receive" data into
and from an object similar to how you can do so with a generator.
Why
First, an explanation. It would be nice to be able to do this because I have an object type that I'll call Obj, whose instances must be incrementally created and distilled. I have implemented the incremental object iteration sequence as a series of generators; I did this because the object is a very "choose your own adventure" type of thing, with different paths forking depending on what came before, and each logic step can be complicated and potentially changed in the future. One part of the logic looks kind of like this (it's much more complicated):
# objiter module
def PartX(obj):
# obj does not yet have attr8
yield 'PartX'
# obj now has attr8
if obj.attr1 == 1:
yield from PartY(obj)
if obj.attr1 == 2:
yield from PartZ(obj)
def PartY(obj):
yield 'PartY'
def PartZ(obj):
# obj does not yet have Z
yield from PartZ1
# obj now has Z
if hasattr(obj.Z, 'zattr1'):
yield from PartZ1
if hasattr(obj.Z, 'zattr2'):
yield from PartZ2
The object iteration sequence can be kicked off by the start_iter method. Each yielded string from the incremental logic is paired with a line from an input file.
There is a Builder class which is a descriptor; it receives data and actually makes the incremental changes to the Obj instance. The data is created by each line (from the input file) and label (from the incremental logic generators) using the get_data method.
Here is a basic representation:
from .objiter import PartA
from .parse import parse_line
class Obj():
# builder is a descriptor and context manager, used below in load_file
builder = Builder()
#staticmethod
def get_data(line, label):
'''Makes data out of a line, label pair.'''
parse_line(line, label)
def _start_iter(self):
'''Kick off object incremental logic.'''
# begin incremental logic
yield from PartA(self)
#classmethod
def load_file(cls, file_lines):
'''Construct an instance from an input file.'''
self = Obj() # an "empty" object
iter_logic = self._start_iter()
# start object creation
with self.builder as obj_builder:
for line, label in zip(lines, iter_logic):
data = obj.get_data(line, label)
obj_builder.send(data, label)
# object creation complete
What I would like to do
It would be nice to be able to simply inject data directly into- and receive data from- the object instance rather than using the builder descriptor instance. My preferred solution would allow me to write code like this (see last couple of lines):
for line, label in zip(lines, iter_logic):
data = obj.get_data(line, label)
self.send(data, label)
more_data = next(self)
I could do something like the following:
def send(self, data, label):
self.handle_data(data, label)
def __next__(self):
return <some_data>
However, this doesn't allow the desired behavior, which is to stay inside of a logic sequence and pass control back to the code calling the object's next or send methods.
What I've tried
Right now, these are both illegal:
class C(FunctionType): pass
class C(GeneratorType): pass
So you can't make "generator-like" objects. I have considered a context manager which returns a generator, like so:
def generate_me(self):
'''The logic of handling commands and data.'''
while True:
command = yield
<handle the command by sending/receiving data>
def __enter__(self):
self.generator = self.generate_me()
return self.generator
def __exit__(self, owner, value, tb):
self.generator.close()
del self.generator
Then I can do:
obj = Obj()
with obj as obj_gen:
obj_gen.send(data)
data = next(obj_gen)
This is all well and good, but it's still just a way of creating a separate generator with a little bit nicer syntax.
Is there a way to get real, actual generator behavior working in a user-defined object?
I've been trying to create a simple linked list implementation in Python as a code exercise and, although I have most of the stuff working (inserting, removing, pretty print, swapping the content of two nodes), I've been stuck on swapping two nodes for a few days.
I've looked around on the internet and most people seem to recommend deleting/inserting the nodes or swapping the data. Both are very fine and functional options but I wanted to challenge myself and see if I could swap the nodes the "correct" way.
Ideally I would like to have a generic function that can handle all edge cases (moving to the begin, end and swapping random nodes). This has proven to be way more challenging than I expected.
I've experimented a bit with pen and paper and search around and I found the following discussion and example implementation:
Discussion about swapping in C
Example implementation in C
The issue I run into is that my node1.next and my node2.prev are swapped and that my node2.next refers to itself and not to the next node in the list.
The comment on the page of the example implementation specifically addresses this problem and mentions that it should not happen with his implementation.
I can't seem to figure out what I've done wrong. I guess I could "cheat" and force them to take the correct values at the end but that gives a lot of problems when the node is the first/last.
__author__ = 'laurens'
from django.core.exceptions import ObjectDoesNotExist
import logging
import copy
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
class DoublyLinkedList(object):
"""This class implements a basic doubly linked list in Django, it depends
on a Django model with the following field:
id : PK
prev: integer previous node
data_field: Foreign key
next: integer next node
the prev and next fields don't need to be self-referencing
When instantiating the class you have to link this class to a Django model
and specify a data field. The data field can link to a foreign key
or contain data
"""
def __init__(self, doubly_linked_list_model, data_field):
self.doubly_linked_list_model = doubly_linked_list_model
self.data_field = data_field
def get_node_from_node_id(self, node_id=None):
"""This function returns the node associated with a certain node_id"""
if node_id is None:
node = None
else:
try:
node = self.doubly_linked_list_model.get(id=node_id)
except ObjectDoesNotExist:
node = None
return node
#staticmethod
def _update_node(node, prev=None, next=None):
node.prev = prev
node.next = next
logger.debug('updating node: %s', node.id)
logger.debug('node.prev = %s', node.prev)
logger.debug('node.next = %s', node.next)
try:
node.save()
except Exception as e: #Todo: specify this
logger.debug('Error saving node: %s', node.id)
def move_node(self, node1=None, node2=None):
"""
This function swaps the position of node1 with the position of node2
"""
#swapping two nodes!
logger.debug('Swapping two random nodes!: %s, %s', node1.id, node2.id)
# Swapping next nodes
logger.debug('Swapping next node')
tmp = copy.deepcopy(node1.next)
self._update_node(node=node1,
prev=node1.prev,
next=node2.next)
#Todo: Check if tmp changes or is stored as a copy
self._update_node(node=node2,
prev=node2.prev,
next=tmp)
if node1.next is not None:
logger.debug('Connect the next node to node 1')
node_next = self.get_node_from_node_id(node1.next)
self._update_node(node=node_next,
prev=node1.id,
next=node_next.next)
if node2.next is not None:
logger.debug('Connect the next node to node 2')
node_next = self.get_node_from_node_id(node2.next)
self._update_node(node=node_next,
prev=node2.id,
next=node_next.next)
logger.debug('Swap prev nodes')
tmp = copy.deepcopy(node1.prev)
self._update_node(node=node1,
prev=node2.prev,
next=node1.next)
self._update_node(node=node2,
prev=tmp,
next=node2.next)
# Connect the node before node1 to node 1
if node1.prev is not None:
logger.debug('Connect the prev to node 1')
node_prev = self.get_node_from_node_id(node1.prev)
self._update_node(node=node_prev,
prev=node_prev.prev,
next=node1.id)
# Connect the node before node2 to node 2
if node2.prev is not None:
logger.debug('Connect the prev to node 2')
node_prev = self.get_node_from_node_id(node2.prev)
self._update_node(node=node_prev,
prev=node_prev.prev,
next=node2.id)
The _update_node function does nothing more than taking my input and committing it to the database; it can handle None values.
get_node_from_node_id takes an integer as input and returns the node object associated with it. I use it so that I don't have to work with self-referencing foreign keys (is that the correct term?) in the database, for now I would like to continue working this way. Once I have this working I'll move on to fixing it in the database in the correct way.
tip: I get answers much more quickly when I provide a minimal, complete, verifiable example (MCVE), also known as a short, self-contained, compilable example (SSCCE).
Your example code fails to demonstrate the problem, making it impossible for us to help you.
Please make it easy for us to help you.
I ran your example code, but I didn't see any output.
The issue I run into is that ... that my node2.next refers to itself and
not to the next node in the list.
Why is node2.next referring to node2 a problem?
As far as I can tell, the part of the code you gave us so far works fine.
Some of the most difficult debugging sessions I've ever had ended only when I realized that everything was actually working correctly, that the "bug" I thought I was hunting didn't actually exist.
Yes, there is an intermediate step halfway through the algorithm where a node refers to itself, which may seem obviously wrong.
But then the second half of the algorithm makes further changes.
By the time the algorithm finishes,
the "next" chain correctly runs all the way from the beginning to the end, and
the "forward" chain correctly runs in the reverse order as the "next" chain, from the end to the beginning, and
the order of those chains is similar to the original order, except that node1 and node2 have swapped logical position (but are still in the same physical position).
Every node points to 2 other nodes (or to the NULL node), never to itself.
Isn't that what you wanted?
When I ran the following test script, I get lots of output --
but I don't see any problems in the output.
(I used a simple array with integer indexes rather than true pointers or references to make the code shorter and easier to debug compared to a SQL database).
Would you mind pointing out which particular line of output is not what you expected, and spelling out what you expected that line of output to actually say?
#!/usr/bin/env python
# https://stackoverflow.com/questions/24610889/trying-to-write-a-swap-position-function-for-a-doubly-linked-list
# Is this the shortest possible program that exhibits the bug?
# Before running this probram, you may need to install
# sudo apt-get install python-django
# 2015-03-12: David added some test scaffolding
# 2014-07-07: Laurens posted to StackOverflow
__author__ = 'laurens'
from django.core.exceptions import ObjectDoesNotExist
import logging
import copy
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
class DoublyLinkedList(object):
"""This class implements a basic doubly linked list in Django, it depends
on a Django model with the following field:
id : PK
prev: integer previous node
data_field: Foreign key
next: integer next node
the prev and next fields don't need to be self-referencing
When instantiating the class you have to link this class to a Django model
and specify a data field. The data field can link to a foreign key
or contain data
"""
def __init__(self, doubly_linked_list_model, data_field):
self.doubly_linked_list_model = doubly_linked_list_model
self.data_field = data_field
def get_node_from_node_id(self, node_id=None):
"""This function returns the node associated with a certain node_id"""
if node_id is None:
node = None
else:
try:
node = self.doubly_linked_list_model.get(id=node_id)
except ObjectDoesNotExist:
node = None
return node
#staticmethod
def _update_node(node, prev=None, next=None):
node.prev = prev
node.next = next
logger.debug('updating node: %s', node.id)
logger.debug('node.prev = %s', node.prev)
logger.debug('node.next = %s', node.next)
try:
node.save()
except Exception as e: #Todo: specify this
logger.debug('Error saving node: %s', node.id)
def move_node(self, node1=None, node2=None):
"""
This function swaps the position of node1 with the position of node2
"""
#swapping two nodes!
logger.debug('Swapping two random nodes!: %s, %s', node1.id, node2.id)
# Swapping next nodes
logger.debug('Swapping next node')
tmp = copy.deepcopy(node1.next)
self._update_node(node=node1,
prev=node1.prev,
next=node2.next)
#Todo: Check if tmp changes or is stored as a copy
self._update_node(node=node2,
prev=node2.prev,
next=tmp)
if node1.next is not None:
logger.debug('Connect the next node to node 1')
node_next = self.get_node_from_node_id(node1.next)
self._update_node(node=node_next,
prev=node1.id,
next=node_next.next)
if node2.next is not None:
logger.debug('Connect the next node to node 2')
node_next = self.get_node_from_node_id(node2.next)
self._update_node(node=node_next,
prev=node2.id,
next=node_next.next)
logger.debug('Swap prev nodes')
tmp = copy.deepcopy(node1.prev)
self._update_node(node=node1,
prev=node2.prev,
next=node1.next)
self._update_node(node=node2,
prev=tmp,
next=node2.next)
# Connect the node before node1 to node 1
if node1.prev is not None:
logger.debug('Connect the prev to node 1')
node_prev = self.get_node_from_node_id(node1.prev)
self._update_node(node=node_prev,
prev=node_prev.prev,
next=node1.id)
# Connect the node before node2 to node 2
if node2.prev is not None:
logger.debug('Connect the prev to node 2')
node_prev = self.get_node_from_node_id(node2.prev)
self._update_node(node=node_prev,
prev=node_prev.prev,
next=node2.id)
global_test_array = []
obfuscation = 0xaa
class trivial_test_node_class(object):
def __init__(self, prev, id, next, data):
print "initializing test class."
# self.stuff = [ ["first", 0, 1], ["second", 0, 1] ]
self.id = id
self.prev = prev
self.next = next
self.data = data
def something(self):
print "something"
def save(self):
id = self.id
global_test_array[id] = self
def __repr__(self):
# print self.prev, self.id, self.next, self.data
the_string = "%s(%r)\n" % (self.__class__, self.__dict__)
return the_string
class trivial_test_list_model_class(object):
def __init__(self):
print "initializing test class."
#self.stuff = [ ["first", 0, 1], ["second", 0, 1] ]
self.stuff = [ 0 for i in xrange(0,10) ]
data = 'a'
for i in xrange(1,10):
self.stuff[i] = trivial_test_node_class(i-1,i,i+1,data);
data += 'r' # talk like a pirate day
self.stuff[-1].next = 0 # use 0 as NULL id.
global global_test_array
global_test_array = self.stuff
def something(self):
print "something"
def get(self,id):
return self.stuff[id]
def __repr__(self):
# for i in xrange(1,10):
# print self.stuff[i]
the_string = "%s(%r)" % (self.__class__, self.__dict__)
return the_string
if __name__ == '__main__':
# test code that only gets run when this file is run directly,
# not when this file is imported from some other python script.
print "Hello, world."
trivial_test_model = trivial_test_list_model_class()
print trivial_test_model
testll = DoublyLinkedList( trivial_test_model, "data" )
left_node = trivial_test_model.get(3)
right_node = trivial_test_model.get(4)
testll.move_node( left_node, right_node )
print trivial_test_model
# recommended by http://wiki.python.org/moin/vim
# vim: tabstop=8 expandtab shiftwidth=4 softtabstop=4 :
i'm having trouble creating an insert function with the following parameters. The insert function should take in a priority queue, and an element and inserts it using the priority rules -
The priority queue will take a series of tasks and order them
based on their importance. Each task has an integer priority from 10 (highest priority) to 1
(lowest priority). If two tasks have the same priority, the order should be based on the order
they were inserted into the priority queue (earlier first).
So, as of right now i've created the following code to initialize some of the things needed...
class Tasks():
__slots__ = ('name', 'priority')
def __init__(bval):
bval.name = myName
bval.priority = myPriority
return bval
class PriorityQueue():
__slots__ = ('queue', 'element')
def __init__(aval):
aval.queue = queue
aval.element = element
return aval
The code i'm trying to write is insert(element, queue): which should insert the elements using the priority queue. Similarly, myPriorty is an integer from 1 to 10.
Similarly can I do the following to insure that I create a priority from 1 to 10...
def __init__(bval , myPriority = 10):
bval.priority = myPriority
bval.pq = [[] for priority in range(bval.myPriority)]
so that I can replace myPriority in the insert task with bval.pq
Why are you trying to re-invent the wheel?
from Queue import PriorityQueue
http://docs.python.org/2/library/queue.html?highlight=priorityqueue#Queue.PriorityQueue
The lowest valued entries are retrieved first (the lowest valued entry is the one returned by sorted(list(entries))[0]). A typical pattern for entries is a tuple in the form:
(priority_number, data).
I use such a module to communicate between the UI and a background polling thread.
READ_LOOP = 5
LOW_PRI = 3
MED_PRI = 2
HI_PRI = 1
X_HI_PRI = 0
and then something like this:
CoreGUI.TX_queue.put((X_HI_PRI,'STOP',[]))
Note that there is a Queue. If you are okay with it being synchronized, I would use that.
Otherwise, you should use a heap to maintain your queue. See Python documentation with an example of that.
From a great book "Modern Python Standard Library Cookbook" by Alessandro Molina
Heaps are a perfect match for everything that has priorities, such as
a priority queue:
import time
import heapq
class PriorityQueue:
def __init__(self):
self._q = []
def add(self, value, priority=0):
heapq.heappush(self._q, (priority, time.time(), value))
def pop(self):
return heapq.heappop(self._q)[-1]
Example:
>>> def f1(): print('hello')
>>> def f2(): print('world')
>>>
>>> pq = PriorityQueue()
>>> pq.add(f2, priority=1)
>>> pq.add(f1, priority=0)
>>> pq.pop()()
hello
>>> pq.pop()()
world
A deque (from collections import deque) is the python implementation of a single queue. You can add items to one end and remove them from the other. If you have a deque for each priority level, you can add to the priority level you want.
Together, it looks a bit like this:
from collections import deque
class PriorityQueue:
def __init__(self, priorities=10):
self.subqueues = [deque() for _ in range(levels)]
def enqueue(self, priorty, value):
self.subqueues[priority].append(value)
def dequeue(self):
for queue in self.subqueues:
try:
return queue.popleft()
except IndexError:
continue