I would like to build a priority queue in python in which the queue contains different dictionaries with their priority numbers. So when a "get function" is called, the dictionary with the highest priority(lowest number) will be pulled out of the queue and when "add function" is called, the new dictionary will be added to the queue and sorted based on its priority number.
Please do help out...
Thanks in advance!
Use the heapq module in the standard library.
You don't specify how you wanted to associate priorities with dictionaries, but here's a simple implementation:
import heapq
class MyPriQueue(object):
def __init__(self):
self.heap = []
def add(self, d, pri):
heapq.heappush(self.heap, (pri, d))
def get(self):
pri, d = heapq.heappop(self.heap)
return d
This is what I usually present as a side note in some of my patterns talks:
class PriorityQueue(object):
def __init__(self, key=lambda x: x):
self.l = []
self.key = key
def __len__(self):
return len(self.l)
def push(self, obj):
heapq.heappush(self.l, (self.key(obj), obj))
def pop(self):
return heapq.heappop(self.l)[-1]
The OP's requirements are apparently to use operator.itemgetter('priority') as the key argument when instantiating PriorityQueue (needs an import operator at top of module, of course;-).
You can do this by adding a dict object to the class, and search it inside.
Related
I want to use a PriorityQueue to which I need to add objects of a class (say Node) which I cannot modify. I need these objects prioritized based on a field of the object.
I tried adding them as tuples(node.val, node) to the queue. It still gives me "TypeError: '<' not supported between instances of 'Node' and 'Node'".
It still compares the 'node' object to resolve the ties.
a = Node(3)
b = Node(4)
q = queue.PriorityQueue()
q.put(a)
q.put(b) // Errors TypeError: '<' not supported between instances of 'Node' and 'Node'
q.put((a.val, a))
q.put((b.val, b)) // Errors TypeError: '<' not supported between instances of 'Node' and 'Node'
I understand that the cause of this error is that the Node objects need to be comparable to one another, and the way to achieve this according to the documentation is to at least implement lt and eq methods for Node. https://docs.python.org/3.1/library/stdtypes.html#comparisons.
But for my problem, since I wont be able to modify the Node class, will I be able to pass a way( a lambda or a Comparator) to the PriorityQueue to help it determine the ordering.(I am expecting something Java like)
q = PriorityQueue(comparator)
Any alternative to achieve this is also appreciated.(remember Node is not modifiable)
One solution, without passing a comparator, is to wrap your Node objects in another object, say ComparableNode, which implements the comparisons you would like on the Node object.
Let's assume you have this setup:
class Node:
def __init__(self, val):
self.val = val
def __repr__(self):
return "Node(val={})".format(self.val)
a = Node(3)
b = Node(4)
c = Node(4)
But you can't modify Node. So you can wrap it in a class that you create.
#functools.total_ordering
class ComparableNode(Node):
def __gt__(self, other):
return self.val > other.val
def __eq__(self, other):
return self.val == other.val
aa = ComparableNode(3)
bb = ComparableNode(4)
cc = ComparableNode(4)
or if you want to just pass Node objects that exist already:
#functools.total_ordering
class ComparableNode:
def __init__(self, node):
self.node = node
def __gt__(self, other):
return self.node.val > other.node.val
def __eq__(self, other):
return self.node.val == other.node.val
def __repr__(self):
return repr(self.node)
aa = ComparableNode(a)
bb = ComparableNode(b)
cc = ComparableNode(c)
Then just add them normally:
p = PriorityQueue()
p.put(aa)
p.put(bb)
p.put(cc)
p.queue # [Node(val=3), Node(val=4), Node(val=4)]
Another way, if you can specify the priority of each element as you push it in. This is possible with heapq for example.
With heapq:
import heapq
q = []
heapq.heappush(q, (a.val, a))
heapq.heappush(q, (b.val, b))
q # [(3, Node(val=3)), (4, Node(val=4))]
With queue.PriorityQueue, use tuples as suggested by the doc
from queue import PriorityQueue
p = PriorityQueue()
p.put((a.val, a))
p.put((b.val, b))
p.queue # [(3, Node(val=3)), (4, Node(val=4))]
If you have duplicate values, then the comparison will look at the second element of the tuple, and will break because your Nodes are not comparable. In that case, it depends how you want to deal with duplicate values. If you want to preserve insert order in case of duplicates, you can do this:
from queue import PriorityQueue
from itertools import count
p = PriorityQueue()
index = count(0)
p.put((b.val, next(index), b))
p.put((c.val, next(index), c))
p.queue # [(4, 0, Node(val=4)), (4, 1, Node(val=4))]
And proceed similarly with heapq. You could easily wrap this behavior in a small function so that you repeat yourself less.
p = PriorityQueue()
index = count(0)
def put_priority(node):
p.put((node.val, next(index), node))
The PriorityQueue class does not allow for a key parameter, but you can subclass it to implicitly wrap each item inside a comparator object, here called a _Wrapper.
from queue import PriorityQueue
class _Wrapper:
def __init__(self, item, key):
self.item = item
self.key = key
def __lt__(self, other):
return self.key(self.item) < other.key(other.item)
def __eq__(self, other):
return self.key(self.item) == other.key(other.item)
class KeyPriorityQueue(PriorityQueue):
def __init__(self, key):
self.key = key
super().__init__()
def _get(self):
wrapper = super()._get()
return wrapper.item
def _put(self, item):
super()._put(_Wrapper(item, self.key))
Usage
Here is an example of a priority queue which reverses the order or the elements.
key_pq = KeyPriorityQueue(key=lambda x: -x)
If you need a comparator and not a key, you can use the functools.cmp_to_key helper.
from functools import cmp_to_key
key_pq = KeyPriorityQueue(key=cmp_to_key(some_comparator))
I'm trying to port a custom class from Python 2 to Python 3. I can't find the right syntax to port the iterator for the class. Here is a MVCE of the real class and my attempts to solve this so far:
Working Python 2 code:
class Temp:
def __init__(self):
self.d = dict()
def __iter__(self):
return self.d.iteritems()
temp = Temp()
for thing in temp:
print(thing)
In the above code iteritems() breaks in Python 3. According to this highly voted answer, "dict.items now does the thing dict.iteritems did in python 2". So I tried that next:
class Temp:
def __init__(self):
self.d = dict()
def __iter__(self):
return self.d.items()
The above code yields "TypeError: iter() returned non-iterator of type 'dict_items'"
According to this answer, Python 3 requires iterable objects to provide a next() method in addition to the iter method. Well, a dictionary is also iterable, so in my use case I should be able to just pass dictionary's next and iter methods, right?
class Temp:
def __init__(self):
self.d = dict()
def __iter__(self):
return self.d.__iter__
def next(self):
return self.d.next
This time it's giving me "TypeError: iter() returned non-iterator of type 'method-wrapper'".
What am I missing here?
As the error message suggests, your __iter__ function does not return an iterator, which you can easily fix using the built-in iter function
class Temp:
def __init__(self):
self.d = {}
def __iter__(self):
return iter(self.d.items())
This will make your class iterable.
Alternatively, you may write a generator yourself, like so:
def __iter__(self):
for key,item in self.d.items():
yield key,item
If you want to be able to iterate over keys and items separately, i.e. in the form that the usual python3 dictionary can, you can provide additional functions, for example
class Temp:
def __init__(self, dic):
self.d = dic
def __iter__(self):
return iter(self.d)
def keys(self):
return self.d.keys()
def items(self):
return self.d.items()
def values(self):
return self.d.values()
I'm guessing from the way you phrased it that you don't actually want the next() method to be implemented if not needed. If you would, you would have to somehow turn your whole class into an iterator and somehow keep track of where you are momentarily in this iterator, because dictionaries themselves are not iterators. See also this answer.
I don't know what works in Python 2. But on Python 3 iterators can be most easily created using something called a generator. I am providing the name and the link so that you can research further.
class Temp:
def __init__(self):
self.d = {}
def __iter__(self):
for thing in self.d.items():
yield thing
Here's is my simple class:
class row(object):
__slots__ = ("tan_id", "proc_name")
def __init__(
self,
tan_id = "INDETERMINATE",
proc_name = "INDETERMINATE"
):
self.tan_id = tan_id
self.proc_name = proc_name
This class used to be a dictionary, but I ran into serious memory issues. Later on in my code I use a generalized function to loop like this:
for key, val in row.items()
I don't want to change that... so my question is, how do I simulate that item() function in the class so that it works seamlessly in the existing for loop.
You can simply implement an items method that mimicks the previous dict.items() behaviour by producing the same key-value pairs:
class row(object):
# ...
def items(self):
# this is the iterator behaviour of 'items' (Py3) or 'iteritems' (Py2)
yield 'tan_id', self.tan_id
yield 'proc_name', self.proc_name
# for the Py2 items behaviour, put these pairs into a list
# and return the list
# Or, more generally:
def items(self):
for slot in self.__slots__:
yield slot, getattr(self, slot)
I'am trying to implement a 'heapq' or a 'deque' dynamically (according to user's input)
class MyClass():
def __init__(self, choose = True ):
self.Q = []
self.add = self.genAdd(choose)
self.get = self.genGet(choose)
def genAdd(self, ch):
if(ch == True):
def f(Q, elem):
return Q.append
else:
def f(Q):
return heappush
return f
and same for 'genGet'
the execution is correct on one side (x)or the other (but not both at the same time). I get things like
TypeError: f() takes exactly 1 argument (2 given)
tried multiple inhreitance but got
TypeError: Error when calling the metaclass bases
metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases
the problem is that heapq is called with
heappush(Q, elem)
and queue with
Q.append(elem)
I hope the point is clear. I think there should be a way to fix that (maybe using lambda)
Thanks
Inheritance isn't going to help here.
First, heapq isn't even a class, so you can't inherit from it. You can write a class that wraps up its functionality (or find one on the ActiveState recipes or in a PyPI package), but you have to have a class to inherit.
But, more importantly, the whole point of inheritance is to give you an "is-a" relationship. This thing you're building isn't-a deque, or a heapq-wrapping object, it's a thing with an interface that you've defined (add and get) that happens to use either a deque or a list with heapq for implementation.
So, just do that explicitly. You're trying to define a function that either calls append on a deque, or calls heapq.heappush on a list. You're not trying to write a curried function that returns a function that does the thing, just a function that does the thing.
def genAdd(self, ch):
# As a side note, you don't need to compare == True, nor
# do you need to wrap if conditions in parens.
if ch:
def f(elem):
self.Q.append(elem)
else:
def f(elem):
heappush(self.Q, elem)
return f
There are a few other problems here. First, you definitely need to set self.Q = deque() instead of self.Q = [] if you wanted a deque. And you probably want to wrap these functions up as a types.MethodType instead of using self as a closure variable (this will work, it's just less readable, because it may not be clear to many people why it works). And so on. But this is the fundamental problem.
For example:
from collections import deque
from heapq import heappush
class MyClass(object):
def __init__(self, choose=True):
self.Q = deque() if choose else []
self.add = self.genAdd(choose)
def genAdd(self, ch):
if ch:
def f(elem):
self.Q.append(elem)
else:
def f(elem):
heappush(self.Q, elem)
return f
d = MyClass(True)
d.add(3)
d.add(2)
print(d.Q)
h = MyClass(False)
h.add(3)
h.add(2)
print(h.Q)
This will print:
deque([3, 2])
[2, 3]
That being said, there's probably a much better design: Create a class that wraps a deque in your interface. Create another class that wraps a list with heapq in your interface. Create a factory function that returns one or the other:
class _MyClassDeque(object):
def __init__(self):
self.Q = deque()
def add(self, elem):
self.Q.append(elem)
class _MyClassHeap(object):
def __init__(self):
self.Q = []
def add(self, elem):
heappush(self.Q, elem)
def MyClass(choose=True):
return _MyClassDeque() if choose else _MyClassHeap()
Now you get the same results, but the code is a lot easier to understand (and slightly more efficient, if you care…).
I want to write a custom class that behaves like dict - so, I am inheriting from dict.
My question, though, is: Do I need to create a private dict member in my __init__() method?. I don't see the point of this, since I already have the dict behavior if I simply inherit from dict.
Can anyone point out why most of the inheritance snippets look like the one below?
class CustomDictOne(dict):
def __init__(self):
self._mydict = {}
# other methods follow
Instead of the simpler...
class CustomDictTwo(dict):
def __init__(self):
# initialize my other stuff here ...
# other methods follow
Actually, I think I suspect the answer to the question is so that users cannot directly access your dictionary (i.e. they have to use the access methods that you have provided).
However, what about the array access operator []? How would one implement that? So far, I have not seen an example that shows how to override the [] operator.
So if a [] access function is not provided in the custom class, the inherited base methods will be operating on a different dictionary?
I tried the following snippet to test out my understanding of Python inheritance:
class myDict(dict):
def __init__(self):
self._dict = {}
def add(self, id, val):
self._dict[id] = val
md = myDict()
md.add('id', 123)
print md[id]
I got the following error:
KeyError: < built-in function id>
What is wrong with the code above?
How do I correct the class myDict so that I can write code like this?
md = myDict()
md['id'] = 123
[Edit]
I have edited the code sample above to get rid of the silly error I made before I dashed away from my desk. It was a typo (I should have spotted it from the error message).
class Mapping(dict):
def __setitem__(self, key, item):
self.__dict__[key] = item
def __getitem__(self, key):
return self.__dict__[key]
def __repr__(self):
return repr(self.__dict__)
def __len__(self):
return len(self.__dict__)
def __delitem__(self, key):
del self.__dict__[key]
def clear(self):
return self.__dict__.clear()
def copy(self):
return self.__dict__.copy()
def has_key(self, k):
return k in self.__dict__
def update(self, *args, **kwargs):
return self.__dict__.update(*args, **kwargs)
def keys(self):
return self.__dict__.keys()
def values(self):
return self.__dict__.values()
def items(self):
return self.__dict__.items()
def pop(self, *args):
return self.__dict__.pop(*args)
def __cmp__(self, dict_):
return self.__cmp__(self.__dict__, dict_)
def __contains__(self, item):
return item in self.__dict__
def __iter__(self):
return iter(self.__dict__)
def __unicode__(self):
return unicode(repr(self.__dict__))
o = Mapping()
o.foo = "bar"
o['lumberjack'] = 'foo'
o.update({'a': 'b'}, c=44)
print 'lumberjack' in o
print o
In [187]: run mapping.py
True
{'a': 'b', 'lumberjack': 'foo', 'foo': 'bar', 'c': 44}
Like this
class CustomDictOne(dict):
def __init__(self,*arg,**kw):
super(CustomDictOne, self).__init__(*arg, **kw)
Now you can use the built-in functions, like dict.get() as self.get().
You do not need to wrap a hidden self._dict. Your class already is a dict.
Check the documentation on emulating container types. In your case, the first parameter to add should be self.
UserDict from the Python standard library is designed for this purpose.
Here is an alternative solution:
class AttrDict(dict):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.__dict__ = self
a = AttrDict()
a.a = 1
a.b = 2
This is my best solution. I used this many times.
class DictLikeClass:
...
def __getitem__(self, key):
return getattr(self, key)
def __setitem__(self, key, value):
setattr(self, key, value)
...
You can use like:
>>> d = DictLikeClass()
>>> d["key"] = "value"
>>> print(d["key"])
A python class that acts like dict
What's wrong with this?
Can anyone point out why most of the inheritance snippets look like the one below?
class CustomDictOne(dict):
def __init__(self):
self._mydict = {}
Presumably there's a good reason to inherit from dict (maybe you're already passing one around and you want a more specific kind of dict) and you have a good reason to instantiate another dict to delegate to (because this will instantiate two dicts per instance of this class.) But doesn't that sound incorrect?
I never run into this use-case myself. I do like the idea of typing dicts where you are using dicts that are type-able. But in that case I like the idea of typed class attributes even moreso - and the whole point of a dict is you can give it keys of any hashable type, and values of any type.
So why do we see snippets like this? I personally think it's an easily made mistake that went uncorrected and thus perpetuated over time.
I would rather see, in these snippets, this, to demonstrate code reuse through inheritance:
class AlternativeOne(dict):
__slots__ = ()
def __init__(self):
super().__init__()
# other init code here
# new methods implemented here
or, to demonstrate re-implementing the behavior of dicts, this:
from collections.abc import MutableMapping
class AlternativeTwo(MutableMapping):
__slots__ = '_mydict'
def __init__(self):
self._mydict = {}
# other init code here
# dict methods reimplemented and new methods implemented here
By request - adding slots to a dict subclass.
Why add slots? A builtin dict instance doesn't have arbitrary attributes:
>>> d = dict()
>>> d.foo = 'bar'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'dict' object has no attribute 'foo'
If we create a subclass the way most are doing it here on this answer, we see we don't get the same behavior, because we'll have a __dict__ attribute, causing our dicts to take up to potentially twice the space:
my_dict(dict):
"""my subclass of dict"""
md = my_dict()
md.foo = 'bar'
Since there's no error created by the above, the above class doesn't actually act, "like dict."
We can make it act like dict by giving it empty slots:
class my_dict(dict):
__slots__ = ()
md = my_dict()
So now attempting to use arbitrary attributes will fail:
>>> md.foo = 'bar'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'my_dict' object has no attribute 'foo'
And this Python class acts more like a dict.
For more on how and why to use slots, see this Q&A: Usage of __slots__?
I really don't see the right answer to this anywhere
class MyClass(dict):
def __init__(self, a_property):
self[a_property] = a_property
All you are really having to do is define your own __init__ - that really is all that there is too it.
Another example (little more complex):
class MyClass(dict):
def __init__(self, planet):
self[planet] = planet
info = self.do_something_that_returns_a_dict()
if info:
for k, v in info.items():
self[k] = v
def do_something_that_returns_a_dict(self):
return {"mercury": "venus", "mars": "jupiter"}
This last example is handy when you want to embed some kind of logic.
Anyway... in short class GiveYourClassAName(dict) is enough to make your class act like a dict. Any dict operation you do on self will be just like a regular dict.
The problem with this chunk of code:
class myDict(dict):
def __init__(self):
self._dict = {}
def add(id, val):
self._dict[id] = val
md = myDict()
md.add('id', 123)
...is that your 'add' method (...and any method you want to be a member of a class) needs to have an explicit 'self' declared as its first argument, like:
def add(self, 'id', 23):
To implement the operator overloading to access items by key, look in the docs for the magic methods __getitem__ and __setitem__.
Note that because Python uses Duck Typing, there may actually be no reason to derive your custom dict class from the language's dict class -- without knowing more about what you're trying to do (e.g, if you need to pass an instance of this class into some code someplace that will break unless isinstance(MyDict(), dict) == True), you may be better off just implementing the API that makes your class sufficiently dict-like and stopping there.
Don’t inherit from Python built-in dict, ever! for example update method woldn't use __setitem__, they do a lot for optimization. Use UserDict.
from collections import UserDict
class MyDict(UserDict):
def __delitem__(self, key):
pass
def __setitem__(self, key, value):
pass