Given a list:
source = [{'A':123},{'B':234},{'C':345}]
I need a dict from this list in the format:
newDict = {'A':123,'B':234,'C':345}
What is the syntactically cleanest way to accomplish this?
Use a dict-comprehension:
>>> l = [{'A':123},{'B':234},{'C':345}]
>>> d = {k: v for dct in l for k, v in dct.items()}
>>> d
{'A': 123, 'B': 234, 'C': 345}
However it's probably opinion-based if that's the "syntactically cleanest way" but I like it.
Here's an additional approach, provided here to give you a flavor for how Python implements the functional programming technique called reduction, via the reduce() function. In Python 3, reduce() is in the functools package. In Python 2, reduce() is a built-in function. I use Python 3 in the example below:
from functools import reduce # don't import if you are using Python 2
def updater(dict_orig, dict_add):
dict_orig.update(dict_add)
return dict_orig
new_dict = reduce(updater, l, dict())
The first argument to reduce() is the function to operate on the iterable, the second is the iterable itself (your list l), and the third is the optional initializer object to put at the beginning of the list to reduce.
Each step of the reduction requires an object to be operated on: namely, the result of the previous step. But dict.update() does not return anything, so we need the updater() function above, which performs the update and then returns the dict being updated, thus providing the required object for the next step. Were it not for dict.update() not having a return value, this would all be a one-liner.
Because dict.update() operates directly on the original dict, we need that optional empty dict() initializer object to start out the reduction - without it, the first dict in your original l list would be modified.
For all these reasons, I like #MSeifert's dict-comprehension approach much better, but I posted this anyway just to illustrate Python reduction for you.
If you use it often, you might want to define a merge function, which you can then pass to reduce :
from functools import reduce # Needed for Python3
source = [{'A':123},{'B':234},{'C':345}]
def merge(a,b):
d = a.copy()
d.update(b)
return d
print(reduce(merge, source))
#=> {'A': 123, 'C': 345, 'B': 234}
Related
This question already has answers here:
How do I initialize a dictionary of empty lists in Python?
(7 answers)
Closed 2 years ago.
I came across this behavior that surprised me in Python 2.6 and 3.2:
>>> xs = dict.fromkeys(range(2), [])
>>> xs
{0: [], 1: []}
>>> xs[0].append(1)
>>> xs
{0: [1], 1: [1]}
However, dict comprehensions in 3.2 show a more polite demeanor:
>>> xs = {i:[] for i in range(2)}
>>> xs
{0: [], 1: []}
>>> xs[0].append(1)
>>> xs
{0: [1], 1: []}
>>>
Why does fromkeys behave like that?
Your Python 2.6 example is equivalent to the following, which may help to clarify:
>>> a = []
>>> xs = dict.fromkeys(range(2), a)
Each entry in the resulting dictionary will have a reference to the same object. The effects of mutating that object will be visible through every dict entry, as you've seen, because it's one object.
>>> xs[0] is a and xs[1] is a
True
Use a dict comprehension, or if you're stuck on Python 2.6 or older and you don't have dictionary comprehensions, you can get the dict comprehension behavior by using dict() with a generator expression:
xs = dict((i, []) for i in range(2))
In the first version, you use the same empty list object as the value for both keys, so if you change one, you change the other, too.
Look at this:
>>> empty = []
>>> d = dict.fromkeys(range(2), empty)
>>> d
{0: [], 1: []}
>>> empty.append(1) # same as d[0].append(1) because d[0] references empty!
>>> d
{0: [1], 1: [1]}
In the second version, a new empty list object is created in every iteration of the dict comprehension, so both are independent from each other.
As to "why" fromkeys() works like that - well, it would be surprising if it didn't work like that. fromkeys(iterable, value) constructs a new dict with keys from iterable that all have the value value. If that value is a mutable object, and you change that object, what else could you reasonably expect to happen?
To answer the actual question being asked: fromkeys behaves like that because there is no other reasonable choice. It is not reasonable (or even possible) to have fromkeys decide whether or not your argument is mutable and make new copies every time. In some cases it doesn't make sense, and in others it's just impossible.
The second argument you pass in is therefore just a reference, and is copied as such. An assignment of [] in Python means "a single reference to a new list", not "make a new list every time I access this variable". The alternative would be to pass in a function that generates new instances, which is the functionality that dict comprehensions supply for you.
Here are some options for creating multiple actual copies of a mutable container:
As you mention in the question, dict comprehensions allow you to execute an arbitrary statement for each element:
d = {k: [] for k in range(2)}
The important thing here is that this is equivalent to putting the assignment k = [] in a for loop. Each iteration creates a new list and assigns it to a value.
Use the form of the dict constructor suggested by #Andrew Clark:
d = dict((k, []) for k in range(2))
This creates a generator which again makes the assignment of a new list to each key-value pair when it is executed.
Use a collections.defaultdict instead of a regular dict:
d = collections.defaultdict(list)
This option is a little different from the others. Instead of creating the new list references up front, defaultdict will call list every time you access a key that's not already there. You can there fore add the keys as lazily as you want, which can be very convenient sometimes:
for k in range(2):
d[k].append(42)
Since you've set up the factory for new elements, this will actually behave exactly as you expected fromkeys to behave in the original question.
Use dict.setdefault when you access potentially new keys. This does something similar to what defaultdict does, but it has the advantage of being more controlled, in the sense that only the access you want to create new keys actually creates them:
d = {}
for k in range(2):
d.setdefault(k, []).append(42)
The disadvantage is that a new empty list object gets created every time you call the function, even if it never gets assigned to a value. This is not a huge problem, but it could add up if you call it frequently and/or your container is not as simple as list.
Let's assume we have a dictionary like this:
>>> d={'a': 4, 'b': 2, 'c': 1.5}
If I want to select the first item of d, I can simply run the following:
>>> first_item = list(d.items())[0]
('a', 4)
However, I am trying to have first_item return a dict instead of a tuple i.e., {'a': 4}. Thanks for any tips.
Use itertools.islice to avoid creating the entire list, that is unnecessarily wasteful. Here's a helper function:
from itertools import islice
def pluck(mapping, pos):
return dict(islice(mapping.items(), pos, pos + 1))
Note, the above will return an empty dictionary if pos is out of bounds, but you can check that inside pluck and handle that case however you want (IMO it should probably raise an error).
>>> pluck(d, 0)
{'a': 4}
>>> pluck(d, 1)
{'b': 2}
>>> pluck(d, 2)
{'c': 1.5}
>>> pluck(d, 3)
{}
>>> pluck(d, 4)
{}
Note, accessing an element by position in a dict requires traversing the dict. If you need to do this more often, for arbitrary positions, consider using a sequence type like list which can do it in constant time. Although dict objects maintain insertion order, the API doesn't expose any way to manipulate the dict as a sequence, so you are stuck with using iteration.
Dictionary is a collections of key-value pairs. It is not really ordered collection, though since python 3.7 it keeps the order in which keys were added.
Anyway if you really want some "first" element you can get it in this manner:
some_item = next(iter(d.items()))
You should not convert it into a list because it will eat much (O(n)) memory and walk through whole dict as well.
Anyway I'd recommend not to think that dictionary has "first" element. It has keys and values. You can iterate over them in some unknown order (if you do not control how it is created)
This question already has answers here:
How do I initialize a dictionary of empty lists in Python?
(7 answers)
Closed 2 years ago.
I came across this behavior that surprised me in Python 2.6 and 3.2:
>>> xs = dict.fromkeys(range(2), [])
>>> xs
{0: [], 1: []}
>>> xs[0].append(1)
>>> xs
{0: [1], 1: [1]}
However, dict comprehensions in 3.2 show a more polite demeanor:
>>> xs = {i:[] for i in range(2)}
>>> xs
{0: [], 1: []}
>>> xs[0].append(1)
>>> xs
{0: [1], 1: []}
>>>
Why does fromkeys behave like that?
Your Python 2.6 example is equivalent to the following, which may help to clarify:
>>> a = []
>>> xs = dict.fromkeys(range(2), a)
Each entry in the resulting dictionary will have a reference to the same object. The effects of mutating that object will be visible through every dict entry, as you've seen, because it's one object.
>>> xs[0] is a and xs[1] is a
True
Use a dict comprehension, or if you're stuck on Python 2.6 or older and you don't have dictionary comprehensions, you can get the dict comprehension behavior by using dict() with a generator expression:
xs = dict((i, []) for i in range(2))
In the first version, you use the same empty list object as the value for both keys, so if you change one, you change the other, too.
Look at this:
>>> empty = []
>>> d = dict.fromkeys(range(2), empty)
>>> d
{0: [], 1: []}
>>> empty.append(1) # same as d[0].append(1) because d[0] references empty!
>>> d
{0: [1], 1: [1]}
In the second version, a new empty list object is created in every iteration of the dict comprehension, so both are independent from each other.
As to "why" fromkeys() works like that - well, it would be surprising if it didn't work like that. fromkeys(iterable, value) constructs a new dict with keys from iterable that all have the value value. If that value is a mutable object, and you change that object, what else could you reasonably expect to happen?
To answer the actual question being asked: fromkeys behaves like that because there is no other reasonable choice. It is not reasonable (or even possible) to have fromkeys decide whether or not your argument is mutable and make new copies every time. In some cases it doesn't make sense, and in others it's just impossible.
The second argument you pass in is therefore just a reference, and is copied as such. An assignment of [] in Python means "a single reference to a new list", not "make a new list every time I access this variable". The alternative would be to pass in a function that generates new instances, which is the functionality that dict comprehensions supply for you.
Here are some options for creating multiple actual copies of a mutable container:
As you mention in the question, dict comprehensions allow you to execute an arbitrary statement for each element:
d = {k: [] for k in range(2)}
The important thing here is that this is equivalent to putting the assignment k = [] in a for loop. Each iteration creates a new list and assigns it to a value.
Use the form of the dict constructor suggested by #Andrew Clark:
d = dict((k, []) for k in range(2))
This creates a generator which again makes the assignment of a new list to each key-value pair when it is executed.
Use a collections.defaultdict instead of a regular dict:
d = collections.defaultdict(list)
This option is a little different from the others. Instead of creating the new list references up front, defaultdict will call list every time you access a key that's not already there. You can there fore add the keys as lazily as you want, which can be very convenient sometimes:
for k in range(2):
d[k].append(42)
Since you've set up the factory for new elements, this will actually behave exactly as you expected fromkeys to behave in the original question.
Use dict.setdefault when you access potentially new keys. This does something similar to what defaultdict does, but it has the advantage of being more controlled, in the sense that only the access you want to create new keys actually creates them:
d = {}
for k in range(2):
d.setdefault(k, []).append(42)
The disadvantage is that a new empty list object gets created every time you call the function, even if it never gets assigned to a value. This is not a huge problem, but it could add up if you call it frequently and/or your container is not as simple as list.
I have an interview recently. The interviewer asked me the ways to iterate dict in python. I said all the ways use for statement. But he told me that how about lambda?
I feel confused very much and I consider lambda as an anonymity function, but how it iterates a dict? some code like this:
new_dict = sorted(old_dict.items(), lambda x: x[1]) # sorted by value in dict
But in this code, the lambda is used as a function to provide the compared key. What do you think this question?
You don't iterate with lambda. There are following ways to iterate an iterable object in Python:
for statement (your answer)
Comprehension, including list [x for x in y], dictionary {key: value for key, value in x} and set {x for x in y}
Generator expression: (x for x in y)
Pass to function that will iterate it (map, all, itertools module)
Manually call next function until StopIteration happens.
Note: 3 will not iterate it unless you iterate over that generator later. In case of 4 it depends on function.
For iterating specific collections like dict or list there can be more techniques like while col: remove element or with index slicing tricks.
Now lambda comes into the picture. You can use lambdas in some of those functions, for example: map(lambda x: x*2, [1, 2, 3]). But lambda here has nothing to do with iteration process itself, you can pass a regular function map(func, [1, 2, 3]).
You can iterate dict using lambda like this:
d = {'a': 1, 'b': 2}
values = map(lambda key: d[key], d.keys())
Using a plain lambda to iterate anything in Python sounds very wrong. Certainly the most Pythonic method to iterate sequences and collections is to use list comprehensions and generator expressions like #Andrey presented.
If the interviewer was leaning on the more theoretical/Computer Sciencey answers, it is worth noting that using lambdas to iterate is quite possible, although I must stress that this is not Pythonic nor useful at any context other than academic exercises:
# the legendary Y combinator makes it possible
# to let nameless functions recurse using an indirection
Y = lambda f: (lambda x: x(x))(lambda y: f(lambda *args: y(y)(*args)))
# our iterator lambda
it = lambda f: lambda Lst: (Lst[0], f(Lst[1:])) if Lst else None
# see it in action:
Y(it)([1,2,3])
=> (1, (2, (3, None)))
lambda itself doesn't iterate anything. As you thought, it just defines an anonymous function - aside from the syntactic rule about only being able to have an expression, a lambda does nothing more than a similar function made using def. The code inside the lambda might iterate something, but only in the same ways as any other function might use (provided they are expressions, and so valid inside a lambda).
In the example you mention using sorted, the key function is called on each element of the list being sorted - but it is sorted itself that does this, and which does the iteration. When you provide a key function, sorted does something broadly similar to this:
def sorted(seq, key):
decorated = [(key(elem), i) for i, elem in enumerate(seq)]
# Sort using the normal tuple lexicographic comparisons
decorated.sort()
return [seq[i] for _,i in decorated]
As you can see, sorted does the iteration here, not the lambda. Indeed, there is no reason why the key has to be a lambda - any function (or any callable) will do as far as sorted is concerned.
At the lowest level, there is only really one way to iterate a dict (or, indeed, any other iterable) in Python, which is to use the iterator protocol. This is what the for loop does behind the scenes, and you could also use a while statement like this:
it = iter(my_iterable)
while True:
try:
val = next(it)
except StopIteration:
# Run else clause of for loop
break
else:
# Run for loop body
The comments in this aren't strictly part of the iterator protocol, they are instead part of the for loop (but having at least a loop body in there is mostly the point of iterating in the first place).
Other functions and syntax that consume iterables (such as list, set and dict comprehensions, generator expressions or builtins like sum, sorted or max) all use this protocol, by either:
Using a Python for loop,
Doing something like the above while loop (especially for modules written in C),
Delegating to another function or piece of syntax that uses one of these
A class can be made so that its instances become iterable in either of two ways:
Provide the iterator protocol directly. You need a method called __iter__ (called by iter), which returns an iterator. That iterator has a method called __next__ (just next in Python 2) which is called by next and returns the value at the iterator's current location and advances it (or raises StopIteration if it is already at the end); or
Implement part of the sequence protocol (which means behaving like a list or tuple). For forward iteration, it is sufficient to define __getitem__ in such a way that doing my_sequence[0], my_sequence[1], up until my_sequence[n-1] (where n is the number of items in the sequence), and higher indexes raise an error. You usually want to define __len__ as well, which is used when you do len(my_sequence).
Dictionary iteration using lambda
dct = {1: '1', 2 : '2'}
Iterating over Dictionary using lambda:
map(lambda x : str(x[0]) + x[1], dct.iteritems())
here x[0] is the key
and x[1] is the value
Result :
['11', '22']
Filtering on Dictionary using lambda:
filter(lambda x : x[0] > 1, dct.iteritems())
Result :
[(2, '2')]
The best way to iterate dict in python is:
dic ={}
iter_dic = dic.iteritems().next
iter_dic()
...
iter_dic()
But you can build it with lambda func:
iter_dic = lambda dic: dic.keys()[0],dic.pop(dic.keys()[0])
iter_dic(dic)
...
iter_dic(dic)
Could we say
count = {8: 'u', 4: 't', 9: 'z', 10: 'j', 5: 'k', 3: 's'}
count_updated = dict(filter(lambda val: val[0] % 3 == 0,
count.items()))
is iterating Dictionary with Lambda function?
It will produce:
{8: 'u', 4: 't', 9: 'z', 10: 'j', 5: 'k', 3: 's'}
{9: 'z', 3: 's'}
from Python dictionary filter + Examples.
I'm trying to use map() on the dict_values object returned by the values() function on a dictionary. However, I can't seem to be able to map() over a dict_values:
map(print, h.values())
Out[31]: <builtins.map at 0x1ce1290>
I'm sure there's an easy way to do this. What I'm actually trying to do is create a set() of all the Counter keys in a dictionary of Counters, doing something like this:
# counters is a dict with Counters as values
whole_set = set()
map(lambda x: whole_set.update(set(x)), counters.values())
Is there a better way to do this in Python?
In Python 3, map returns an iterator, not a list. You still have to iterate over it, either by calling list on it explicitly, or by putting it in a for loop. But you shouldn't use map this way anyway. map is really for collecting return values into an iterable or sequence. Since neither print nor set.update returns a value, using map in this case isn't idiomatic.
Your goal is to put all the keys in all the counters in counters into a single set. One way to do that is to use a nested generator expression:
s = set(key for counter in counters.values() for key in counter)
There's also the lovely dict comprehension syntax, which is available in Python 2.7 and higher (thanks Lattyware!) and can generate sets as well as dictionaries:
s = {key for counter in counters.values() for key in counter}
These are both roughly equivalent to the following:
s = set()
for counter in counters.values():
for key in counter:
s.add(key)
You want the set-union of all the values of counters? I.e.,
counters[1].union(counters[2]).union(...).union(counters[n])
? That's just functools.reduce:
import functools
s = functools.reduce(set.union, counters.values())
If counters.values() aren't already sets (e.g., if they're lists), then you should turn them into sets first. You can do it using a dict comprehension using iteritems, which is a little clunky:
>>> counters = {1:[1,2,3], 2:[4], 3:[5,6]}
>>> counters = {k:set(v) for (k,v) in counters.iteritems()}
>>> print counters
{1: set([1, 2, 3]), 2: set([4]), 3: set([5, 6])}
or of course you can do it inline, since you don't care about counters.keys():
>>> counters = {1:[1,2,3], 2:[4], 3:[5,6]}
>>> functools.reduce(set.union, [set(v) for v in counters.values()])
set([1, 2, 3, 4, 5, 6])