I've got a list of tuples extracted from a table in a DB which looks like (key , foreignkey , value). There is a many to one relationship between the key and foreignkeys and I'd like to convert it into a dict indexed by the foreignkey containing the sum of all values with that foreignkey, i.e. { foreignkey , sumof( value ) }. I wrote something that's rather verbose:
myDict = {}
for item in myTupleList:
if item[1] in myDict:
myDict [ item[1] ] += item[2]
else:
myDict [ item[1] ] = item[2]
but after seeing this question's answer or these two there's got to be a more concise way of expressing what I'd like to do. And if this is a repeat, I missed it and will remove the question if you can provide the link.
Assuming all your values are ints, you could use a defaultdict to make this easier:
from collections import defaultdict
myDict = defaultdict(int)
for item in myTupleList:
myDict[item[1]] += item[2]
defaultdict is like a dictionary, except if you try to get a key that isn't there it fills in the value returned by the callable - in this case, int, which returns 0 when called with no arguments.
UPDATE: Thanks to #gnibbler for reminding me, but tuples can be unpacked in a for loop:
from collections import defaultdict
myDict = defaultdict(int)
for _, key, val in myTupleList:
myDict[key] += val
Here, the 3-item tuple gets unpacked into the variables _, key, and val. _ is a common placeholder name in Python, used to indicate that the value isn't really important. Using this, we can avoid the hairy item[1] and item[2] indexing. We can't rely on this if the tuples in myTupleList aren't all the same size, but I bet they are.
(We also avoid the situation of someone looking at the code and thinking it's broken because the writer thought arrays were 1-indexed, which is what I thought when I first read the code. I wasn't alleviated of this until I read the question. In the above loop, however, it's obvious that myTupleList is a tuple of three elements, and we just don't need the first one.)
from collections import defaultdict
myDict = defaultdict(int)
for _, key, value in myTupleList:
myDict[key] += value
Here's my (tongue in cheek) answer:
myDict = reduce(lambda d, t: (d.__setitem__(t[1], d.get(t[1], 0) + t[2]), d)[1], myTupleList, {})
It is ugly and bad, but here is how it works.
The first argument to reduce (because it isn't clear there) is lambda d, t: (d.__setitem__(t[1], d.get(t[1], 0) + t[2]), d)[1]. I will talk about this later, but for now, I'll just call it joe (no offense to any people named Joe intended). The reduce function basically works like this:
joe(joe(joe({}, myTupleList[0]), myTupleList[1]), myTupleList[2])
And that's for a three element list. As you can see, it basically uses its first argument to sort of accumulate each result into the final answer. In this case, the final answer is the dictionary you wanted.
Now for joe itself. Here is joe as a def:
def joe(myDict, tupleItem):
myDict[tupleItem[1]] = myDict.get(tupleItem[1], 0) + tupleItem[2]
return myDict
Unfortunately, no form of = or return is allowed in a Python lambda so that has to be gotten around. I get around the lack of = by calling the dicts __setitem__ function directly. I get around the lack of return in by creating a tuple with the return value of __setitem__ and the dictionary and then return the tuple element containing the dictionary. I will slowly alter joe so you can see how I accomplished this.
First, remove the =:
def joe(myDict, tupleItem):
# Using __setitem__ to avoid using '='
myDict.__setitem__(tupleItem[1], myDict.get(tupleItem[1], 0) + tupleItem[2])
return myDict
Next, make the entire expression evaluate to the value we want to return:
def joe(myDict, tupleItem):
return (myDict.__setitem__(tupleItem[1], myDict.get(tupleItem[1], 0) + tupleItem[2]),
myDict)[1]
I have run across this use-case for reduce and dict many times in my Python programming. In my opinion, dict could use a member function reduceto(keyfunc, reduce_func, iterable, default_val=None). keyfunc would take the current value from the iterable and return the key. reduce_func would take the existing value in the dictionary and the value from the iterable and return the new value for the dictionary. default_val would be what was passed into reduce_func if the dictionary was missing a key. The return value should be the dictionary itself so you could do things like:
myDict = dict().reduceto(lambda t: t[1], lambda o, t: o + t, myTupleList, 0)
Maybe not exactly readable but it should work:
fks = dict([ (v[1], True) for v in myTupleList ]).keys()
myDict = dict([ (fk, sum([ v[2] for v in myTupleList if v[1] == fk ])) for fk in fks ])
The first line finds all unique foreign keys. The second line builds your dictionary by first constructing a list of (fk, sum(all values for this fk))-pairs and turning that into a dictionary.
Look at SQLAlchemy and see if that does all the mapping you need and perhaps more
Related
I have a dictionary with tuples as keys and floats as values. I have code that looks like
For each longtuplekey in dictionary:
dictionary[longtuplekey]+=func(dictionary[longtuplekey])
dictionary[longtyplekey]*=func2(dictionary[longtuplekey])
I'd like it to shorten it to something like
For each longtuplekey in dictionary:
val=dictionary[longtuplekey]
val+=func(val)
val*=func2(val)
Except of course that the above code defines val as an immutable float and fails to modify the dictionary entry. Is there a way to do this?
Edit: I've selected the answer that says "No, there's no explicit way to do this" and provides an alternative. If anyone comes up with a way to do exactly what I wanted, I will change the answer.
There's no need to use 2 assignments here, just do both funcs inside the imul assignment and use dict.items to loop over the keys and values simultaneously:
for longtuplekey, val in dictionary.items():
dictionary[longtuplekey] *= func2(val + func(val))
There's no bulletproof way to make Python recognize that assignments to val should update the value in the dictionary too. If you want to update the dict, you'll have to assign back to it. On the bright side, you can reduce the number of required dictionary[longtuplekey] usages by iterating over the dict's items() rather than just its keys.
for longtuplekey, val in d.items():
val += func(val)
val *= func2(val)
d[longtuplekey] = val
You can do something like this:
for longtuplekey in dictionary:
val = dictionary[longtuplekey]
val += func(val)
val *= func2(val)
dictionary[longtuplekey] = val
class SpreadsheetRow(object):
def __init__(self,Account1):
self.Account1=Account1
self.Account2=0
I have a while loop that fills a list of objects ,and another loop that fills a dictionary associating Var1:Account2. But, I need to get that dictionary's value into each object, if the key matches the object's Account1.
So basically, I have:
listofSpreadsheetRowObjects=[SpreadsheetRow1, SpreadsheetRow2, SpreadsheetRow3]
dict_var1_to_account2={1234:888, 1991:646, 90802:5443}
I've tried this:
for k, v in dict_var1_to_account2.iteritems():
if k in listOfSpreadsheetRowObjects:
if self.account1=k:
self.account2=v
But, it's not working, and I suspect it's my first "if" statement, because listOfSpreadsheetRowObjects is just a list of those objects. How would I access account1 of each object, so I can match them as needed?
Eventually, I should have three objects with the following information:
SpreadsheetRow
self.Account1=Account1
self.Account2=(v from my dictionary, if account1 matches the key in my dictionary)
You can use a generator expression within any() to check if any account1 attribute of those objects is equal with k:
if any(k == item.account1 for item in listOfSpreadsheetRows):
You can try to use the next function like this:
next(i for i in listOfSpreadsheetRows if k == i.account1)
If you have a dictionary d and want to get the value associated to the key x then you look up that value like this:
v = d[x]
So if your dictionary is called dict_of_account1_to_account2 and the key is self.Account1 and you want to set that value to self.Account2 then you would do:
self.Account2 = dict_of_account1_to_account2[self.Account1]
The whole point of using a dictionary is that you don't have to iterate through the entire thing to look things up.
Otherwise if you are doing this initialization of .Account2 after creating all the SpreadsheetRow objects then using self doesn't make sense, you would need to iterate through each SpreadsheetRow item and do the assignment for each one, something like this:
for row in listofSpreadsheetRowObjects:
for k, v in dict_of_account1_to_account2.iteritems():
if row.Account1 == k:
row.Account2 = v
But again, you don't have to iterate over the dictionary to make the assignment, just look up row.Account1 from the dict:
for row in listofSpreadsheetRowObjects:
row.Account2 = dict_of_account1_to_account2[row.Account1]
I am practically repeating the same code with only one minor change in each function, but an essential change.
I have about 4 functions that look similar to this:
def list_expenses(self):
explist = [(key,item.amount) for key, item in self.expensedict.iteritems()] #create a list from the dictionary, making a tuple of dictkey and object values
sortedlist = reversed(sorted(explist, key = lambda (k,a): (a))) #sort the list based on the value of the amount in the tuples of sorted list. Reverse to get high to low
for ka in sortedlist:
k, a = ka
print k , a
def list_income(self):
inclist = [(key,item.amount) for key, item in self.incomedict.iteritems()] #create a list from the dictionary, making a tuple of dictkey and object values
sortedlist = reversed(sorted(inclist, key = lambda (k,a): (a))) #sort the list based on the value of the amount in the tuples of sorted list. Reverse to get high to low
for ka in sortedlist:
k, a = ka
print k , a
I believe this is what they refer to as violating "DRY", however I don't have any idea how I can change this to be more DRYlike, as I have two seperate dictionaries(expensedict and incomedict) that I need to work with.
I did some google searching and found something called decorators, and I have a very basic understanding of how they work, but no clue how I would apply it to this.
So my request/question:
Is this a candidate for a decorator, and if a decorator is
necessary, could I get hint as to what the decorator should do?
Pseudocode is fine. I don't mind struggling. I just need something
to start with.
What do you think about using a separate function (as a private method) for list processing? For example, you may do the following:
def __list_processing(self, list):
#do the generic processing of your lists
def list_expenses(self):
#invoke __list_processing with self.expensedict as a parameter
def list_income(self):
#invoke __list_processing with self.incomedict as a parameter
It looks better since all the complicated processing is in a single place, list_expenses and list_income etc are the corresponding wrapper functions.
I have a list of dictionaries=
a = [{"ID":1, "VALUE":2},{"ID":2, "VALUE":2},{"ID":3, "VALUE":4},...]
"ID" is a unique identifier for each dictionary. Considering the list is huge, what is the fastest way of checking if a dictionary with a certain "ID" is in the list, and if not append to it? And then update its "VALUE" ("VALUE" will be updated if the dict is already in list, otherwise a certain value will be written)
You'd not use a list. Use a dictionary instead, mapping ids to nested dictionaries:
a = {
1: {'VALUE': 2, 'foo': 'bar'},
42: {'VALUE': 45, 'spam': 'eggs'},
}
Note that you don't need to include the ID key in the nested dictionary; doing so would be redundant.
Now you can simply look up if a key exists:
if someid in a:
a[someid]['VALUE'] = newvalue
I did make the assumption that your ID keys are not necessarily sequential numbers. I also made the assumption you need to store other information besides VALUE; otherwise just a flat dictionary mapping ID to VALUE values would suffice.
A dictionary lets you look up values by key in O(1) time (constant time independent of the size of the dictionary). Lists let you look up elements in constant time too, but only if you know the index.
If you don't and have to scan through the list, you have a O(N) operation, where N is the number of elements. You need to look at each and every dictionary in your list to see if it matches ID, and if ID is not present, that means you have to search from start to finish. A dictionary will still tell you in O(1) time that the key is not there.
If you can, convert to a dictionary as the other answers suggest, but in case you you have reason* to not change the data structure storing your items, here's what you can do:
items = [{"ID":1, "VALUE":2}, {"ID":2, "VALUE":2}, {"ID":3, "VALUE":4}]
def set_value_by_id(id, value):
# Try to find the item, if it exists
for item in items:
if item["ID"] == id:
break
# Make and append the item if it doesn't exist
else: # Here, `else` means "if the loop terminated not via break"
item = {"ID": id}
items.append(id)
# In either case, set the value
item["VALUE"] = value
* Some valid reasons I can think of include preserving the order of items and allowing duplicate items with the same id. For ways to make dictionaries work with those requirements, you might want to take a look at OrderedDict and this answer about duplicate keys.
Convert your list into a dict and then checking for values is much more efficient.
d = dict((item['ID'], item['VALUE']) for item in a)
for new_key, new_value in new_items:
if new_key not in d:
d[new_key] = new_value
Also need to update on key found:
d = dict((item['ID'], item['VALUE']) for item in a)
for new_key, new_value in new_items:
d.setdefault(new_key, 0)
d[new_key] = new_value
Answering the question you asked, without changing the datastructure around, there's no real faster way of looking without a loop and checking every element and doing a dictionary lookup for each one - but you can push the loop down to the Python runtime instead of using Python's for loop.
I haven't tried if it ends up faster though.
a = [{"ID":1, "VALUE":2},{"ID":2, "VALUE":2},{"ID":3, "VALUE":4}]
id = 2
tmp = filter(lambda d: d['ID']==id, a)
# the filter will either return an empty list, or a list of one item.
if not tmp:
tmp = {"ID":id, "VALUE":"default"}
a.append(tmp)
else:
tmp = tmp[0]
# tmp is bound to the found/new dictionary
I have a dict that has string-type keys whose exact values I can't know (because they're generated dynamically elsewhere). However, I know that that the key I want contains a particular substring, and that a single key with this substring is definitely in the dict.
What's the best, or "most pythonic" way to retrieve the value for this key?
I thought of two strategies, but both irk me:
for k,v in some_dict.items():
if 'substring' in k:
value = v
break
-- OR --
value = [v for (k,v) in some_dict.items() if 'substring' in k][0]
The first method is bulky and somewhat ugly, while the second is cleaner, but the extra step of indexing into the list comprehension (the [0]) irks me. Is there a better way to express the second version, or a more concise way to write the first?
There is an option to write the second version with the performance attributes of the first one.
Use a generator expression instead of list comprehension:
value = next(v for (k,v) in some_dict.iteritems() if 'substring' in k)
The expression inside the parenthesis will return an iterator which you will then ask to provide the next, i.e. first element. No further elements are processed.
How about this:
value = (v for (k,v) in some_dict.iteritems() if 'substring' in k).next()
It will stop immediately when it finds the first match.
But it still has O(n) complexity, where n is the number of key-value pairs. You need something like a suffix list or a suffix tree to speed up searching.
If there are many keys but the string is easy to reconstruct from the substring, then it can be faster reconstructing it. e.g. often you know the start of the key but not the datestamp that has been appended on. (so you may only have to try 365 dates rather than iterate through millions of keys for example).
It's unlikely to be the case but I thought I would suggest it anyway.
e.g.
>>> names={'bob_k':32,'james_r':443,'sarah_p':12}
>>> firstname='james' #you know the substring james because you have a list of firstnames
>>> for c in "abcdefghijklmnopqrstuvwxyz":
... name="%s_%s"%(firstname,c)
... if name in names:
... print name
...
james_r
class MyDict(dict):
def __init__(self, *kwargs):
dict.__init__(self, *kwargs)
def __getitem__(self,x):
return next(v for (k,v) in self.iteritems() if x in k)
# Defining several dicos ----------------------------------------------------
some_dict = {'abc4589':4578,'abc7812':798,'kjuy45763':1002}
another_dict = {'boumboum14':'WSZE x478',
'tagada4783':'ocean11',
'maracuna102455':None}
still_another = {12:'jfg',45:'klsjgf'}
# Selecting the dicos whose __getitem__ method will be changed -------------
name,obj = None,None
selected_dicos = [ (name,obj) for (name,obj) in globals().iteritems()
if type(obj)==dict
and all(type(x)==str for x in obj.iterkeys())]
print 'names of selected_dicos ==',[ name for (name,obj) in selected_dicos]
# Transforming the selected dicos in instances of class MyDict -----------
for k,v in selected_dicos:
globals()[k] = MyDict(v)
# Exemple of getting a value ---------------------------------------------
print "some_dict['7812'] ==",some_dict['7812']
result
names of selected_dicos == ['another_dict', 'some_dict']
some_dict['7812'] == 798
I prefer the first version, although I'd use some_dict.iteritems() (if you're on Python 2) because then you don't have to build an entire list of all the items beforehand. Instead you iterate through the dict and break as soon as you're done.
On Python 3, some_dict.items(2) already results in a dictionary view, so that's already a suitable iterator.