We Keep Coding

Averaging values within a list of lists

I'm messing around with lists of lists containing strings and values EX: LofL = [["string", 4.0, 1.1, -3.0, -7.2],["string", 2.0, -1.0, 3.3], ["string", 4.4, 5.5, -6.6, 1.1]] and I'm trying to take the values within each list within the list, and average them as long as the values are not below 0. For example the first would be 5.1/2 since the third digit is negative. This in the end would make the List of lists look like: LofL =[["string", 5.1/2],["string", 2/1], ["string", 9.9/2]]. I've tried this so far:
LofL = *see above example*
avgLofL = LofL
for sublist in LofL:
while sublist in range(1,len(sublist)) > 0.0:
rowavg = [sum(sublist) / range(1,len(sublist)) for sublist in LofL]
for sublist in avgLofL:
for sublist in range(1,len(sublist)):
avgLofL.append(rowavg)
return avgLofL
It says my rowavg isn't referenced before assingment, but when I intitialize it as rowavg = 0 my list has no length. I'm unsure where I'm making a mistake

This is a possible solution:
from statistics import mean
avgLofL = [[next(x for x in lst if isinstance(x, str)),
mean(x for x in lst if not isinstance(x, str) and x >= 0)]
for lst in LofL]

Ok, I think this is what you actually asked for:
LofL = [["string", 4.0, 1.1, -3.0, -7.2],
["string", 2.0, -1.0, 3.3],
["string", 4.4, 5.5, -6.6, 1.1]]
avgLofL = []
for row in LofL:
sublist = []
for x in row[1:]:
if x>=0:
sublist.append(x)
else:
break
avgLofL.append([row[0], sum(sublist)/float(len(sublist))])
print(avgLofL)
Its result seems to match the example:
[['string', 2.55], ['string', 2.0], ['string', 4.95]]
It processes one row at a time, completely. Assumes that the first element is a string which should be kept, then collects the other elements in sublist until it finds a negative one. Then calculates the average of the collection, builds and stores a "[string,average]" pair, and continues with the next row.
In its current form it will die on having a negative number right at the start (division by zero). You can either drop an explicit if somewhere, or some dirty hack, like sum(sublist)/max(1,float(len(sublist))).

def avgofList(LofL):
avgLofL = []
for sublist in LofL:
total = 0
count = 0
for item in sublist:
if isinstance(item, float) and item > 0:
total += item
count += 1
sl = [x for x in sublist if not isinstance(x, float)]
sl.append(total / count)
avgLofL.append(sl)
return avgLofL

Very similar to the answer provided by #tevemadar except that no intermediate lists are used and also accounts for the first negative number being in index 1 of the sub-lists. It does however assume that the sublists are not empty and that the first element is to be retained.
LofL = [["string", 4.0, 1.1, -3.0, -7.2],
["string", 2.0, -1.0, 3.3],
["string", 4.4, 5.5, -6.6, 1.1]]
def process(e):
t, n = 0, 0
for v in e[1:]:
if v >= 0:
t += v
n += 1
else:
break
return [e[0], t / n] if n > 0 else [e[0]]
result = [process(e) for e in LofL]
print(result)
Output:
[['string', 2.55], ['string', 2.0], ['string', 4.95]]

Sum of duplicate values in 2d array

So, I'm sure similar questions have been asked before but I couldn't find quite what I need.
I have a program that outputs a 2D array like the one below:
arr = [[0.2, 3], [0.3, "End"], ...]
There may be more or less elements, but each is a 2-element array, where the first value is a float and the second can be a float or a string.
Both of those values may repeat. In each of those arrays, the second element takes on only a few possible values.
What I want to do is sum the first elements' value within the arrays that have the same value of the second element and output a similar array that does not have those duplicated values.
For example:
input = [[0.4, 1.5], [0.1, 1.5], [0.8, "End"], [0.05, "End"], [0.2, 3.5], [0.2, 3.5]]
output = [[0.5, 1.5], [0.4, 3.5], [0.85, "End"]]
I'd appreciate if the output array was sorted by this second element (floats ascending, strings at the end), although it's not necessary.
EDIT: Thanks for both answers; I've decided to use the one by Chris, because the code was more comprehensible to me, although groupby seems like a function designed to solved this very problem, so I'll try to read up on that, too.
UPDATE: The values of floats were always positive, by nature of the task at hand, so I used negative values to stop the usage of any strings - now I have a few if statements that check for those "encoded" negative values and replace them with strings again just before they're printed out, so sorting is now easier.

You could use a dictionary to accumulate the sum of the first value in the list keyed by the second item.
To get the 'string' items at the end of the list, the sort key could be set to positive infinity, float('inf'), in the sort key .
input_ = [[0.4, 1.5], [0.1, 1.5], [0.8, "End"], [0.05, "End"], [0.2, 3.5], [0.2, 3.5]]
d = dict()
for pair in input_:
d[pair[1]] = d.get(pair[1], 0) + pair[0]
L = []
for k, v in d.items():
L.append([v,k])
L.sort(key=lambda x: x[1] if type(x[1]) == float else float('inf'))
print(L)
This prints:
[[0.5, 1.5], [0.4, 3.5], [0.8500000000000001, 'End']]

You can try to play with itertools.groupby:
import itertools
out = [[key, sum([elt[0]for elt in val])] for key, val in itertools.groupby(a, key=lambda elt: elt[1])]
>>> [[0.5, 1.5], [0.8500000000000001, 'End'], [0.4, 3.5]]
Explanation:
Groupby the 2D list according to the 2nd element of each sublist using itertools.groupby and the key parameters. We define the lambda key=lambda elt: elt[1] to groupby on the 2nd element:
for key, val in itertools.groupby(a, key=lambda elt: elt[1]):
print(key, val)
# 1.5 <itertools._grouper object at 0x0000026AD1F6E160>
# End <itertools._grouper object at 0x0000026AD2104EF0>
# 3.5 <itertools._grouper object at 0x0000026AD1F6E160>
For each value of the group, compute the sum using the buildin function sum:
for key, val in itertools.groupby(a, key=lambda elt: elt[1]):
print(sum([elt[0]for elt in val]))
# 0.5
# 0.8500000000000001
# 0.4
Compute the desired output:
out = []
for key, val in itertools.groupby(a, key=lambda elt: elt[1]):
out.append([sum([elt[0]for elt in val]), key])
print(out)
# [[0.5, 1.5], [0.8500000000000001, 'End'], [0.4, 3.5]]
Then you said about sorting on the 2nd value but there are strings and numbers, it's quite a problem for the computer. It can't make a choice between a number and a string. Objects must be comparable.

Find the k dict items whose values are the closest to a certain value

Let's say we want to find the 2 items which have their value the closest to 10:
A = {'abc': 12.3, 'def': 17.3, 'dsfsf': 18, 'ppp': 3.2, "jlkljkjlk": 9.23}
It works with:
def nearest(D, centre, k=10):
return sorted([[d, D[d], abs(D[d] - centre)] for d in D], key=lambda e: e[2])[:k]
print(nearest(A, centre=10, k=2))
[['jlkljkjlk', 9.23, 0.7699999999999996], ['abc', 12.3, 2.3000000000000007]]
But is there a Python built-in way to do this and/or a more optimized version when the dict has a much larger size (hundreds of thousands of items)?

If you do not mind using Pandas:
import pandas as pd
closest = (pd.Series(A) - 10).abs().sort_values()[:2]
#jlkljkjlk 0.77
#abc 2.30
closest.to_dict()
#{'jlkljkjlk': 0.7699999999999996, 'abc': 2.3000000000000007}

You could use heapq.nsmallest():
from heapq import nsmallest
A = {'abc': 12.3, 'def': 17.3, 'dsfsf': 18, 'ppp': 3.2, 'jlkljkjlk': 9.23}
def nearest(D, centre, k=10):
return [[x, D[x], abs(D[x] - centre)] for x in nsmallest(k, D, key=lambda x: abs(D[x] - centre))]
print(nearest(A, centre=10, k=2))
# [['jlkljkjlk', 9.23, 0.7699999999999996], ['abc', 12.3, 2.3000000000000007]]
As far as time complexity, this runs in O(n log(k)) time instead of O(n log(n)) of the solution based on sorting the dictionary.

Given you need to perform a lookup quite often, we can make this an O(log n) algorithm, by first storing the data in a sorted list:
from operator import itemgetter
ks = sorted(A.items(), key=itemgetter(1))
vs = list(map(itemgetter(1), ks))
Then for each item we can use the bisect.bisect_left point to determine the insertion point. We can then check the two surrounding values, to check the smallest, and return the corresponding key. It is also possible that
from bisect import bisect_left
from operator import itemgetter
def closests(v):
idx = bisect_left(vs, v)
i, j = max(0, idx-1), min(idx+2, len(ks))
part = ks[i:j]
return sorted([[*pi, abs(pi[-1]-v)] for pi in part], key=itemgetter(-1))[:2]
The above might not look as an improvement, but here we will always evaluate at most three elements in the sorted(..), and bisect_left will evaluate a logarithmic number of elements.
For example:
>>> closests(1)
[['ppp', 3.2, 2.2], ['jlkljkjlk', 9.23, 8.23]]
>>> closests(3.2)
[['ppp', 3.2, 0.0], ['jlkljkjlk', 9.23, 6.03]]
>>> closests(5)
[['ppp', 3.2, 1.7999999999999998], ['jlkljkjlk', 9.23, 4.23]]
>>> closests(9.22)
[['jlkljkjlk', 9.23, 0.009999999999999787], ['abc', 12.3, 3.08]]
>>> closests(9.24)
[['jlkljkjlk', 9.23, 0.009999999999999787], ['abc', 12.3, 3.0600000000000005]]
The "loading" phase thus takes O(n log n) (with n the number of elements). Then if we generalize the above method to fetch k elements (by increasing the slice), it would take O(log n + k log k) to perform a lookup.

find indices of duplicate floats in a list

I have an input of a very large list of floating point numbers, a sample is given
[1.2, 2.4, 3.1, 4.0, 5.6, 6.5, 1.2, 3.1, 8.1, 23.6, 29.3]
I want to find all the duplicates and their index i.e. location in the list. The duplicates will only occur as a pair; never more than twice.
The output should be like
1.2 1 7
3.1 3 8
so there are just two entries 1.2 and 3.1 which occur as duplicates, and their positions are 1, 7 and 3, 8 respectively.
any suggestions with python?

Taking xi_'s answer a bit further. By adding the list comprehension it will provide a list of all indices that contain the value.
x = [1.2, 2.4, 3.1, 4.0, 5.6, 6.5, 1.2, 3.1, 8.1, 23.6, 29.3]
for el in set(x):
if x.count(el) > 1:
print el, " ".join([str(index) for index, value in enumerate(x) if value == el])
You will get an output of: (0-based index)
1.2 0 6
3.1 2 7
Edit
Explanation of [str(index) for index, value in enumerate(x) if value == el]
This is enumerating x which creates an enumerate object of the list which will return tuple pairs of (<index>, <value>)
Then it loops through this enumerate object using the for index, value in enumerate(x)
The if value == el checks each value and if it is equal to el then we evaluate, otherwise we do nothing.
The str(index) is the part that gets evaluated based on the condition we defined above. It returns a string version of index which is an integral type.
This will provide a list (all the code between the [ and ]) which will then be passed to the string method join(list) which joins all of the items in the list with the value in the " " (in this case a space, it could be any string.) providing a string of space separated values from the list that was created.
I also assume that you may even want this data later on other than just printing it. Here is a version to do that. This creates an empty dictionary y = {} then we create a new entry with a key of the value (el), providing it a list of the indices.
x = [1.2, 2.4, 3.1, 4.0, 5.6, 6.5, 1.2, 3.1, 8.1, 23.6, 29.3]
y = {}
for el in set(x):
if x.count(el) > 1:
y[el] = [str(index) for index, value in enumerate(x) if value == el]
If you do a print y this is what you should get:
{3.1: ['2', '7'], 1.2: ['0', '6']}
Edit2
To print y so that it matches the output you specified. Do something like this:
print "\n".join(["{} {}".format(key, " ".join(vals)) for key, vals in y.iteritems()])
output:
3.1 2 7
1.2 0 6
What this is doing is iterating through the dictionary y with: (for key, vals in y.iteritems()) making a string of "<key> <values...>" with: ("{} {}".format(key, " ".join(vals))) This returns a list of strings so we join them using "\n" to make them on each line.
Now it is important to note that since dictionary is a hash, that your output order of the keys will not be sorted. If you want to do that, then you could change the code above to this:
print "\n".join(["{} {}".format(key, " ".join(y[key])) for key in sorted(y.keys())])
output:
1.2 0 6
3.1 2 7

You could try something like that:
x = [1.2, 2.4, 3.1, 4.0, 5.6, 6.5, 1.2, 3.1, 8.1, 23.6, 29.3]
for el in set(x):
if x.count(el) > 1:
print el, x.count(el), len(x) - x[::-1].index(el)
Output (element with duplicates, quantity, index of last occurrence):
1.2 2 7
3.1 2 8

Fastest way to rank items with multiple values and weightings

I have a collection of key value pairs like this:
{
'key1': [value1_1, value2_1, value3_1, ...],
'key2': [value1_2, value2_2, value3_2, ...],
...
}
and also a list which is in the same order as the values list, which contains the weight each variable should have applied. So it looks like [weight_1, weight_2, weight_3, ...].
My goal is to end up with an ordered list of keys in accordance to which has the highest overall score of values. Note that the values aren't all standardized / normalized, so value1_x could range from 1 - 10 but value 2_x could range from 1 - 100000. This has been the tricky part for me as I have to normalize the data somehow.
I'm trying to make this algorithm run to scale for many different values, so it would take the same amount of time for 1 or for 100 (or at least logarithmically more time). Is that possible? Is there any really efficient way I can go about this?

You can't get linear-time, but you can do it faster; this looks like a matrix-multiply to me, so I suggest you use numpy:
import numpy as np
keys = ['key1', 'key2', 'key3']
values = np.matrix([
[1.1, 1.2, 1.3, 1.4],
[2.1, 2.2, 2.3, 2.4],
[3.1, 3.2, 3.3, 3.4]
])
weights = np.matrix([[10., 20., 30., 40.]]).transpose()
res = (values * weights).transpose().tolist()[0]
items = zip(res, keys)
items.sort(reverse=True)
which gives
[(330.0, 'key3'), (230.0, 'key2'), (130.0, 'key1')]
Edit: with thanks to #Ondro for np.dot and to #unutbu for np.argsort, here is an improved version entirely in numpy:
import numpy as np
# set up values
keys = np.array(['key1', 'key2', 'key3'])
values = np.array([
[1.1, 1.2, 1.3, 1.4], # values1_x
[2.1, 2.2, 2.3, 2.4], # values2_x
[3.1, 3.2, 3.3, 3.4] # values3_x
])
weights = np.array([10., 20., 30., 40.])
# crunch the numbers
res = np.dot(values, -weights) # negative of weights!
order = res.argsort(axis=0) # sorting on negative value gives
# same order as reverse-sort; there does
# not seem to be any way to reverse-sort
# directly
sortedkeys = keys[order].tolist()
which results in ['key3', 'key2', 'key1'].

Here's a normalization function, that will linearly transform your values into [0,1]
def normalize(val, ilow, ihigh, olow, ohigh):
return ((val-ilow) * (ohigh-olow) / (ihigh - ilow)) + olow
Now, use normalize to compute a new dictionary with normalized values. Then, sort by the weighted sum:
def sort(d, weights, ranges):
# ranges is a list of tuples containing the lower and upper bounds of the corresponding value
newD = {k:[normalize(v,ilow, ihigh, 0, 1) for v,(ilow, ihigh) in zip(vals, ranges)] for k,val in d.iteritems()} # d.items() in python3
return sorted(newD, key=lambda k: sum(v*w for v,w in zip(newD[k], weights)))