Related
Is there a way to return various sums of a list of integers? Pythonic or otherwise.
For e.g. various sum totals from [1, 2, 3, 4] would produce 1+2=3, 1+3=4, 1+4=5, 2+3=5, 2+4=6, 3+4=7. Integers to be summed could by default be stuck to two integers only or more I guess.
Can't seem to wrap my head around how to tackle this and can't seem to find an example or explanation on the internet as they all lead to "Sum even/odd numbers in list" and other different problems.
You can use itertools.combinations and sum:
from itertools import combinations
li = [1, 2, 3, 4]
# assuming we don't need to sum the entire list or single numbers,
# and that x + y is the same as y + x
for sum_size in range(2, len(li)):
for comb in combinations(li, sum_size):
print(comb, sum(comb))
outputs
(1, 2) 3
(1, 3) 4
(1, 4) 5
(2, 3) 5
(2, 4) 6
(3, 4) 7
(1, 2, 3) 6
(1, 2, 4) 7
(1, 3, 4) 8
(2, 3, 4) 9
is this what you are looking for ?
A=[1,2,3,4] for i in A: for j in A: if i!=j: print(i+j)
I have a dictionary in this format:
d = {'type 1':[1,2,3],'type 2':['a','b','c']}
It's a symmetric dictionary, in the sense that for each key I'll always have the same number of elements.
There is a way to loop as if it were rows:
for row in d.rows():
print row
So I get the output:
[1]: 1, a
[2]: 2, b
[3]: 3, b
You can zip the .values(), but the order is not guaranteed unless you are using a collections.OrderedDict (or a fairly recent version of PyPy):
for row in zip(*d.values()):
print(row)
e.g. when I run this, I get
('a', 1)
('b', 2)
('c', 3)
but I could have just as easily gotten:
(1, 'a')
(2, 'b')
(3, 'c')
(and I might get it on future runs if hash randomization is enabled).
If you want a specified order and have a finite set of keys that you know up front, you can just zip those items directly:
zip(d['type 1'], d['type 2'])
If you want your rows ordered alphabetically by key you might consider:
zip(*(v for k, v in sorted(d.iteritems())))
# [(1, 'a', 'd', 4), (2, 'b', 'e', 5), (3, 'c', 'f', 6)]
If your dataset is huge then consider itertools.izip or pandas.
import pandas as pd
df = pd.DataFrame(d)
print df.to_string(index=False)
# type 1 type 2 type 3 type 4
# 1 a d 4
# 2 b e 5
# 3 c f 6
print df.to_csv(index=False, header=False)
# 1,a,d,4
# 2,b,e,5
# 3,c,f,6
Even you pass the dictionary to OrderedDict you will not get ordered result as dictionary entry. So here tuples of tuple can be a good option.
See the code and output below:
Code (Python 3):
import collections
t = (('type 1',[1,2,3]),
('type 2',['a','b','c']),
('type 4',[4,5,6]),
('type 3',['d','e','f']))
d = collections.OrderedDict(t)
d_items = list(d.items())
values_list = []
for i in range(len(d_items)):
values_list.append(d_items[i][1])
values_list_length = len(values_list)
single_list_length = len(values_list[0])
for i in range(0,single_list_length):
for j in range(0,values_list_length):
print(values_list[j][i],' ',end='')
print('')
Output:
1 a 4 d
2 b 5 e
3 c 6 f
Background:
I have a list of 44906 items: large = [1, 60, 17, ...]. I also have a personal computer with limited memory (8GB), running Ubuntu 14.04.4 LTS.
The Goal:
I need to find all the pair-wise combinations of large in a memory-efficient manner, without filling a list with all the combinations beforehand.
The Problem & What I've Tried So Far:
When I use itertools.combinations(large, 2), and try to assign it to a list my memory fills up immediately, and I get very slow performance. The reason for this is that the number of pairwise combinations goes like n*(n-1)/2 where n is the number of elements of the list.
The number of combinations for n=44906 comes out to 44906*44905/2 = 1008251965. A list with this many entries is much too large to store in memory. I would like to be able to design a function so that I can plug in a number i to find the ith pair-wise combination of numbers in this list, and a way to somehow dynamically compute this combination, without referring to a 1008251965 element list that's impossible to store in memory.
An Example of What I Am Trying To Do:
Let's say I have an array small = [1,2,3,4,5]
In the configuration in which I have the code, itertools.combinations(small, 2) will return a list of tuples as such:
[(1, 2), # 1st entry
(1, 3), # 2nd entry
(1, 4), # 3rd entry
(1, 5), # 4th entry
(2, 3), # 5th entry
(2, 4), # 6th entry
(2, 5), # 7th entry
(3, 4), # 8th entry
(3, 5), # 9th entry
(4, 5)] # 10th entry
A call to a the function like this: `find_pair(10)' would return:
(4, 5)
, giving the 10th entry in the would-be array, but without calculating the entire combinatorial explosion beforehand.
The thing is, I need to be able to drop in to the middle of the combinations, not starting from the beginning every time, which is what it seems like an iterator does:
>>> from itertools import combinations
>>> it = combinations([1, 2, 3, 4, 5], 2)
>>> next(it)
(1, 2)
>>> next(it)
(1, 3)
>>> next(it)
(1, 4)
>>> next(it)
(1, 5)
So, instead of having to execute next() 10 times to get to the 10th combination, I would like to be able to retrieve the tuple returned by the 10th iteration with one call.
The Question
Are there any other combinatorial functions that behave this way designed to deal with huge data sets? If not, is there a good way to implement a memory-saving algorithm that behaves this way?
Except itertools.combinations does not return a list - it returns an iterator. Here:
>>> from itertools import combinations
>>> it = combinations([1, 2, 3, 4, 5], 2)
>>> next(it)
(1, 2)
>>> next(it)
(1, 3)
>>> next(it)
(1, 4)
>>> next(it)
(1, 5)
>>> next(it)
(2, 3)
>>> next(it)
(2, 4)
and so on. It's extremely memory-efficient: only one pair is produced per invocation.
Of course it is possible to write a function that returns the n'th result, but before bothering with that (which will be slower and more involved), are you quite sure you can't just use combinations() the way it was designed to be used (i.e., iterating over it, instead of forcing it to produce a giant list)?
If you want random access to any combination you can use this function to return the index of a corresponding lower triangular representation of the cross-product
def comb(k):
row=int((math.sqrt(1+8*k)+1)/2)
column=int(k-(row-1)*(row)/2)
return [row,column]
using your small array for example
small = [1,2,3,4,5]
length = len(small)
size = int(length * (length-1)/2)
for i in range(size):
[n,m] = comb(i)
print(i,[n,m],"(",small[n],",",small[m],")")
will give
0 [1, 0] ( 2 , 1 )
1 [2, 0] ( 3 , 1 )
2 [2, 1] ( 3 , 2 )
3 [3, 0] ( 4 , 1 )
4 [3, 1] ( 4 , 2 )
5 [3, 2] ( 4 , 3 )
6 [4, 0] ( 5 , 1 )
7 [4, 1] ( 5 , 2 )
8 [4, 2] ( 5 , 3 )
9 [4, 3] ( 5 , 4 )
obviously if your access method is in order other methods will be more practical.
Note also that the comb function is independent of the size of the problem.
As suggested by #Blckknght in the comments to get the same order as itertools version change to
for i in range(size):
[n,m] = comb(size-1-i)
print(i,[n,m],"(",small[length-1-n],",",small[length-1-m],")")
0 [4, 3] ( 1 , 2 )
1 [4, 2] ( 1 , 3 )
2 [4, 1] ( 1 , 4 )
3 [4, 0] ( 1 , 5 )
4 [3, 2] ( 2 , 3 )
5 [3, 1] ( 2 , 4 )
6 [3, 0] ( 2 , 5 )
7 [2, 1] ( 3 , 4 )
8 [2, 0] ( 3 , 5 )
9 [1, 0] ( 4 , 5 )
I started with that triangular arrangement, finding the subscript k for list members indexed row and col. Then I reversed the process, deriving row and col from k.
For a list large of N items, let
b = 2*N - 1
Now, to get the kth combination in the list ...
row = (b - math.sqrt(b*b - 8*k)) // 2
col = k - (2*N - row + 1)*row / 2
kth_pair = large[row][col]
This allows you to access any member of the combinations list without ever generating that list.
So you've got 44906 items. Notice, however, that if you build your combinations the same way you build them in the example then there are 44905 combinations with large[0] as the first number. Furthermore, combination i for i <= 44905 looks like (large[0], large[i]).
For 44905 < i <= 89809, it looks like (large[1],large[i-44904]).
If I'm not mistaken, this pattern should continue with something like (large[j],large[i-(exclusive lower bound for j)+1]). You can check my math on that but I'm pretty sure it's right. Anyways, you could iterate to find these lower bounds (so for j=0, it's 0, for j=1, it's 44905, etc.) Iterating should be easy because you just add the next descending number: 44905, 44905+44904, 44905+44904+44903...
For well defined order of created pairs, indices of first and second elements should be related to n and length of a sequence. If you'll find them, you'll be able to achieve const-time performance, since indexing lists is O(1) operation.
Pseudocode would look like this:
def find_nth_pair(seq, n):
idx1 = f1(n, len(seq)) # some formula of n and len(seq)
idx2 = f2(n, len(seq)) # some formula of n and len(seq)
return (seq[idx1], seq[idx2])
You only need to find formulas for idx1 and idx2.
I know how to get the most frequent element of list of list, e.g.
a = [[3,4], [3,4],[3,4], [1,2], [1,2], [1,1],[1,3],[2,2],[3,2]]
print max(a, key=a.count)
should print [3, 4] even though the most frequent number is 1 for the first element and 2 for the second element.
My question is how to do the same kind of thing with Pandas.DataFrame.
For example, I'd like to know the implementation of the following method get_max_freq_elem_of_df:
def get_max_freq_elem_of_df(df):
# do some things
return freq_list
df = pd.DataFrame([[3,4], [3,4],[3,4], [1,2], [1,2], [1,1],[1,3],[2,2],[4,2]])
x = get_max_freq_elem_of_df(df)
print x # => should print [3,4]
Please notice that DataFrame.mode() method does not work. For above example, df.mode() returns [1, 2] not [3,4]
Update
have explained why DataFrame.mode() doesn't work.
You could use groupby.size and then find the max:
>>> df.groupby([0,1]).size()
0 1
1 1 1
2 2
3 1
2 2 1
3 4 3
4 2 1
dtype: int64
>>> df.groupby([0,1]).size().idxmax()
(3, 4)
In python you'd use Counter*:
In [11]: from collections import Counter
In [12]: c = Counter(df.itertuples(index=False))
In [13]: c
Out[13]: Counter({(3, 4): 3, (1, 2): 2, (1, 3): 1, (2, 2): 1, (4, 2): 1, (1, 1): 1})
In [14]: c.most_common(1) # get the top 1 most common items
Out[14]: [((3, 4), 3)]
In [15]: c.most_common(1)[0][0] # get the item (rather than the (item, count) tuple)
Out[15]: (3, 4)
* Note that your solution
max(a, key=a.count)
(although it works) is O(N^2), since on each iteration it needs to iterate through a (to get the count), whereas Counter is O(N).
This question already has answers here:
How can I iterate over overlapping (current, next) pairs of values from a list?
(12 answers)
Closed 2 years ago.
Given a list
l = [1, 7, 3, 5]
I want to iterate over all pairs of consecutive list items (1,7), (7,3), (3,5), i.e.
for i in xrange(len(l) - 1):
x = l[i]
y = l[i + 1]
# do something
I would like to do this in a more compact way, like
for x, y in someiterator(l): ...
Is there a way to do do this using builtin Python iterators? I'm sure the itertools module should have a solution, but I just can't figure it out.
Just use zip
>>> l = [1, 7, 3, 5]
>>> for first, second in zip(l, l[1:]):
... print first, second
...
1 7
7 3
3 5
If you use Python 2 (not suggested) you might consider using the izip function in itertools for very long lists where you don't want to create a new list.
import itertools
for first, second in itertools.izip(l, l[1:]):
...
Look at pairwise at itertools recipes: http://docs.python.org/2/library/itertools.html#recipes
Quoting from there:
def pairwise(iterable):
"s -> (s0,s1), (s1,s2), (s2, s3), ..."
a, b = tee(iterable)
next(b, None)
return izip(a, b)
A General Version
A general version, that yields tuples of any given positive natural size, may look like that:
def nwise(iterable, n=2):
iters = tee(iterable, n)
for i, it in enumerate(iters):
next(islice(it, i, i), None)
return izip(*iters)
I would create a generic grouper generator, like this
def grouper(input_list, n = 2):
for i in xrange(len(input_list) - (n - 1)):
yield input_list[i:i+n]
Sample run 1
for first, second in grouper([1, 7, 3, 5, 6, 8], 2):
print first, second
Output
1 7
7 3
3 5
5 6
6 8
Sample run 1
for first, second, third in grouper([1, 7, 3, 5, 6, 8], 3):
print first, second, third
Output
1 7 3
7 3 5
3 5 6
5 6 8
Generalizing sberry's approach to nwise with comprehension:
def nwise(lst, k=2):
return list(zip(*[lst[i:] for i in range(k)]))
Eg
nwise(list(range(10)),3)
[(0, 1, 2), (1, 2, 3), (2, 3, 4), (3, 4, 5), (4, 5, 6), (5, 6,
7), (6, 7, 8), (7, 8, 9)]
A simple means to do this without unnecessary copying is a generator that stores the previous element.
def pairs(iterable):
"""Yield elements pairwise from iterable as (i0, i1), (i1, i2), ..."""
it = iter(iterable)
try:
prev = next(it)
except StopIteration:
return
for item in it:
yield prev, item
prev = item
Unlike index-based solutions, this works on any iterable, including those for which indexing is not supported (e.g. generator) or slow (e.g. collections.deque).
You could use a zip.
>>> list(zip(range(5), range(2, 6)))
[(0, 1), (1, 2), (2, 3), (3, 4), (4, 5)]
Just like a zipper, it creates pairs. So, to to mix your two lists, you get:
>>> l = [1,7,3,5]
>>> list(zip(l[:-1], l[1:]))
[(1, 7), (7, 3), (3, 5)]
Then iterating goes like
for x, y in zip(l[:-1], l[1:]):
pass
If you wanted something inline but not terribly readable here's another solution that makes use of generators. I expect it's also not the best performance wise :-/
Convert list into generator with a tweak to end before the last item:
gen = (x for x in l[:-1])
Convert it into pairs:
[(gen.next(), x) for x in l[1:]]
That's all you need.