Frequency analysis of values produced by generator - python

I'm doing some analysis of images, and I have a generator that gives me all pixels in image:
def pixels_g(img):
w, h = img.shape
for y in range(0, h):
for x in range(0, w):
yield img[y][x]
It's output if converted to list would be something like
[0, 1, 2, 5, 240, 5, ... ]
Now I'd like to get "frequency table" for it:
{
0: 0,
1: 5,
2: 10,
3: 0,
4: 0,
#snip
255: 7
}
I've found some useful options in this question, but they all work with list, and I don't think that creating a list from my generator is a good idea - it can have millions of elements.
I'm therefore looking for a way to do this while preserving the benefits of generator. I'll process many images and don't want to hog resources too much.

Use Counter from "collections". It works with any iterable, not just lists.
from collections import Counter
pixels = pixels_g(img)
c = Counter(pixels)
print c[4]
If you need to reuse the contents of 'pixels' generator instance after running it through Counter, use 'tee' from itertools:
from collections import Counter
from itertools import tee
(pixels, hist) = tee(pixels_g(img))
c = Counter(pixels)
# can use hist for something else

Related

How to get two synchronised generators from a function

i have a nested tuple like this one :
this_one = (w,h,(1,2,4,8,16),(0,2,3,4,6),("0"),("0"),(0,1))
It will be used to feed:
itertools.product(*this_one)
w and h have to be both generators.
Generated values from h depends on generated values from w as in this function:
def This_Function(maxvalue):
for i in range(maxvalue):
w_must_generate = i
for j in range(i+1):
h_must_generate = j
Iv tried using a yield in This_function(maxvalue):
def This_Function(maxvalue):
for i in range(maxvalue):
for j in range(i+1):
yield i,j
Returning a single generator object which gives:
which_gives = itertools.product(*this_one)
for result in which_gives:
print(result)
...
((23,22),(16),(6),(0),(0),(0))
((23,22),(16),(6),(0),(0),(1))
((23,23),(1),(0),(0),(0),(0))
...
A single tuple at result[0] holding 2 values.
And this is not what i want .
The result i want has to be like the following:
the_following = itertools.product(*this_one)
for result in the_following:
print(result)
...
((23),(22),(16),(6),(0),(0),(0))
((23),(22),(16),(6),(0),(0),(1))
((23),(23),(1),(0),(0),(0),(0))
...
Without having to do something like :
the_following = itertools.product(*this_one)
for result in the_following:
something_like = tuple(result[0][0],result[0][1],result[1],...,result[5])
print(something_like)
...
((23),(22),(16),(6),(0),(0),(0))
((23),(22),(16),(6),(0),(0),(1))
((23),(23),(1),(0),(0),(0),(0))
...
Any ideas ?
That's fundamentally not how product works. From the documentation:
Before product() runs, it completely consumes the input iterables, keeping pools of values in memory to generate the products. Accordingly, it is only useful with finite inputs.
Thus, having the second one be dependent won't get you the result you want, assuming that is for h values to always be w+1..max in the resulting output sequence.
I'm not sure why you couldn't just run it with your This_function approach and destructure it with a generator expression:
result = ((a, b, c, d, e, f, g)
for ((a, b), c, d, e, f, g) in itertools.product(*this_one))
Since the values are all entirely determined, this works and is fairly legible:
from itertools import product
maxvalue = 5
w = (i for i in range(maxvalue) for j in range(i + 1))
h = (j for i in range(maxvalue) for j in range(i + 1))
this_one = (w, h, (1, 2, 4, 8, 16), (0, 2, 3, 4, 6), ("0"), ("0"), (0, 1))
result = product(*this_one)
The values yielded from w are basically 0, 1, 1, 2, 2, 2, 3, 3, 3, 3, ... and the values yielded from h are 0, 0, 1, 0, 1, 2, 0, 1, 2, 3, .... Plus, the code pretty much looks like your description of the desired behaviour, which is generally a good thing.
Both w and h are generators, no space is wasted and the solution is fairly optimal. I just can't tell if result is what you actually need, but that depends on what you need and expect product() to do here.

Most Pythonic way to iterate by multiples

I was reviewing some algorithms and they had a for loop that increases by a constant multiple. What would be the most Pythonic way of solving this?
This is not an issue of how to solve the problem, but more of a discussion on what the best solution would be?
This is the Java snip:
for (int i = 1; i <=n; i *= c) {
// some stuff
}
Here is an actual solution in python. I don't think it is the most Pythonic method:
i = 1
while i < limit:
# some stuff, remember to use i - 1 as array index
i *= constant
Pythonic way I could see (That does not exist):
for i in mrange(1, limit, c):
# some stuff
First post here. Hope I tagged and all correctly...
You still can do this :
def mrange(start, stop, step):
i = start
while i < stop:
yield i
i *= step
And then :
for i in mrange(1, 100, 4):
print(i)
Prints :
1
4
16
64
Python cannot provide default range functions to fit every needs, but it is pythonic to create your own generators.
If you don't like this solution, the while alternative looks ok too.
You can use itertools.accumulate; start with a range 1 to n, then apply a function which multiplies its first argument by your constant and ignores its second argument.
>>> from itertools import accumulate
>>> [x for x in accumulate(range(1,10), lambda x,_: 4*x)]
[1, 4, 16, 64, 256, 1024, 4096, 16384, 65536]
Having missed that you want to take the values less than n, start with the infinite sequence [c, c**2, c**3, c**4, ...] and use takewhile to "filter" it. (Also, I just realized you only need map, not accumulate, although accumulate may be more efficient. Note the difference in starting points when using map vs accumulate, too.):
>>> from itertools import count, takewhile
>>> n = 100
>>> [x for x in takewhile(lambda x: x < n, map(lambda x: 4**x, count(0)))]
[1, 4, 16, 64]
>>> [x for x in takewhile(lambda x: x < n, accumulate(count(1), lambda x,_: x*4))]
[1, 4, 16, 64]
Using math module:-
for i in range(math.ceil(math.log(limit, const))):
# code goes here
Ex:-
>>> for i in range(math.ceil(math.log(20, 2))):
... print("runs")
...
runs
runs
runs
runs
runs
which is similar to:-
i = 1
while i< 20:
print('runs')
i*=2
Finally; in easy seeming way:-
>>> import math
>>> mrange = lambda i, limit, const: range(i, math.ceil(math.log(limit, const)))
>>> for i in mrange(0, 20, 2):
print('whoa')
..
whoa
whoa
whoa
whoa
whoa

Can you for loop completely through a range, but starting from the nth element?

I would like to know if there exists a base solution to do something like this:
for n in range(length=8, start_position= 3, direction= forward)
The problem I'm encountering is I would like the loop to continue past the final index, and pick up again at idx =0, then idx=1, etc. and stop at idx= 3, the start_position.
To give context, I seek all possible complete solutions to the n-queen problem.
Based on your latest edit, you need a "normal" range and the modulo operator:
for i in range(START, START + LEN):
do_something_with(i % LEN)
from itertools import chain
for n in chain(range(3,8), range(3)):
...
The chain() returns an iterator with 3, 4, ..., 7, 0, 1, 2
Another option for solving this is to use modular arithmetic. You could do something like this, for example:
for i in range(8)
idx = (i + 3) % 8
# use idx
This easily can be generalized to work with different lengths and offsets.
def loop_around_range(length, start_position, direction='forward'):
looped_range = [k % length for k in range(start_position, start_position+length)]
if direction == 'forward':
return looped_range
else:
return looped_range[::-1]
You could implement this for an arbitrary iterable by using itertools.cycle.
from itertools import cycle
def circular_iterator(iterable, skip=0, length=None, reverse=False):
"""Produces a full cycle of #iterable#, skipping the first #skip# elements
then tacking them on to the end.
if #iterable# does not implement #__len__#, you must provide #length#
"""
if reverse:
iterable = reversed(iterable)
cyc_iter = cycle(iterable)
for _ in range(skip):
next(cyc_iter, None)
if length:
total_length = length
else:
total_length = len(iterable)
for _ in range(total_length):
yield next(cyc_iter, None)
>>> lst = [x for x in range(1, 9)]
# [1, 2, 3, 4, 5, 6, 7, 8]
>>> list(circular_iterator(lst, skip=3))
[4, 5, 6, 7, 8, 1, 2, 3]

Get all combinations from list values

I'm using python 2.7 and I have this list:
new_out_filename = ['OFF_B8', 0, 'ON_B8', 1, 'ON_B16', 4, 'OFF_B0', 7]
I want to get all the combinations of the strings like OFF_B8_vs_ON_B8, OFF_B8_vs_ON_B16, OFF_B8_vs_OFf_B0, ON_B8_vs_ON_16, etc.
Is there an easy way to achieve it?
I tried something like:
for k in range(0, len(new_out_filename), 2):
combination = new_out_filename[k]+'_vs_'+new_out_filename[k+2]
print combination
But my list is out of index and also I don't get the appropriate result.
Can you help me please?
just use combinations on a sliced list to ignore the numbers:
import itertools
new_out_filename = ['OFF_B8', 0, 'ON_B8', 1, 'ON_B16', 4, 'OFF_B0', 7]
for a,b in itertools.combinations(new_out_filename[::2],2):
print("{}_vs_{}".format(a,b))
result:
OFF_B8_vs_ON_B8
OFF_B8_vs_ON_B16
OFF_B8_vs_OFF_B0
ON_B8_vs_ON_B16
ON_B8_vs_OFF_B0
ON_B16_vs_OFF_B0
or with comprehension:
result = ["{}_vs_{}".format(*c) for c in itertools.combinations(new_out_filename[::2],2)]
result:
['OFF_B8_vs_ON_B8', 'OFF_B8_vs_ON_B16', 'OFF_B8_vs_OFF_B0', 'ON_B8_vs_ON_B16', 'ON_B8_vs_OFF_B0', 'ON_B16_vs_OFF_B0']
I just added extra for loop and it is working.
new_out_filename = ['OFF_B8', 0, 'ON_B8', 1, 'ON_B16', 4, 'OFF_B0', 7]
for k in range(0, len(new_out_filename), 2):
sd = new_out_filename[k+2:] #it will slice the element of new_out_filename from start in the multiple of 2
for j in range(0, len(sd), 2):
combination = new_out_filename[k]+'_vs_'+sd[j]
print (combination)
output:
OFF_B8_vs_ON_B8
OFF_B8_vs_ON_B16
OFF_B8_vs_OFF_B0
ON_B8_vs_ON_B16
ON_B8_vs_OFF_B0
ON_B16_vs_OFF_B0

Generating list of iterations on own output in Python

Sorry for what seems like a basic question, but I could not find it anywhere. In Python 2, I would like to apply a 1-variable function to its own output storing the list of all steps, i.e. if f(x) returns x*x then iterating from 2, i need to get
[2, 4, 16, 256, 65536, ...]
Ideally, I would need to pass in my function f, the first input 1, and the number of iterations I would like to keep.
I guess this is, in some sense, the opposite of reduce and somewhat similar to unfold from functional programming.
A naive way to do this is to write
out = [2]
for x in xrange(5):
out.append(f(out[-1]))
What is a good Pythonic way to do this?
Thank you very much.
What you need is a "Generator". For example,
def f(x, n):
for _ in range(n):
yield x
x = x * x
l = list(f(2, 5))
print(l) # [2, 4, 16, 256, 65536]
Or
def f(x):
while True:
yield x
x = x * x
for v in f(2):
if v > 100000:
break
print(v), # 2 4 16 256 65536
Ideally, I would need to pass in my function f, the first input 1, and
the number of iterations I would like to keep.
Here is an unfold function that accepts a function, a starting value, and an iteration count.
def unfold(function, start, iterations):
results = []
for _ in range(iterations):
results.append(start)
start = function(start)
return results
Which you can use as expected:
>>> print unfold(lambda x: x*x, 2, 5)
[2, 4, 16, 256, 65536]

Categories