Related
I'm trying to iterate a bunch of data ranges at once, to get the combinations of all their values.
The number of ranges can differ, but I have them collected in a list.
Is there a way to iterate them using list comprehension or a similar clean, pythonic way?
This is what I mean by iterating together:
[print(i, j) for i in r1 for j in r2]
So that's a simple example with two known ranges, but what I need is more like
[print(i, j, ...) for i in r1 for j in r2 for k in r3...]
Note: i don't just need a list of number combinations, the iterators are my own iterator class which works similarly to range() but allows me to also get current state without calling next(), which would alter the state.
My iterator class sets its value back to the start on StopIteration, so it can be looped through more than once.
Here you can see the class:
#dataclass
class Range:
start: float
end: float
step: float = field(default=1)
includeEnd: bool = field(default=True)
def __post_init__(self):
self.value = self.start
def __next__(self):
v = self.value
end = v > self.end if self.includeEnd else v >= self.end
if not end:
self.value += self.step
return v
else:
self.value = self.start
raise StopIteration
def __iter__(self):
return self
But how would you get the product of n iterators using itertools.product(), when you have a list of n iterators?
itertools.product(*the_list). Nothing special about product() there. The leading * is general Python syntax for treating a list (more generally, an iterable) as a sequence of individual arguments.
>>> from itertools import product
>>> args = [range(2), range(3), (i**2 for i in [5, 9])]
>>> args
[range(0, 2), range(0, 3), <generator object <genexpr> at 0x000001E2E7A710B0>]
>>> for x in product(*args):
... print(x)
(0, 0, 25)
(0, 0, 81)
(0, 1, 25)
(0, 1, 81)
(0, 2, 25)
(0, 2, 81)
(1, 0, 25)
(1, 0, 81)
(1, 1, 25)
(1, 1, 81)
(1, 2, 25)
(1, 2, 81)
I'm taking an AI / ML online course, and some of the assignments include Python language phrases like:
input_img = tf.keras.Input(shape=input_shape)
Z1 = tfl.Conv2D(8, kernel_size=4,strides=(1,1), padding='SAME')(input_img)
It seems that the general syntax of the second line is:
obj1 = <class method>(<method parameters>)(obj2),
where obj1, obj2 are some class instances.
Could not find an explanation to this syntax.
Please direct me to a reference / example clarifying said syntax.
The syntax is that tfl.Conv2D(8, kernel_size=4,strides=(1,1), padding='SAME') returns something that is callable, like a callable object OR a method, then you call it with input_img params
A simple example
class A:
def __init__(self, *args):
self.values = list(args)
def run(self, *more_values):
print(f"A.run with {self.values} - {more_values}")
class B:
#staticmethod
def get_method(a, b, c):
return A(a, b, c).run
In the case B.get_method() returns a run method that is not called, so when retrieving it, you can call it, and split in 2 steps to understand better
B.get_method(1, 2, 3)(4, 5) # A.run with [1, 2, 3] - (4, 5)
my_method = B.get_method(1, 2, 3)
my_method(4, 5) # A.run with [1, 2, 3] - (4, 5)
I'm making a container class in Python which will either inherit from list or just implement all the standard list methods (I don't really care which).
How would I create a method that will act on only the items that are returned from a slice? I've been able to make a method that will act on the entire container (see below), but I can't seem to figure out how to act on just the slice.
I'm using python 2.7.6 and use from __future__ import print_function, division in all my code.
Example code:
from __future__ import print_function, division
import itertools
class MyContainerClass(list):
""" For now, I'm inheriting from list. Is this my problem? """
def __init__(self, data_array):
list.__init__(self, data_array)
def __getitem__(self, *args):
arg = args[0]
# specific indices MyContainerClass[0, 3, 5] => indexes 0, 3, and 5
if isinstance(arg, (list, tuple)) and not isinstance(arg[0], bool):
return [list.__getitem__(self, _i) for _i in arg]
# standard slice notation
elif isinstance(arg, slice):
return list.__getitem__(self, arg)
# Using an array mask MyContainerClass[[True, False, False,...,True]] => index 0 and -1]
elif isinstance(arg[0], bool): # or isinstance(arg, np.ndarray):
return list(itertools.compress(self, arg))
else:
# I'll eventually replace this with and exception raise.
return 'error'
def my_method(self):
"""
Will act on entire list, but I want it to act on only what's
returned by the slice (which may be the entire list in some cases).
"""
return "a, ".join([str(_i) for _i in self])
Here is an example of the usage that I'd like:
>>> data = MyContainerClass([1, 2, 3, 4, 5, 6, 7])
>>> data[5:]
[6, 7]
>>> data.my_method() # This works as expected
"1a, 2a, 3a, 4a, 5a, 6a, 7"
>>> data[0:3].my_method() # Doesn't work
"1a, 2a, 3" # but should return this
Looks like things are working now. Thanks a bunch guys! Here's what I've come up with:
from __future__ import print_function, division
import itertools
class MyContainerClass(list):
"""
"""
def __init__(self, array):
if isinstance(array, int):
list.__init__(self, [array])
else:
list.__init__(self, array)
def __getitem__(self, arg):
# Standard Slice notation
if isinstance(arg, slice):
retval = super(MyContainerClass, self).__getitem__(arg)
# specific indices
elif isinstance(arg, (list, tuple)) and not isinstance(arg[0], bool):
retval = [list.__getitem__(self, _i) for _i in arg]
# a single specific index
elif isinstance(arg, int):
retval = list.__getitem__(self, arg)
# an array mask of T/F values
elif isinstance(arg[0], bool): # or isinstance(arg, np.ndarray):
retval = list(itertools.compress(self, arg))
# raise an error on unknown
else:
raise SyntaxError("Unknown notation for list slice or index")
retval = type(self)(retval)
return retval
def __getslice__(self, i, j):
# Python 2 built-in types only
return self.__getitem__(slice(i, j))
def my_method(self):
return "a, ".join([str(_i) for _i in self])
And acts like:
>>> data = MyContainerClass([1, 2, 3, 4, 5, 6, 7])
>>> mask = [True, True, False, True, False, False, False]
>>> print(data)
[1, 2, 3, 4, 5, 6, 7]
>>> print(type(data[5:]))
<class '__main__.MyContainerClass'>
>>> print(data.my_method())
1a, 2a, 3a, 4a, 5a, 6a, 7
>>> print(data[0:5].my_method())
1a, 2a, 3a, 4a, 5
>>> print(data[1, 5, 2].my_method())
2a, 6a, 3
>>> print(data[mask].my_method())
1a, 2a, 4
>>> print(data[2].my_method())
3
You'll have to make sure your __getitem__ returns your type again:
class MyContainerClass(list):
""" For now, I'm inheriting from list. Is this my problem? """
def __init__(self, data_array):
list.__init__(self, data_array)
def my_method(self):
"""
Will act on entire list, but I want it to act on only what's
returned by the slice (which may be the entire list in some cases).
"""
return "a, ".join([str(_i) for _i in self])
def __getitem__(self, index):
retval = super(MyContainerClass, self).__getitem__(index)
if isinstance(index, slice):
retval = type(self)(retval)
return retval
def __getslice__(self, i, j):
# Python 2 built-in types only
return self.__getitem__(slice(i, j))
The additional __getslice__ method is only needed in Python 2, and then only if you are inheriting from a type that already implements __getslice__. list is such a type.
Demo:
>>> data = MyContainerClass([1, 2, 3, 4, 5, 6, 7])
>>> data[:5]
[1, 2, 3, 4, 5]
>>> type(data[:5])
<class '__main__.MyContainerClass'>
>>> data.my_method()
'1a, 2a, 3a, 4a, 5a, 6a, 7'
>>> data[:3].my_method()
'1a, 2a, 3'
I'm currently trying to code an equivalent for the built-in min-max function in python, and my code return a pretty weird exception which I don't understand at all:
TypeError: 'generator' object is not subscriptable, min, 7, , 9
when i try it with:
min(abs(i) for i in range(-10, 10))
Here is my code:
def min(*args, **kwargs):
key = kwargs.get("key", None)
argv=0
for i in args:
argv+=1
if argv == 1 and (type(args) is list or type(args) is tuple or type(args) is str):
min=args[0][0]
for i in args[0]:
if key != None:
if key(i) < key(min):
min = i
else:
if i < min:
min = i
return min
else:
min=args[0]
for i in args:
if key != None:
if key(i) < key(min):
min = i
else:
if i < min:
min = i
return min
According to the documentation, i should be able to iterate over a generator...
Here is my implementation:
def max(*args, **kwargs):
key = kwargs.get("key", lambda x: x)
if len(args) == 1:
args = args[0]
maxi = None
for i in args:
if maxi == None or key(i) > key(maxi):
maxi = i
return maxi
def min(*args, **kwargs):
key = kwargs.get("key", lambda x: x)
if len(args) == 1:
args = args[0]
mini = None
for i in args:
if mini == None or key(i) < key(mini):
mini = i
return mini
A little bit more concise than preview post.
The issue you are having is due to the fact that min has two function signatures. From its docstring:
min(...)
min(iterable[, key=func]) -> value
min(a, b, c, ...[, key=func]) -> value
So, it will accept either a single positional argument (an iterable, who's values you need to compare) or several positional arguments which are the values themselves. I think you need to test which mode you're in at the start of your function. It is pretty easy to turn the one argument version into the multiple argument version simply by doing args = args[0].
Here's my attempt to implement the function. key is a keyword-only argument, since it appears after *args.
def min(*args, key=None): # args is a tuple of the positional arguments initially
if len(args) == 1: # if there's just one, assume it's an iterable of values
args = args[0] # replace args with the iterable
it = iter(args) # get an iterator
try:
min_val = next(it) # take the first value from the iterator
except StopIteration:
raise ValueError("min() called with no values")
if key is None: # separate loops for key=None and otherwise, for efficiency
for val in it: # loop on the iterator, which has already yielded one value
if val < min_val
min_val = val
else:
min_keyval = key(min_val) # initialize the minimum keyval
for val in it:
keyval = key(val)
if keyval < min_keyval: # compare keyvals, rather than regular values
min_val = val
min_keyval = keyval
return min_val
Here's some testing:
>>> min([4, 5, 3, 2])
2
>>> min([1, 4, 5, 3, 2])
1
>>> min(4, 5, 3, 2)
2
>>> min(4, 5, 3, 2, 1)
1
>>> min(4, 5, 3, 2, key=lambda x: -x)
5
>>> min(4, -5, 3, -2, key=abs)
-2
>>> min(abs(i) for i in range(-10, 10))
0
Functions in question have a lot in common. In fact, the only difference is comparison (< vs >). In the light of this fact we can implement generic function for finding and element, which will use comparison function passed as an argument. The min and max example might look as follows:
def lessThan(val1, val2):
return val1 < val2
def greaterThan(val1, val2):
return val1 > val2
def find(cmp, *args, **kwargs):
if len(args) < 1:
return None
key = kwargs.get("key", lambda x: x)
arguments = list(args[0]) if len(args) == 1 else args
result = arguments[0]
for val in arguments:
if cmp(key(val), key(result)):
result = val
return result
min = lambda *args, **kwargs: find(lessThan, *args, **kwargs)
max = lambda *args, **kwargs: find(greaterThan, *args, **kwargs)
Some tests:
>>> min(3, 2)
2
>>> max(3, 2)
3
>>> max([1, 2, 0, 3, 4])
4
>>> min("hello")
'e'
>>> max(2.2, 5.6, 5.9, key=int)
5.6
>>> min([[1, 2], [3, 4], [9, 0]], key=lambda x: x[1])
[9, 0]
>>> min((9,))
9
>>> max(range(6))
5
>>> min(abs(i) for i in range(-10, 10))
0
>>> max([1, 2, 3], [5, 6], [7], [0, 0, 0, 1])
[7]
I'm attempting to write a genetic algorithm framework in Python, and am running into issues with shallow/deep copying. My background is mainly C/C++, and I'm struggling to understand how these connections are persisting.
What I am seeing is an explosion in the length of an attribute list within a subclass. My code is below...I'll point out the problems.
This is the class for a single gene. Essentially, it should have a name, value, and boolean flag. Instances of Gene populate a list within my Individual class.
# gene class
class Gene():
# constructor
def __init__(self, name, is_float):
self.name_ = name
self.is_float_ = is_float
self.value_ = self.randomize_gene()
# create a random gene
def randomize_gene(self):
return random.random()
This is my Individual class. Each generation, a population of these are created (I'll show the creation code after the class declaration) and have the typical genetic algorithm operations applied. Of note is the print len(self.Genes_) call, which grows each time this class is instantiated.
# individual class
class Individual():
# genome definition
Genes_ = [] # genes list
evaluated_ = False # prevent re-evaluation
fitness_ = 0.0 # fitness value (from evaluation)
trace_ = "" # path to trace file
generation_ = 0 # generation to which this individual belonged
indiv_ = 0 # identify this individual by number
# constructor
def __init__(self, gen, indv):
# assign indices
self.generation_ = gen
self.indiv_ = indv
self.fitness_ = random.random()
# populate genome
for lp in cfg.params_:
g = Gene(lp[0], lp[1])
self.Genes_.append(g)
print len(self.Genes_)
> python ga.py
> 24
> 48
> 72
> 96
> 120
> 144
......
As you can see, each Individual should have 24 genes, however this population explodes quite rapidly.
I create an initial population of new Individuals like this:
# create a randomized initial population
def createPopulation(self, gen):
loc_population = []
for i in range(0, cfg.population_size_):
indv = Individual(gen, i)
loc_population.append(indv)
return loc_population
and later on my main loop (apologies for the whole dump, but felt it was necessary - if my secondary calls (mutation/crossover) are needed please let me know))
for i in range(0, cfg.generations_):
# evaluate current population
self.evaluate(i)
# sort population on fitness
loc_pop = sorted(self.population_, key=operator.attrgetter('fitness_'), reverse=True)
# create next population & preserve elite individual
next_population = []
elitist = copy.deepcopy(loc_pop[0])
elitist.generation_ = i
next_population.append(elitist)
# perform selection
selection_pool = []
selection_pool = self.selection(elitist)
# perform crossover on selection
new_children = []
new_children = self.crossover(selection_pool, i)
# perform mutation on selection
muties = []
muties = self.mutation(selection_pool, i)
# add members to next population
next_population = next_population + new_children + muties
# fill out the rest with random
for j in xrange(len(next_population)-1, cfg.population_size_ - 1):
next_population.append(Individual(i, j))
# copy next population over old population
self.population_ = copy.deepcopy(next_population)
# clear old lists
selection_pool[:] = []
new_children[:] = []
muties[:] = []
next_population[:] = []
I'm not not completely sure that I understand your question, but I suspect that your problem is that the Genes_ variable in your Individual() class is declared in the class namespace. This namespace is available to all members of the class. In other words, each instance of Individual() will share the same variable Genes_.
Consider the following two snippets:
class Individual():
# genome definition
genes = []
def __init__(self):
for i in xrange(10):
self.genes.append(i)
ind_1 = Individual()
print ind_1.genes
ind_2 = Individual()
print ind_1.genes
print ind_2.genes
and
class Individual():
# genome definition
def __init__(self):
self.genes = []
for i in xrange(10):
self.genes.append(i)
ind_1 = Individual()
print ind_1.genes
ind_2 = Individual()
print ind_1.genes
print ind_2.genes
The first snippet outputs
>>> [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
while the second snippet outputs
>>> [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
In the first scenario, when the second Individual() is instantiated the genes list variable already exists, and the genes from the second individual are added to this existing list.
Rather than creating the Individual() class like this,
# individual class
class Individual():
# genome definition
Genes_ = [] # genes list
# constructor
def __init__(self, gen, indv):
# assign indices
self.generation_ = gen
self.indiv_ = indv
self.fitness_ = random.random()
you should consider declaring the Genes_ variable in init so that each Individual() instance gets its own gene set
# individual class
class Individual():
# constructor
def __init__(self, gen, indv):
# genome definition
self.Genes_ = [] # genes list
# assign indices
self.generation_ = gen
self.indiv_ = indv
self.fitness_ = random.random()
When you create a class, you are really creating exactly one 'class object'. These are objects just like any other object in Python; everything in Python is an object, and what those objects do is defined by their methods, not their class! That is the magic of duck typing. In Python you can even create new classes dynamically on the fly.
Anyway, you are adding exactly one list object to the "Genes_" attribute of the one and only "Individuals" class object. The upshot is that every instance object of the "Individual" class object is accessing the same "Genes_" list object.
Consider this
# In 2.2 <= Python < 3.0 you should ALWAYS inherit from 'object'.
class Foobar(object):
doodah = []
a = Foobar()
b = Foobar()
assert id(a.doodah) == id(b.doodah) # True
In this case, as you can see, "a.doodah" and "b.doodah" are the same object!
class Foobar(object):
def __init__(self):
self.doodah = []
a = Foobar()
b = Foobar()
assert id(a.doodah) != id(b.doodah) # True
In this case, they are different objects.
It's possible to have your cake and eat it too. Consider this
class Foobar(object):
doodah = []
a = Foobar()
b = Foobar()
a.doodah = 'hlaghalgh'
assert id(a.doodah) != id(b.doodah) # True
In this case a "doodah" attribute is added to the "a" object, which overrides the class attribute.
Hope this helps!