We Keep Coding

Use metaclass to allow forward declarations

I want to do something decidedly unpythonic. I want to create a class that allows for forward declarations of its class attributes. (If you must know, I am trying to make some sweet syntax for parser combinators.)
This is the kind of thing I am trying to make:
a = 1
class MyClass(MyBaseClass):
b = a # Refers to something outside the class
c = d + b # Here's a forward declaration to 'd'
d = 1 # Declaration resolved
My current direction is to make a metaclass so that when d is not found I catch the NameError exception and return an instance of some dummy class I'll call ForwardDeclaration. I take some inspiration from AutoEnum, which uses metaclass magic to declare enum values with bare identifiers and no assignment.
Below is what I have so far. The missing piece is: how do I continue normal name resolution and catch the NameErrors:
class MetaDict(dict):
def __init__(self):
self._forward_declarations = dict()
def __getitem__(self, key):
try:
### WHAT DO I PUT HERE ??? ###
# How do I continue name resolution to see if the
# name already exists is the scope of the class
except NameError:
if key in self._forward_declarations:
return self._forward_declarations[key]
else:
new_forward_declaration = ForwardDeclaration()
self._forward_declarations[key] = new_forward_declaration
return new_forward_declaration
class MyMeta(type):
def __prepare__(mcs, name, bases):
return MetaDict()
class MyBaseClass(metaclass=MyMeta):
pass
class ForwardDeclaration:
# Minimal behavior
def __init__(self, value=0):
self.value = value
def __add__(self, other):
return ForwardDeclaration(self.value + other)

To start with:
def __getitem__(self, key):
try:
return super().__getitem__(key)
except KeyError:
...
But that won't allow you to retrieve the global variables outside the class body.
You can also use the __missin__ method which is reserved exactly for subclasses of dict:
class MetaDict(dict):
def __init__(self):
self._forward_declarations = dict()
# Just leave __getitem__ as it is on "dict"
def __missing__(self, key):
if key in self._forward_declarations:
return self._forward_declarations[key]
else:
new_forward_declaration = ForwardDeclaration()
self._forward_declarations[key] = new_forward_declaration
return new_forward_declaration
As you can see, that is not that "UnPythonic" - advanced Python stuff such as SymPy and SQLAlchemy have to resort to this kind of behavior to do their nice magic - just be sure to get it very well documented and tested.
Now, to allow for global (module) variables, you have a to get a little out of the way - and possibly somthing that may not be avaliablein all Python implementations - that is: introspecting the frame where the class body is to get its globals:
import sys
...
class MetaDict(dict):
def __init__(self):
self._forward_declarations = dict()
# Just leave __getitem__ as it is on "dict"
def __missing__(self, key):
class_body_globals = sys._getframe().f_back.f_globals
if key in class_body_globals:
return class_body_globals[key]
if key in self._forward_declarations:
return self._forward_declarations[key]
else:
new_forward_declaration = ForwardDeclaration()
self._forward_declarations[key] = new_forward_declaration
return new_forward_declaration
Now that you are here - your special dictionaries are good enough to avoid NameErrors, but your ForwardDeclaration objects are far from smart enough - when running:
a = 1
class MyClass(MyBaseClass):
b = a # Refers to something outside the class
c = d + b # Here's a forward declaration to 'd'
d = 1
What happens is that c becomes a ForwardDeclaration object, but summed to the instant value of d which is zero. On the next line, d is simply overwritten with the value 1 and is no longer a lazy object. So you might just as well declare c = 0 + b .
To overcome this, ForwardDeclaration has to be a class designed in a smartway, so that its values are always lazily evaluated, and it behaves as in the "reactive programing" approach: i.e.: updates to a value will cascade updates into all other values that depend on it. I think giving you a full implementation of a working "reactive" aware FOrwardDeclaration class falls off the scope of this question. - I have some toy code to do that on github at https://github.com/jsbueno/python-react , though.
Even with a proper "Reactive" ForwardDeclaration class, you have to fix your dictionary again so that the d = 1 class works:
class MetaDict(dict):
def __init__(self):
self._forward_declarations = dict()
def __setitem__(self, key, value):
if key in self._forward_declarations:
self._forward_declations[key] = value
# Trigger your reactive update here if your approach is not
# automatic
return None
return super().__setitem__(key, value)
def __missing__(self, key):
# as above
And finally, there is a way to avoid havign to implement a fully reactive aware class - you can resolve all pending FOrwardDependencies on the __new__ method of the metaclass - (so that your ForwardDeclaration objects are manually "frozen" at class creation time, and no further worries - )
Something along:
from functools import reduce
sentinel = object()
class ForwardDeclaration:
# Minimal behavior
def __init__(self, value=sentinel, dependencies=None):
self.dependencies = dependencies or []
self.value = value
def __add__(self, other):
if isinstance(other, ForwardDeclaration):
return ForwardDeclaration(dependencies=self.dependencies + [self])
return ForwardDeclaration(self.value + other)
class MyMeta(type):
def __new__(metacls, name, bases, attrs):
for key, value in list(attrs.items()):
if not isinstance(value, ForwardDeclaration): continue
if any(v.value is sentinel for v in value.dependencies): continue
attrs[key] = reduce(lambda a, b: a + b.value, value.dependencies, 0)
return super().__new__(metacls, name, bases, attrs)
def __prepare__(mcs, name, bases):
return MetaDict()
And, depending on your class hierarchy and what exactly you are doing, rememebr to also update one class' dict _forward_dependencies with the _forward_dependencies created on its ancestors.
AND if you need any operator other than +, as you will have noted, you will have to keep information on the operator itself - at this point, hou might as well jsut use sympy.

how to implement a function like sum(2)(3)(4)......(n) in python?

how to implement a function that will be invoked in the following way sum_numbers(2)(3)(4)......(n) in python?
the result should be 2+3+4+.....+n
The hint that I have is since functions are object in pythons there is way to do those using a nested function but I am not sure.
def sum_number(x):
def sum_number_2(y):
def sum_number_3(z):
....................
def sum_number_n(n)
return n
return sum_number_n
return sum_number_3
return sum_number_2
return sum_number
But instead of writing so many nested functions we should be able to do it in couple nested functions to compute sum of n values when invoked in the following way sum_numbers(2)(3)(4)......(n)

Use Python's data model features to convert the result into the desired type.
class sum_number(object):
def __init__(self, val):
self.val = val
def __call__(self, val):
self.val += val
return self
def __float__(self):
return float(self.val)
def __int__(self):
return int(self.val)
print '{}'.format(int(sum_number(2)(3)(8)))
print '{}'.format(float(sum_number(2)(3)(8)))

You could create a subclass of int that is callable:
class sum_numbers (int):
def __new__ (cls, *args, **kwargs):
return super().__new__(cls, *args, **kwargs)
def __call__ (self, val):
return sum_numbers(self + val)
That way, you have full compatibility with a normal integer (since objects of that type are normal integers), so the following examples work:
>>> sum_numbers(2)(3)(4)(5)
14
>>> isinstance(sum_numbers(2)(3), int)
True
>>> sum_numbers(2)(3) + 4
9
Of course, you may want to override additional methods, e.g. __add__ so that adding a normal integer will still return an object of your type. Otherwise, you would have to call the type with the result, e.g.:
>>> sum_numbers(sum_numbers(2)(3) + 5)(6)
16

If your function is returning another function, you can't just chain calls together and expect a human readable result. If you want a function that does what you want without the final result, this works:
def sum(x):
def f(y):
return sum(x+y)
return f
If you're fine with printing out the operations you can try this:
def sum(x):
print(x)
def f(y):
return sum(x+y)
return f
If you absolutely, absolutely need a return value then this is a dirty, horrible hack you could try:
def sum(x, v):
v[0] = x
def f(y, v):
return sum(x+y, v)
return f
v = [0]
sum(1,v)(2,v)(3,v)
print(v[0]) # Should return 6
Here's another solution that uses classes:
class sum(object):
def __init__(self, x=0):
self.x=x
def __call__(self, *y):
if len(y) > 0:
self.x += y[0]
return self
return self.x
print(sum(1)(2)(3)()) # Prints 6

What you're asking for is not possible in Python since you aren't providing a way to determine the end of the call chain, as cricket_007 mentions in the comments. However, if you do provide a way to indicate that there are no more calls then the function is easy to code. One way to indicate the end of the chain is to make the last call with no arguments.
I'm using rsum (recursive sum) as the name of the function in my code because sum is a built-in function and unnecessarily shadowing the Python built-ins is not a good coding practice: it makes the code potentially confusing, or at least harder to read because you have to keep remembering that the name isn't referring to what you normally expect it to refer to, and can lead to subtle bugs.
def rsum(val=None, tot=0):
if val is None:
return tot
tot += val
return lambda val=None, tot=tot: rsum(val, tot)
print rsum(42)()
print rsum(1)(2)()
print rsum(4)(3)(2)(1)()
print rsum(4100)(310000)(9)(50)()
output
42
3
10
314159

class MetaSum(type):
def __repr__(cls):
sum_str = str(cls.sum)
cls.sum = 0
return sum_str
def __call__(cls, *args):
for arg in args:
cls.sum += arg
return cls
class sum_numbers(object, metaclass = MetaSum):
sum = 0
print (sum_numbers(2)(3)(4)) # this only works in python 3

Comparing ndarray derived classes

Following the instructions here I created a subclass of ndarray that adds new attributes to the ndarray class. Now I want to define a comparison operator for the new class that besides comparing the data, also compares the values of the attributes. So I tried this:
def __eq__(self, other):
return (self._prop1 == other._prop1) and \
(self._prop2 == other._prop2) and \
(self.data == other.data)
This allows for comparison like T1 == T2 and returns a boolean value. However since I would like to use these arrays interchangeably with other ndarrays I would like the comparison to return a boolean array. If I don't define my __eq__ function then the comparison returns a boolean array, but then I can't check for the attributes. How can I combine the two?

As per the suggestion by hpaulj I figured out how to do this by looking at np.ma.core.MaskedArray.__eq__. Here's the minimum implementation for reference. The main idea is to call the numpy __eq__() on a view of self in the type of the base class of DerivedArray.
class DerivedArray(np.ndarray):
def __new__(cls, input_array, prop1, prop2):
_baseclass = getattr(input_array, '_baseclass', type(input_array))
obj = np.asarray(input_array).view(cls)
obj._prop1 = prop1
obj._prop2 = prop2
obj._baseclass = _baseclass
return obj
def __array_finalize__(self, obj):
if obj is None:
return
else:
if not isinstance(obj, np.ndarray):
_baseclass = type(obj)
else:
_baseclass = np.ndarray
self._prop1 = getattr(obj, '_prop1', None)
self._prop2 = getattr(obj, '_prop2', None)
self._baseclass= getattr(obj, '_baseclass', _baseclass)
def _get_data(self):
"""Return the current data, as a view of the original
underlying data.
"""
return np.ndarray.view(self, self._baseclass)
_data = property(fget=_get_data)
data = property(fget=_get_data)
def __eq__(self, other):
attsame = (self._prop1 == other._prop1) and (self._prop2 == other._prop2)
if not attsame: return False
return self._data.__eq__(other)

reload subclass of frozenset

I have a class which I want to have all the functions of frozenset but i don't want him to be configurable (by init, frozenset gets iterable).
Additionally, I want him to have the function 'reload' - I am loading static list from a server so the user can not change it (so I don't want the user to think he can change it).
The list on the server can be changed by the admin so I need the reload option.
That's what I hoped for:
class A(frozenset):
def __init__(self, list_id):
super().__init__()
self.list_id = list_id
self.reload()
def reload(self):
#loading staff by self.list_id...
pass
But I didn't find a way to 'add' new staff to the class (I tried to re-init it).
May be I am using the wrong staff so if you have anther way for this it fine (I need the option to compare difference between to difference objects):
a = A(1)
b = A(2)
len(a)
iter(a)
a.difference(b)
May be overloading add and update of set will be good but i don't want to do that (it looks bad in code because there are more update-like functions).

You cannot update the frozenset contents, no; it remains immutable even when subclassed.
You can subclass the collections.abc.Set() Abstract Base Class instead; it models an immutable set too; all you need to do really is implement the methods listed in the Abstract Methods column and the rest is taken care of for you:
from collections.abc import Set
class A(Set):
def __init__(self, list_id):
self.list_id = list_id
self.reload()
def reload(self):
values = get_values(self.list_id)
self._values = frozenset(values)
def __contains__(self, item):
return item in self._values
def __iter__(self):
return iter(self._values)
def __len__(self):
return len(self._values)
Not all methods of the built-in frozenset type are implemented; you can easily supply the missing ones as these are aliases of the operator methods:
def issubset(self, other):
return self <= frozenset(other)
def issuperset(self, other):
return self >= frozenset(other)
def union(self, *others):
res = self
for o in others:
res |= frozenset(o)
return res
def intersection(self, *others):
res = self
for o in others:
res &= frozenset(o)
return res
def difference(self, *others):
res = self
for o in others:
res -= frozenset(o)
return res
def symmetric_difference(self, other):
return self ^ frozenset(other)

python lazy variables? or, delayed expensive computation

I have a set of arrays that are very large and expensive to compute, and not all will necessarily be needed by my code on any given run. I would like to make their declaration optional, but ideally without having to rewrite my whole code.
Example of how it is now:
x = function_that_generates_huge_array_slowly(0)
y = function_that_generates_huge_array_slowly(1)
Example of what I'd like to do:
x = lambda: function_that_generates_huge_array_slowly(0)
y = lambda: function_that_generates_huge_array_slowly(1)
z = x * 5 # this doesn't work because lambda is a function
# is there something that would make this line behave like
# z = x() * 5?
g = x * 6
While using lambda as above achieves one of the desired effects - computation of the array is delayed until it is needed - if you use the variable "x" more than once, it has to be computed each time. I'd like to compute it only once.
EDIT:
After some additional searching, it looks like it is possible to do what I want (approximately) with "lazy" attributes in a class (e.g. http://code.activestate.com/recipes/131495-lazy-attributes/). I don't suppose there's any way to do something similar without making a separate class?
EDIT2: I'm trying to implement some of the solutions, but I'm running in to an issue because I don't understand the difference between:
class sample(object):
def __init__(self):
class one(object):
def __get__(self, obj, type=None):
print "computing ..."
obj.one = 1
return 1
self.one = one()
and
class sample(object):
class one(object):
def __get__(self, obj, type=None):
print "computing ... "
obj.one = 1
return 1
one = one()
I think some variation on these is what I'm looking for, since the expensive variables are intended to be part of a class.

The first half of your problem (reusing the value) is easily solved:
class LazyWrapper(object):
def __init__(self, func):
self.func = func
self.value = None
def __call__(self):
if self.value is None:
self.value = self.func()
return self.value
lazy_wrapper = LazyWrapper(lambda: function_that_generates_huge_array_slowly(0))
But you still have to use it as lazy_wrapper() not lazy_wrapper.
If you're going to be accessing some of the variables many times, it may be faster to use:
class LazyWrapper(object):
def __init__(self, func):
self.func = func
def __call__(self):
try:
return self.value
except AttributeError:
self.value = self.func()
return self.value
Which will make the first call slower and subsequent uses faster.
Edit: I see you found a similar solution that requires you to use attributes on a class. Either way requires you rewrite every lazy variable access, so just pick whichever you like.
Edit 2: You can also do:
class YourClass(object)
def __init__(self, func):
self.func = func
#property
def x(self):
try:
return self.value
except AttributeError:
self.value = self.func()
return self.value
If you want to access x as an instance attribute. No additional class is needed. If you don't want to change the class signature (by making it require func), you can hard code the function call into the property.

Writing a class is more robust, but optimizing for simplicity (which I think you are asking for), I came up with the following solution:
cache = {}
def expensive_calc(factor):
print 'calculating...'
return [1, 2, 3] * factor
def lookup(name):
return ( cache[name] if name in cache
else cache.setdefault(name, expensive_calc(2)) )
print 'run one'
print lookup('x') * 2
print 'run two'
print lookup('x') * 2

Python 3.2 and greater implement an LRU algorithm in the functools module to handle simple cases of caching/memoization:
import functools
#functools.lru_cache(maxsize=128) #cache at most 128 items
def f(x):
print("I'm being called with %r" % x)
return x + 1
z = f(9) + f(9)**2

You can't make a simple name, like x, to really evaluate lazily. A name is just an entry in a hash table (e.g. in that which locals() or globals() return). Unless you patch access methods of these system tables, you cannot attach execution of your code to simple name resolution.
But you can wrap functions in caching wrappers in different ways.
This is an OO way:
class CachedSlowCalculation(object):
cache = {} # our results
def __init__(self, func):
self.func = func
def __call__(self, param):
already_known = self.cache.get(param, None)
if already_known:
return already_known
value = self.func(param)
self.cache[param] = value
return value
calc = CachedSlowCalculation(function_that_generates_huge_array_slowly)
z = calc(1) + calc(1)**2 # only calculates things once
This is a classless way:
def cached(func):
func.__cache = {} # we can attach attrs to objects, functions are objects
def wrapped(param):
cache = func.__cache
already_known = cache.get(param, None)
if already_known:
return already_known
value = func(param)
cache[param] = value
return value
return wrapped
#cached
def f(x):
print "I'm being called with %r" % x
return x + 1
z = f(9) + f(9)**2 # see f called only once
In real world you'll add some logic to keep the cache to a reasonable size, possibly using a LRU algorithm.

To me, it seems that the proper solution for your problem is subclassing a dict and using it.
class LazyDict(dict):
def __init__(self, lazy_variables):
self.lazy_vars = lazy_variables
def __getitem__(self, key):
if key not in self and key in self.lazy_vars:
self[key] = self.lazy_vars[key]()
return super().__getitem__(key)
def generate_a():
print("generate var a lazily..")
return "<a_large_array>"
# You can add as many variables as you want here
lazy_vars = {'a': generate_a}
lazy = LazyDict(lazy_vars)
# retrieve the variable you need from `lazy`
a = lazy['a']
print("Got a:", a)
And you can actually evaluate a variable lazily if you use exec to run your code. The solution is just using a custom globals.
your_code = "print('inside exec');print(a)"
exec(your_code, lazy)
If you did your_code = open(your_file).read(), you could actually run your code and achieve what you want. But I think the more practical approach would be the former one.