We Keep Coding

Why do function attributes (setattr ones) only become available after assigning it as a property to a class and instantiating it?

I apologize if I'm butchering the terminology. I'm trying to understand the code in this example on how to chain a custom function onto a PySpark dataframe. I'd really want to understand exactly what it's doing, and if it is not awful practice before I implement anything.
From the way I'm understanding the code, it:
defines a function g with sub-functions inside of it, that returns a copy of itself
assigns the sub-functions to g as attributes
assigns g as a property of the DataFrame class
I don't think at any step in the process do any of them become a method (when I do getattr, it always says "function")
When I run a (as best as I can do) simplified version of the code (below), it seems like only when I assign the function as a property to a class, and then instantiate at least one copy of the class, do the attributes on the function become available (even outside of the class). I want to understand what and why that is happening.
An answer [here(https://stackoverflow.com/a/17007966/19871699) indicates that this is a behavior, but doesn't really explain what/why it is. I've read this too but I'm having trouble seeing the connection to the code above.
I read here about the setattr part of the code. He doesn't mention exactly the use case above. this post has some use cases where people do it, but I'm not understanding how it directly applies to the above, unless I've missed something.
The confusing part is when the inner attributes become available.
class SampleClass():
def __init__(self):
pass
def my_custom_attribute(self):
def inner_function_one():
pass
setattr(my_custom_attribute,"inner_function",inner_function_one)
return my_custom_attribute
[x for x in dir(my_custom_attribute) if x[0] != "_"]
returns []
then when I do:
SampleClass.custom_attribute = property(my_custom_attribute)
[x for x in dir(my_custom_attribute) if x[0] != "_"]
it returns []
but when I do:
class_instance = SampleClass()
class_instance.custom_attribute
[x for x in dir(my_custom_attribute) if x[0] != "_"]
it returns ['inner_function']
In the code above though, if I do SampleClass.custom_attribute = my_custom_attribute instead of =property(...) the [x for x... code still returns [].
edit: I'm not intending to access the function itself outside of the class. I just don't understand the behavior, and don't like implementing something I don't understand.

So, setattr is not relevant here. This would all work exactly the same without it, say, by just doing my_custom_attribute.inner_function = inner_function_one etc. What is relevant is that the approach in the link you showed (which your example doesn't exactly make clear what the purpose is) relies on using a property, which is a descriptor. But the function won't get called unless you access the attribute corresponding to the property on an instance. This comes down to how property works. For any property, given a class Foo:
Foo.attribute_name = property(some_function)
Then some_function won't get called until you do Foo().attribute_name. That is the whole point of property.
But this whole solution is very confusingly engineered. It relies on the above behavior, and it sets attributes on the function object.
Note, if all you want to do is add some method to your DataFrame class, you don't need any of this. Consider the following example (using pandas for simplicity):
>>> import pandas as pd
>>> def foobar(self):
... print("in foobar with instance", self)
...
>>> pd.DataFrame.baz = foobar
>>> df = pd.DataFrame(dict(x=[1,2,3], y=['a','b','c']))
>>> df
x y
0 1 a
1 2 b
2 3 c
>>> df.baz()
in foobar with instance x y
0 1 a
1 2 b
2 3 c
That's it. You don't need all that rigamarole. Of course, if you wanted to add a nested accessor, df.custom.whatever, you would need something a bit more complicated. You could use the approach in the OP, but I would prefer something more explicit:
import pandas as pd
class AccessorDelegator:
def __init__(self, accessor_type):
self.accessor_type = accessor_type
def __get__(self, instance, cls=None):
return self.accessor_type(instance)
class CustomMethods:
def __init__(self, instance):
self.instance = instance
def foo(self):
# do something with self.instance as if this were your `self` on the dataframe being augmented
print(self.instance.value_counts())
pd.DataFrame.custom = AccessorDelegator(CustomMethods)
df = pd.DataFrame(dict(a=[1,2,3], b=['a','b','c']))
df.foo()
The above will print:
a b
1 a 1
2 b 1
3 c 1

Because when you call a function the attributes within that function aren't returned only the returned value is passed back.
In other words the additional attributes are only available on the returned function and not with 'g' itself.
Try moving setattr() outside of the function.

class initialization in Python

I found that some classes contain a __init__ function, and some don’t. I’m confused about something described below.
What is the difference between these two pieces of code:
class Test1(object):
i = 1
and
class Test2(object):
def __init__(self):
self.i = 1
I know that the result or any instance created by these two class and the way of getting their instance variable are pretty much the same. But is there any kind of “default” or “hidden” initialization mechanism of Python behind the scene when we don’t define the __init__ function for a class? And why I can’t write the first code in this way:
class Test1(object):
self.i = 1
That’s my questions. Thank you very much!
Thank you very much Antti Haapala! Your answer gives me further understanding of my questions. Now, I understand that they are different in a way that one is a "class variable", and the other is a "instance variable". But, as I tried it further, I got yet another confusing problem.
Here is what it is. I created 2 new classes for understanding what you said:
class Test3(object):
class_variable = [1]
def __init__(self):
self.instance_variable = [2]
class Test4(object):
class_variable = 1
def __init__(self):
self.instance_variable = 2
As you said in the answer to my first questions, I understand the class_variable is a "class variable" general to the class, and should be passed or changed by reference to the same location in the memory. And the instance_variable would be created distinctly for different instances.
But as I tried out, what you said is true for the Test3's instances, they all share the same memory. If I change it in one instance, its value changes wherever I call it.
But that's not true for instances of Test4. Shouldn't the int in the Test4 class also be changed by reference?
i1 = Test3()
i2 = Test3()
>>> i1.i.append(2)
>>> i2.i
[1, 2]
j1 = Test4()
j2 = Test4()
>>> j1.i = 3
>>> j2.i
1
Why is that? Does that "=" create an "instance variable" named "i" without changing the original "Test4.i" by default? Yet the "append" method just handles the "class variable"?
Again, thank you for your exhaustive explanation of the most boring basic concepts to a newbie of Python. I really appreciate that!

In python the instance attributes (such as self.i) are stored in the instance dictionary (i.__dict__). All the variable declarations in the class body are stored as attributes of the class.
Thus
class Test(object):
i = 1
is equivalent to
class Test(object):
pass
Test.i = 1
If no __init__ method is defined, the newly created instance usually starts with an empty instance dictionary, meaning that none of the properties are defined.
Now, when Python does the get attribute (as in print(instance.i) operation, it first looks for the attribute named i that is set on the instance). If that fails, the i attribute is looked up on type(i) instead (that is, the class attribute i).
So you can do things like:
class Test:
i = 1
t = Test()
print(t.i) # prints 1
t.i += 1
print(t.i) # prints 2
but what this actually does is:
>>> class Test(object):
... i = 1
...
>>> t = Test()
>>> t.__dict__
{}
>>> t.i += 1
>>> t.__dict__
{'i': 2}
There is no i attribute on the newly created t at all! Thus in t.i += 1 the .i was looked up in the Test class for reading, but the new value was set into the t.
If you use __init__:
>>> class Test2(object):
... def __init__(self):
... self.i = 1
...
>>> t2 = Test2()
>>> t2.__dict__
{'i': 1}
The newly created instance t2 will already have the attribute set.
Now in the case of immutable value such as int there is not that much difference. But suppose that you used a list:
class ClassHavingAList():
the_list = []
vs
class InstanceHavingAList()
def __init__(self):
self.the_list = []
Now, if you create 2 instances of both:
>>> c1 = ClassHavingAList()
>>> c2 = ClassHavingAList()
>>> i1 = InstanceHavingAList()
>>> i2 = InstanceHavingAList()
>>> c1.the_list is c2.the_list
True
>>> i1.the_list is i2.the_list
False
>>> c1.the_list.append(42)
>>> c2.the_list
[42]
c1.the_list and c2.the_list refer to the exactly same list object in memory, whereas i1.the_list and i2.the_list are distinct. Modifying the c1.the_list looks as if the c2.the_list also changes.
This is because the attribute itself is not set, it is just read. The c1.the_list.append(42) is identical in behaviour to
getattr(c1, 'the_list').append(42)
That is, it only tries read the value of attribute the_list on c1, and if not found there, then look it up in the superclass. The append does not change the attribute, it just changes the value that the attribute points to.
Now if you were to write an example that superficially looks the same:
c1.the_list += [ 42 ]
It would work identical to
original = getattr(c1, 'the_list')
new_value = original + [ 42 ]
setattr(c1, 'the_list', new_value)
And do a completely different thing: first of all the original + [ 42 ] would create a new list object. Then the attribute the_list would be created in c1, and set to point to this new list. That is, in case of instance.attribute, if the attribute is "read from", it can be looked up in the class (or superclass) if not set in the instance, but if it is written to, as in instance.attribute = something, it will always be set on the instance.
As for this:
class Test1(object):
self.i = 1
Such thing does not work in Python, because there is no self defined when the class body (that is all lines of code within the class) is executed - actually, the class is created only after all the code in the class body has been executed. The class body is just like any other piece of code, only the defs and variable assignments will create methods and attributes on the class instead of setting global variables.

I understood my newly added question. Thanks to Antti Haapala.
Now, when Python does the get attribute (as in print(instance.i) operation, it first looks for the attribute named i that is set on the instance). If that fails, the i attribute is looked up on type(i) instead (that is, the class attribute i).
I'm clear about why is:
j1 = Test4()
j2 = Test4()
>>> j1.i = 3
>>> j2.i
1
after few tests. The code
j1.3 = 3
actually creates a new instance variable for j1 without changing the class variable. That's the difference between "=" and methods like "append".
I'm a newbie of Python coming from c++. So, at the first glance, that's weird to me, since I never thought of creating a new instance variable which is not created in the class just using the "=". It's really a big difference between c++ and Python.
Now I got it, thank you all.

Print the variable assignment as output

I have probably a really simple question: Is it possible to print the value of a variable assignment without reentering the variable name?
I mean, when we enter let's say:
foo = 5
We get the following output:
5
I tried things like foo = 5; (as if I was using MATLAB - actually it hides the output) but, couldn't find any character that does this. Even in the tutorials I looked that, this was not mentioned.

The closest you can get to have this is to create a class overriding the __setattr__ method.
class A():
def __setattr__(self,name,value):
print(value)
self.__dict__['name'] = 1
a = A()
a.x = 1
1

No, this is not possible in Python. Variable assignments do not return anything:
>>> print(exec("a = 1"))
None
>>> a
1
>>>

Assigning a variable directly to a function in Python

Consider the following code:
def apples():
print(apples.applecount)
apples.applecount += 1
apples.applecount = 0
apples()
>>> 0
apples()
>>> 1
# etc
Is this a good idea, bad idea or should I just destroy myself?
If you're wondering why I would want this, I got a function repeating itself every 4 seconds, using win32com.client.Dispatch() it uses the windows COM to connect to an application. I think it's unnecessary to recreate that link every 4 seconds.
I could of course use a global variable, but I was wondering if this would be a valid method as well.

It would be more idiomatic to use an instance variable of a class to keep the count:
class Apples:
def __init__(self):
self._applecount = 0
def apples(self):
print(self._applecount)
self._applecount += 1
a = Apples()
a.apples() # prints 0
a.apples() # prints 1
If you need to reference just the function itself, without the a reference, you can do this:
a = Apples()
apples = a.apples
apples() # prints 0
apples() # prints 1

It is basically a namespaced global. Your function apples() is a global object, and attributes on that object are no less global.
It is only marginally better than a regular global variable; namespaces in general are a good idea, after all.

Can I override a class function without creating a new class in Python?

I'm making a game in pygame and I have made an 'abstract' class that's sole job is to store the sprites for a given level (with the intent of having these level objects in a list to facilitate the player being moved from one level to another)
Alright, so to the question. If I can do the equivalent of this in Python(code curtesy of Java):
Object object = new Object (){
public void overriddenFunction(){
//new functionality
};
};
Than when I build the levels in the game I would simply have to override the constructor (or a class/instance method that is responsible for building the level) with the information on where the sprites go, because making a new class for every level in the game isn't that elegant of an answer. Alternatively I would have to make methods within the level class that would then build the level once a level object is instantiated, placing the sprites as needed.
So, before one of the more stanch developers goes on about how anti-python this might be (I've read enough of this site to get that vibe from Python experts) just tell me if its doable.

Yes, you can!
class Foo:
def do_other(self):
print('other!')
def do_foo(self):
print('foo!')
def do_baz():
print('baz!')
def do_bar(self):
print('bar!')
# Class-wide impact
Foo.do_foo = do_bar
f = Foo()
g = Foo()
# Instance-wide impact
g.do_other = do_baz
f.do_foo() # prints "bar!"
f.do_other() # prints "other!"
g.do_foo() # prints "bar!"
g.do_other() # prints "baz!"
So, before one of the more stanch developers goes on about how anti-python this might be
Overwriting functions in this fashion (if you have a good reason to do so) seems reasonably pythonic to me. An example of one reason/way for which you might have to do this would be if you had a dynamic feature for which static inheritance didn't or couldn't apply.
The case against might be found in the Zen of Python:
Beautiful is better than ugly.
Readability counts.
If the implementation is hard to explain, it's a bad idea.

Yes, it's doable. Here, I use functools.partial to get the implied self argument into a regular (non-class-method) function:
import functools
class WackyCount(object):
"it's a counter, but it has one wacky method"
def __init__(self, name, value):
self.name = name
self.value = value
def __str__(self):
return '%s = %d' % (self.name, self.value)
def incr(self):
self.value += 1
def decr(self):
self.value -= 1
def wacky_incr(self):
self.value += random.randint(5, 9)
# although x is a regular wacky counter...
x = WackyCount('spam', 1)
# it increments like crazy:
def spam_incr(self):
self.value *= 2
x.incr = functools.partial(spam_incr, x)
print (x)
x.incr()
print (x)
x.incr()
print (x)
x.incr()
print (x)
and:
$ python2.7 wacky.py
spam = 1
spam = 2
spam = 4
spam = 8
$ python3.2 wacky.py
spam = 1
spam = 2
spam = 4
spam = 8
Edit to add note: this is a per-instance override. It takes advantage of Python's attribute look-up sequence: if x is an instance of class K, then x.attrname starts by looking at x's dictionary to find the attribute. If not found, the next lookup is in K. All the normal class functions are actually K.func. So if you want to replace the class function dynamically, use #Brian Cane's answer instead.

I'd suggest using a different class, via inheritance, for each level.
But you might get some mileage out of copy.deepcopy() and monkey patching, if you're really married to treating Python like Java.