We Keep Coding

classmethod with different overloaded signature between instance and base class

I am trying to write a class with an additional constructing method that accepts extra values. These extra values are expensive to compute, and are saved at the end of the program, so .initialize() effectively serves as an injection to avoid recomputing them again at subsequent runs of the program.
class TestClass:
init_value: str
secondary_value: int
#overload
#classmethod
def initialize(cls: type["TestClass"], init_value: str, **kwargs) -> "TestClass":
...
#overload
#classmethod
def initialize(cls: "TestClass", **kwargs) -> "TestClass":
# The erased type of self "TestClass" is not a supertype of its class "Type[TestClass]
...
#classmethod
def initialize(cls: type["TestClass"] | "TestClass", init_value: str | None = None, **kwargs) -> "TestClass":
if isinstance(cls, type):
instance = cls(init_value=init_value)
# Argument "init_value" to "TestClass" has incompatible type "Optional[str]"; expected "str"
else:
instance = cls
for extra_key, extra_value in kwargs.items():
setattr(instance, extra_key, extra_value)
return instance
def __init__(self, init_value: str) -> None:
self.init_value = init_value
instance1 = TestClass.initialize(init_value="test", secondary_value="test2")
instance2 = TestClass(init_value="test").initialize(secondary_value="test2")
# Missing positional argument "init_value" in call to "initialize" of "TestClass"
instance1.init_value
instance2.init_value
instance1.secondary_value
instance2.secondary_value
How can I make the above work so that TestClass(init_value).initialize() does not require init_value passed to .initialize() because it's already been declared in __init__, while TestClass.initialize() does?
In short, how can I define a classmethod with different typing depending on whether it's called on an instance or a class?
These extra values cannot be declared in __init__, because of complex internals of the class that would be too long to repeat here.

There are two distinct problems here:
how to declare a method that can be called either on an instance or on the class object itself
how to explain that hack to mypy
Let us begin with the easy part and forget about mypy for a while.
When a method is called on an instance object (meaning it is declared on the type of the object), Python knows that it is a method call and prepends the passed arguments with the object itself: it is the common usage for a method.
When a method is called on the class object (meaning it is declared on the object itself), Python sees it at a normal function call and simply forwards the passed arguments. For that reason if foo is a method of Bar class, and if bar is a Bar instance, the following snippets are strictly equivalent:
bar.foo(args...)
Bar.foo(bar, args...)
That means that for your initialize method to be callable from the class object or from an instance of it, it should be declared:
def initialize(init_value, **kwargs)-> "TestClass":
if isinstance(init_value, str): # handle a str from the class object
instance = TestClass(init_value)
elif not isinstance(init_value, TestClass): # reject any other call...
raise TypeError("Argument of initialize must be str")
else: # handle a call from an instance object
instance = init_value
for extra_key, extra_value in kwargs.items():
setattr(instance, extra_key, extra_value)
return instance
This code will work fine at run time.
Now for the mypy part...
Unfortunately, mypy goal is to help the Python programmers to provide consistent and easy to maintain code. For that, it provides a static type analyzer to make sure that the run time parameters are consistent with the declarations and that multiple overload declarations are all consistent. Here begins the hell for your requirement:
to be callable from the class object, the method would require a classmethod overload
to be callable from the instance object and still give access to that instance object, the final declaration shall not be a classmethod nor a staticmethod
Said differently, no way for overload declarations...
So the only way I found is to tell the truth: you have defined an attribute on TestClass which is a function that can receive as its first argument either a TestClass object when it is called as a method, or a string when it is called on the class object itself
...
def initialize(init_value: Union['TestClass', str], **kwargs)-> "TestClass"::
if isinstance(init_value, str):
instance = TestClass(init_value)
...

python property decorator for __name__ attr in class

I find a good desc for python property in this link
How does the #property decorator work in Python?
below example shows how it works, while I find an exception for class attr 'name'
now I have a reload function which will raise an error
#property
def foo(self): return self._foo
really means the same thing as
def foo(self): return self._foo
foo = property(foo)
here is my example
class A(object):
#property
def __name__(self):
return 'dd'
a = A()
print(a.__name__)
dd
this works, however below cannot work
class B(object):
pass
def test(self):
return 'test'
B.t = property(test)
print(B.t)
B.__name__ = property(test)
<property object at 0x7f71dc5e1180>
Traceback (most recent call last):
File "<string>", line 23, in <module>
TypeError: can only assign string to B.__name__, not 'property'
Does anyone knows the difference for builtin name attr, it works if I use normal property decorator, while not works for the 2nd way. now I have a requirement to reload the function when code changes, however this error will block the reload procedure. Can anyone helps? thanks.

The short answer is: __name__ is deep magic in CPython.
So, first, let's get the technicalities out of the way. To quote what you said
#property
def foo(self): return self._foo
really means the same thing as
def foo(self): return self._foo
foo = property(foo)
This is correct. But it can be a bit misleading. You have this A class
class A(object):
#property
def __name__(self):
return 'dd'
And you claim that it's equivalent to this B class
class B(object):
pass
def test(self):
return 'test'
B.__name__ = property(test)
which is not correct. It's actually equivalent to this
def test(self):
return 'test'
class B(object):
__name__ = property(test)
which works and does what you expect it to. And you're also correct that, for most names in Python, your B and my B would be the same. What difference does it make whether I'm assigning to a name inside the class or immediately after its declaration? Replace __name__ with ravioli in the above snippets and either will work. So what makes __name__ special?
That's where the magic comes in. When you define a name inside the class, you're working directly on the class' internal dictionary, so
class A:
foo = 1
def bar(self):
return 1
This defines two things on the class A. One happens to be a number and the other happens to be a function (which will likely be called as a bound method). Now we can access these.
A.foo # Returns 1, simple access
A.bar # Returns the function object bar
A().foo # Returns 1
A().bar # Returns a bound method object
When we look up the names directly on A, we simply access the slots like we would on any object. However, when we look them up on A() (an instance of A), a multi-step process happens
Look up the name on the instance's __dict__ directly.
If that failed, then look up the name on the class' __dict__.
If we found it on the class, see if there's a __get__ on the result and call it.
That third step is what allows bound method objects to work, and it's also the mechanism underlying the property decorators in Python.
Let's go through this whole process with a property called ravioli. No magic here.
class A(object):
#property
def ravioli(self):
return 'dd'
When we do A().ravioli, first we see if there's a ravioli on the instance we just made. There isn't, so we check the class' __dict__, and indeed we find a property object at that position. That property object has a __get__, so we call it, and it returns 'dd', so indeed we get the string 'dd'.
>>> A().ravioli
'dd'
Now I would expect that, if I do A.ravioli, we will simply get the property object. Since we're not calling it on an instance, we don't call __get__.
>>> A.ravioli
<property object at 0x7f5bd3690770>
And indeed, we get the property object, as expected.
Now let's do the exact same thing but replace ravioli with __name__.
class A(object):
#property
def __name__(self):
return 'dd'
Great! Now let's make an instance.
>>> A().__name__
'dd'
Sensible, we looked up __name__ on A's __dict__ and found a property, so we called its __get__. Nothing weird.
Now
>>> A.__name__
'A'
Um... what? If we had just found the property on A's __dict__, then we should see that property here, right?
Well, no, not always. See, in the abstract, foo.bar normally looks in foo.__dict__ for a field called bar. But it doesn't do that if the type of foo defines a __getattribute__. If it defines that, then that method is always called instead.
Now, the type of A is type, the type of all Python types. Read that sentence a few times and make sure it makes sense. And if we do a bit of spelunking into the CPython source code, we see that type actually defines __getattribute__ and __setattr__ for the following names:
__name__
__qualname__
__bases__
__module__
__abstractmethods__
__dict__
__doc__
__text_signature__
__annotations__
That explains how __name__ can serve double duty as a property on the class instances and also as an accessible field on the same class. It also explains why you get that highly specialized error message when reassigning to B.__name__: the line
B.__name__ = property(test)
is actually equivalent to
type.__setattr__(B, '__name__', property(test))
which is calling our special-case checker in CPython.
For any other type in Python, in particular for user-defined types, we could get around this with object.__setattr__. Unfortunately,
>>> object.__setattr__(B, '__name__', property(test))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: can't apply this __setattr__ to type object
There's a really specific check to make sure we don't do exactly this, and the comment reads
/* Reject calls that jump over intermediate C-level overrides. */
We also can't use metaclasses to override __setattr__ and __getattribute__, because the instance lookup procedure specifically doesn't call those (in the above examples, __getattribute__ was called in every case except the one we care about for property purposes). I even tried subclassing str to trick __setattr__ into accepting our made-up value
class NameProperty(str):
def __new__(cls, value, **kwargs):
return str.__new__(cls, value)
def __init__(self, value, method):
self.method = method
def __get__(self, instance, owner):
return self.method(instance)
B.__name__ = NameProperty(B.__name__, method=test)
This actually passes the __setattr__ check, but it doesn't assign to B.__dict__ (since the __setattr__ still assigns to the actual CPython-level name, not to B.__dict__['__name__']), so the property lookup doesn't work.
So... that's how I reached my conclusion of: __name__ is deep magic in CPython. All of the usual Python metaprogramming techniques have failed, and all of the methods getting called are written deep down in C. My advice to you is: Stop using __name__ for things it's not intended for, or be prepared to write some C code and hack on CPython directly.

Why doesn't Python have an instancemethod function?

Why doesn't Python have an instancemethod function analogous to staticmethod and classmethod?
Here is how this arose for me. Suppose I have an object which I know will be hashed frequently and whose hash is expensive to calculate. Under this assumption, it is reasonable to compute the hash value once and cache it, as in the following toy example:
class A:
def __init__(self, x):
self.x = x
self._hash_cache = hash(self.x)
def __hash__(self):
return self._hash_cache
The __hash__ function in this class does very little, just an attribute lookup and a return. Naively, it seems it ought to be equivalent to instead write:
class B:
def __init__(self, x):
self.x = x
self._hash_cache = hash(self.x)
__hash__ = operator.attrgetter('_hash_cache')
According to the documentation, operator.attrgetter returns a callable object that fetches the given attribute from its operand. If its operand is self, then it will return self._hash_cache, which is the desired result. Unfortunately this does not work:
>>> hash(A(1))
1
>>> hash(B(1))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: attrgetter expected 1 arguments, got 0
The reason for this is as follows. If one reads the descriptor HOWTO, one finds that class dictionaries store methods as functions; functions are non-data descriptors whose __get__ method returns a bound method. But operator.attrgetter does not return a function; it returns a callable object. And in fact, it is a callable object with no __get__ method:
>>> hasattr(operator.attrgetter('_hash_cache'), '__get__')
False
Lacking a __get__ method, this of course will not automatically be turned into a bound method. We can make a bound method from it using types.MethodType, but using it in our class B would require creating a bound method for every object instance and assigning it to __hash__.
We can see the fact that operator.attrgetter has no __get__ directly if we browse the CPython source. I'm not very familiar with the CPython API, but I believe that what's going on is as follows. The definition of the attrgetter_type is in Modules/_operator.c, at line 1439 as I write this. This type sets tp_descr_get to 0. And according to the type object documentation, that means an object whose type is attrgetter_type will not have a __get__.
Of course, if we give ourselves a __get__ method, then everything works. This is the case in the first example above, where __hash__ is actually a function and not just a callable. It's also true in some other cases. For example, if we want to lookup a class attribute, we could write the following:
class C:
y = 'spam'
get_y = classmethod(operator.attrgetter('y'))
As written this is terribly un-Pythonic (though it might be defensible if there were a strange custom __getattr__ for which we wanted to provide convenience functions). But at least it gives the desired result:
>>> C.get_y()
'spam'
I can't think of any reason why it would be bad for attrgetter_type to implement __get__. But on the other hand, even if it did, there would be other situations where we run into trouble. For example, suppose we have a class whose instances are callable:
class D:
def __call__(self, other):
...
We can't use an instance of this class as a class attribute and expect instance lookups to generate bound methods. For instance,
d = D()
class E:
apply_d = d
When D.__call__ is called, it will receive self but not other, and that generates a TypeError. This example might be a little far-fetched, but I'd be a little surprised if nobody had ever encountered something like this in practice. It could be fixed by giving D a __get__ method; but if D is from a third-party library that could be inconvenient.
It seems that the easiest solution would be to have an instancemethod function. Then we could write __hash__ = instancemethod(operator.attrgetter('_hash_cache')) and apply_d = instancemethod(d) and they would both work as intended. Yet, as far as I know, no such function exists. Hence my question: Why is there no instancemethod function?
EDIT: Just to be clear, the functionality of instancemethod would be equivalent to:
def instancemethod(func):
#functools.wraps(func)
def wrapper(*args, **kwargs):
return func(*args, **kwargs)
return wrapper
This could be applied as in the original question above. One could also imagine writing a class decorator that could be applied to D that would give it a __get__ method; but this code doesn't do this.
So I'm not talking about adding a new feature to Python. Really the question is one of language design: Why not provide it as, say, functools.instancemethod? If the answer is simply, "The use cases are so obscure that nobody's bothered," that's okay. But I would be happy to learn about other reasons, if there are any.

There is no instancemethod decorator because this is the default behaviour for functions declared inside a class.
class A:
...
# This is an instance method
def __hash__(self):
return self._hash_cache
Any callable which does not have a __get__ method can thus be wrapped into an instance method like so.
class A:
def instance_method(*args):
return any_callable(*args)
Thus creating an instancemethod decorator would just add another syntax for a feature which already exists. This would go against the saying that there should be one-- and preferably only one --obvious way to do it.
Side note
If it is so expensive to hash your instances, you might want to avoid calling you hash function on instantiation and delay it for when the object are hashed.
One way to do that could be to set the attribute _hash_cache in __hash__ instead of __init__. Although, let me suggest a slightly more self-contained methods which relies on caching your hash.
from weakref import finalize
class CachedHash:
def __init__(self, x):
self.x = x
def __hash__(self, _cache={}):
if id(self) not in _cache:
finalize(self, _cache.pop, id(self))
_cache[id(self)] = hash(self.x) # or some complex hash function
return _cache[id(self)]
The use of finalize ensures the cache is cleared of an id when its instance is garbage collected.

I have a satisfying answer to my question. Python does have the internal interface necessary for an instancemethod function, but it's not exposed by default.
import ctypes
import operator
instancemethod = ctypes.pythonapi.PyInstanceMethod_New
instancemethod.argtypes = (ctypes.py_object,)
instancemethod.restype = ctypes.py_object
class A:
def __init__(self, x):
self.x = x
self._hash_cache = hash(x)
__hash__ = instancemethod(operator.attrgetter('_hash_cache'))
a = A(1)
print(hash(a))
The instancemethod function this creates works in essentially the same way as classmethod and staticmethod. These three functions return new objects of types instancemethod, classmethod, and staticmethod, respectively. We can see how they work by looking at Objects/funcobject.c. These objects all have __func__ members which store a callable object. They also have a __get__. For a staticmethod object, the __get__ returns __func__ unchanged. For a classmethod object, __get__ returns a bound method object, where the binding is to the class object. And for a staticmethod object, __get__ returns a bound method object, where the binding is to the object instance. This is precisely the same behavior as __get__ for a function object and is exactly what we want.
The only documentation on these objects seems to be in the Python C API here. My guess is that they're not exposed because they're so rarely needed. I think it would be nice to have PyInstanceMethod_New available as functools.instancemethod.

A function in a class without any decorator or `self`

I have following class with a function:
class A:
def myfn():
print("In myfn method.")
Here, the function does not have self as argument. It also does not have #classmethod or #staticmethod as decorator. However, it works if called with class:
A.myfn()
Output:
In myfn method.
But give an error if called from any instance:
a = A()
a.myfn()
Error output:
Traceback (most recent call last):
File "testing.py", line 16, in <module>
a.myfn()
TypeError: myfn() takes 0 positional arguments but 1 was given
probably because self was also sent as an argument.
What kind of function will this be called? Will it be a static function? Is it advisable to use function like this in classes? What is the drawback?
Edit: This function works only when called with class and not with object/instance. My main question is what is such a function called?
Edit2: It seems from the answers that this type of function, despite being the simplest form, is not accepted as legal. However, as no serious drawback is mentioned in any of many answers, I find this can be a useful construct, especially to group my own static functions in a class that I can call as needed. I would not need to create any instance of this class. In the least, it saves me from typing #staticmethod every time and makes code look less complex. It also gets derived neatly for someone to extend my class. Although all such functions can be kept at top/global level, keeping them in class is more modular. However, I feel there should be a specific name for such a simple construct which works in this specific way and it should be recognized as legal. It may also help beginners understand why self argument is needed for usual functions in a Python class. This will only add to the simplicity of this great language.

The function type implements the descriptor protocol, which means when you access myfn via the class or an instance of the class, you don't get the actual function back; you get instead the result of that function's __get__ method. That is,
A.myfn == A.myfn.__get__(None, A)
Here, myfn is an instance method, though one that hasn't been defined properly to be used as such. When accessed via the class, though, the return value of __get__ is simply the function object itself, and the function can be called the same as a static method.
Access via an instance results in a different call to __get__. If a is an instance of A, then
a.myfn() == A.myfn.__get__(a, A)
Here , __get__ tries to return, essentially, a partial application of myfn to a, but because myfn doesn't take any arguments, that fails.
You might ask, what is a static method? staticmethod is a type that wraps a function and defines its own __get__ method. That method returns the underlying function whether or not the attribute is accessed via the class or an instance. Otherwise, there is very little difference between a static method and an ordinary function.

This is not a true method. Correctly declarated instance methods should have a self argument (the name is only a convention and can be changed if you want hard to read code), and classmethods and staticmethods should be introduced by their respective decorator.
But at a lower level, def in a class declaration just creates a function and assigns it to a class member. That is exactly what happens here: A.my_fn is a function and can successfully be called as A.my_fn().
But as it was not declared with #staticmethod, it is not a true static method and it cannot be applied on a A instance. Python sees a member of that name that happens to be a function which is neither a static nor a class method, so it prepends the current instance to the list of arguments and tries to execute it.
To answer your exact question, this is not a method but just a function that happens to be assigned to a class member.

Such a function isn't the same as what #staticmethod provides, but is indeed a static method of sorts.
With #staticmethod you can also call the static method on an instance of the class. If A is a class and A.a is a static method, you'll be able to do both A.a() and A().a(). Without this decorator, only the first example will work, because for the second one, as you correctly noticed, "self [will] also [be] sent as an argument":
class A:
#staticmethod
def a():
return 1
Running this:
>>> A.a() # `A` is the class itself
1
>>> A().a() # `A()` is an instance of the class `A`
1
On the other hand:
class B:
def b():
return 2
Now, the second version doesn't work:
>>> B.b()
2
>>> B().b()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: b() takes 0 positional arguments but 1 was given

further to #chepnet's answer, if you define a class whose objects implement the descriptor protocol like:
class Descr:
def __get__(self, obj, type=None):
print('get', obj, type)
def __set__(self, obj, value):
print('set', obj, value)
def __delete__(self, obj):
print('delete', obj)
you can embed an instance of this in a class and invoke various operations on it:
class Foo:
foo = Descr()
Foo.foo
obj = Foo()
obj.foo
which outputs:
get None <class '__main__.Foo'>
get <__main__.Foo object at 0x106d4f9b0> <class '__main__.Foo'>
as functions also implement the descriptor protocol, we can replay this by doing:
def bar():
pass
print(bar)
print(bar.__get__(None, Foo))
print(bar.__get__(obj, Foo))
which outputs:
<function bar at 0x1062da730>
<function bar at 0x1062da730>
<bound method bar of <__main__.Foo object at 0x106d4f9b0>>
hopefully that complements chepnet's answer which I found a little terse/opaque

Can someone explain how the source code of staticmethod works in python

First of all, I understand how, in general, a decorator work. And I know #staticmethod strips off the instance argument in the signature, making
class C(object):
#staticmethod
def foo():
print 'foo'
C.foo //<function foo at 0x10efd4050>
C().foo //<function foo at 0x10efd4050>
valid.
However, I don't understand how the sourcec code of staticmethod make this happen.
It seems to me that when wrapping method foo in staticmethod, an instance of staticmethod is instantiated, then some magic happens, making C.foo() legit.
So.. what happen in those magic? what did staticmethod do?
I'm aware the enormous topics on SO regarding staticmethods but none of them addresses my doubts. But maybe I didn't hit the magic keyword. If so, please kindly let me know.
For whoever looking for staticmethod source code, please refer to https://hg.python.org/cpython/file/c6880edaf6f3/Objects/funcobject.c

A staticmethod object is a descriptor. The magic you are missing is that Python calls the __get__ method when accessing the object as an attribute on a class or instance.
So accessing the object as C.foo results in Python translating that to C.__dict__['foo'].__get__(None, C), while instance_of_C.foo becomes type(instace_of_C).__dict__['foo'].__get__(instance_of_C, type(instance_of_C)).
The staticmethod object is defined in C code, but an equivalent in Python would be:
class staticmethod(object):
def __init__(self, callable):
self.f = callable
def __get__(self, obj, type=None):
return self.f
#property
def __func__(self):
return self.f
where self.f is the original wrapped function.
All this is needed because functions are themselves descriptors too; it is the descriptor protocol that gives you method objects (see python bound and unbound method object for more details). Since they too have a __get__ method, without a staticmethod object wrapping the function, a functionobj.__get__ call produces a method object instead, passing in a self argument.
There is also a classmethod, which uses the second argument to descriptor.__get__ to bind a function to the class, and then there are property objects, which translate binding into a function call directly. See How does the #property decorator work?.

The source code of static method is actually written in C, but you can replicate its behavior with this short snippet:
class staticmethod:
def __init__(self, decorated_method):
self.decorated_method = decorated_method
def __get__(self, instance, owner):
return self.decorated_method
Here is brief explanation of the above snippet, from this article I wrote: “Methods are descriptors and hence implement the magic method __get__. #staticmethod is a class decorator, reimplementing __get__ and making it return a version of the decorated method that has not been bound to the object it belongs to. Hence, the decorated method does not have the implicit argument self.” If the notions of descriptors, bound methods or class descriptors are not clear to you, they are explained in the article.