Python infinite __call__ attribute - python

I have this simple function:
def f():
print("heh")
When I am calling f in reality I am calling its call method. But when I am calling a call method of f in reality I a calling a call method of call method of f. And so on and so on.
How does far does python go when call f(), it clearly must stop somewhere?
I was wondering whether this can go to infinity and it turns out that 100 000 is enough to crash Python.
>>> exec('f'+100000*'.__call__'+'()')
========= RESTART ==========
What's the reason of this crash?

A 'call' on an object causes the interpreter to look for a way to call it. When that is resolved by locating a __call__ method, that method is invoked, and then something real happens. The __call__ method can't just invoke the same mechanism on itself.
In the case of a function object, I believe there is an internal method table which is directly consulted first to see if there's a defined (C language) call handler, and that is invoked. There may also be a __call__ attribute which does the same thing, but I think the engine checks the table first (some of this may have been reworked in Py 3).
The 'C' langauge call handler for functions is handed a reference to the function object, and a package of parameters. The function object contains a reference to a code object, and another to the proper global namespace. The code object contains a description of what parameters are expected, and all the information needed to actually set up the call on the python stack.
When you call a method of a class, there's a little binder object with its own call method (containing a pointer to the 'self' and to the actual method').
I guess the main point is that some objects have __call__ methods coded in Python, but for many types the interpreter can go straight to C code after looking in the object's internal type descriptor. Another example is calling a type object , such as str, where the C-language constructor will be invoked.

Related

__del__() with args in additional to self

I've a Python program as follows:
class a:
def __init__(self,n):
self.n=n
def __del__(self,n):
print('dest',self.n,n)
def b():
d=a('d')
c=a('c')
d.__del__(8)
b()
Here, I have given a parameter n in __del__() just to clear my doubt. Its output :
$ python des.py
dest d 8
Exception ignored in: <function a.__del__ at 0xb799b074>
TypeError: __del__() missing 1 required positional argument: 'n'
Exception ignored in: <function a.__del__ at 0xb799b074>
TypeError: __del__() missing 1 required positional argument: 'n'
In classical programming languages like C++ we can't give parameters for the destructor. To know if it is applicable for python too, I've executed this program. Why does the interpreter allow the parameter n to be given as a parameter for the destructor? How can I specify value for that n then? As a try to give an argument for __del__() and it goes fine. But without it how can I specify the value for n?
You can define the __del__ method with an argument, as you've shown. And if you call the method yourself, you can pass in a value, just like you can with any other method. But when the interpreter calls __del__ itself, it's not going to pass anything, and there will be an exception raised.
However, because __del__ methods are often called in precarious situations, like when the interpreter is shutting down, Python doesn't stop everything if one raises an exception. Instead, it just prints out that it's ignoring the exception and keeps doing what it was doing already. This is why you see two "Exception ignored" messages, as your d and c objects get cleaned up.
It's unclear to me what you actually want your __del__ method to do with the n value you were passing in. Your example was a trivial case, usually there's nothing useful you can do there. Indeed, it's only rarely a good idea to write a __del__ method for a class. There are better ways of doing resource allocation, like the context manager protocol (which uses __enter__ and __exit__ methods).
you cannot. pre-defined dunder methods (methods with leading and trailing double underscore) like __del__ have a fixed signature.
If you define them with another signature, then when python calls them using the non-dunder interface (del, len, ...), the number of arguments is wrong and it fails.
To pass n to del, you'll have to define it as an object member.
Python objects become a candidate for garbage collection when there are no more references to them (object tagging), so you do not need to create such a destructor.
If you want to add optional arguments to a method, it's common to set them to None or an empty tuple ()
def other_del(self, x=None):
...

Are Python "function objects" functions?

We all know that functions are objects as well. But how do function objects compare with functions? What are the differences between function objects and functions?
By function object I mean g so defined:
class G(object):
def __call__(self, a):
pass
g = G()
By function I mean this:
def f(a):
pass
Python creates function objects for you when you use a def statement, or you use a lambda expression:
>>> def foo(): pass
...
>>> foo
<function foo at 0x106aafd70>
>>> lambda: None
<function <lambda> at 0x106d90668>
So whenever you are defining a function in python an object like this is created. There is no plain way of function definition.
TL;DR any object that implements __call__ can be called eg: functions, custom classes, etc..
Slightly longer version: (walloftext)
The full answer to your question on what's the difference sits within the implementation of the python virtual machine, so we must take a look at python under the hood. First comes the concept of a code object. Python parses whatever you throw at it into it's own internal language that is the same across all platforms known as bytecode. A very visual represnetation of this is when you get a .pyc file after importing a custom library you wrote. These are the raw instructions for the python VM. Ignoring how these instructions are created from your source code, they are then executed by PyEval_EvalFrameEx in Python/ceval.c. The source code is a bit of a beast, but ultimately works like a simple processor with some of the complicated bits abstracted away. The bytecode is the assembly language for this processor. In particular one of the "opcodes" for this "processor" is (aptly named) CALL_FUNCTION. The callback goes through a number of calls eventually getting to PyObject_Call(). This function takes a pointer to a PyObject and extracts the tp_call attribute from it's type and directly calls it (technically it checks if it's there first):
...
call = func->ob_type->tp_call //func is an arg of PyObject_Call() and is a pointer to a PyObject
...
result = (*call)(func, arg, kw);
Any object that implements __call__ is given a tp_call attribute with a pointer to the actual function. I believe that is handled by the slotdefs[] difinition from Objects/typeobject.c:
FLSLOT("__call__", tp_call, slot_tp_call, (wrapperfunc)wrap_call,
"__call__($self, /, *args, **kwargs)\n--\n\nCall self as a function.",
PyWrapperFlag_KEYWORDS)
The __call__ method itself for functions is defined in the cpython implementation and it defines how the python VM should start executing the bytecode for that function and how data should be returned (if any). When you give an arbitrary class a __call__ method, the attribute is a function object that again refers back to the cpython implementation of __call__. Therefore when you call a "normal" function foo.__call__ is referenced. when you call a callable class, the self.__call__ is equivalent to foo and the actual cpython reference called is self.__call__.im_func.__call__.
disclaimer
This has been a journey into somewhat uncharted waters for me, and it's entirely possible I have misrepresented some of the finer points of the implementation. I mainly took from this blog post on how python callables work under the hood, and some digging of my own through the python source code

Python functional programming reference to a constructor

I would like to have a function pointer ptr that can point to either:
a function,
the method of an object instance, or
the constructor of the object.
In the latter case, the execution of ptr() should instantiate the class.
def function(argument) :
print("Function called with argument: "+str(argument))
class C(object) :
def __init__(self,argument) :
print("C's __init__ method called with argument: "+str(argument))
def m(self,argument) :
print("C's method 'm' called with argument: "+str(argument))
## works
ptr = function
ptr('A')
## works
instance = C('asdf')
ptr = instance.m
ptr('A')
## fails
constructorPtr = C.__init__
constructorPtr('A')
This produces as output:
Function called with argument: A
C's __init__ method called with argument: asdf
C's method 'm' called with argument: A
Traceback (most recent call last): File "tmp.py", line 24, in <module>
constructorPtr('A')
TypeError: unbound method __init__() must be called with C instance as first argument (got str instance instead)
showing that the first two ptr() calls worked, but the last did not.
The reason this doesn't work is that the __init__ method isn't a constructor, it's an initializer.*
Notice that its first argument is self—that self has to be already constructed before its __init__ method gets called, otherwise, where would it come from.
In other words, it's a normal instance method, just like instance.m is, but you're trying to call it as an unbound method—exactly as if you'd tried to call C.m instead of instance.m.
Python does have a special method for constructors, __new__ (although Python calls this a "creator" to avoid confusion with languages with single-stage construction). This is a static method that takes the class to construct as its first argument and the constructor arguments as its other arguments. The default implementation that you've inherited from object just creates an instance of that class and passes the arguments along to its initializer.** So:
constructor = C.__new__
constructor(C, 'A')
Or, if you prefer:
from functools import partial
constructor = partial(C.__new__, C)
constructor('A')
However, it's incredibly rare that you'll ever want to call __new__ directly, except from a subclass's __new__. Classes themselves are callable, and act as their own constructors—effectively that means that they call the __new__ method with the appropriate arguments, but there are some subtleties (and, in every case where they differ, C() is probably what you want, not C.__new__(C)).
So:
constructor = C
constructor('A')
As user2357112 points out in a comment:
In general, if you want a ptr that does whatever_expression(foo) when you call ptr(foo), you should set ptr = whatever_expression
That's a great, simple rule of thumb, and Python has been carefully designed to make that rule of thumb work whenever possible.
Finally, as a side note, you can assign ptr to anything callable, not just the cases you described:
a function,
a bound method (your instance.m),
a constructor (that is, a class),
an unbound method (e.g., C.m—which you can call just fine, but you'll have to pass instance as the first argument),
a bound classmethod (e.g., both C.cm and instance.cm, if you defined cm as a #classmethod),
an unbound classmethod (harder to construct, and less useful),
a staticmethod (e.g., both C.sm and instance.sm, if you defined sm as a #staticmethod),
various kinds of implementation-specific "builtin" types that simulate functions, methods, and classes.
an instance of any type with a __call__ method,
And in fact, all of these are just special cases of the last one—the type type has a __call__ method, as do types.FunctionType and types.MethodType, and so on.
* If you're familiar with other languages like Smalltalk or Objective-C, you may be thrown off by the fact that Python doesn't look like it has two-stage construction. In ObjC terms, you rarely implement alloc, but you call it all the time: [[MyClass alloc] initWithArgument:a]. In Python, you can pretend that MyClass(a) means the same thing (although really it's more like [MyClass allocWithArgument:a], where allocWithArgument: automatically calls initWithArgument: for you).
** Actually, this isn't quite true; the default implementation just returns an instance of C, and Python automatically calls the __init__ method if isinstance(returnvalue, C).
I had a hard time finding the answer to this problem online, but I figured it out, so here is the solution.
Instead of pointing constructorPtr at C.__init__, you can just point it at C, like this.
constructorPtr = C
constructorPtr('A')
which produces as output:
C's __init__ method called with argument: A

Converting functions into a methods of a class in python

I'm trying to get the hang of python 2.6, which is my introduction to a programming language. I'm following Learn Python the Hard Way and I must have missed something. I have a set of functions that I want to combine into a class. These functions are contained here. I tried to combine these into a class called "Parsers". This code is contained here.
The functions themselves work outside of the class. I'm trying to call the functions out of the class like this:
import the module:
>>> from ex48 import parser2
Assign the class:
>>> parser_class = parser2.Parsers()
Call a function:
>>> parser_class.parse_subject(word_list, ('noun', 'player'))
I get the following:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "ex48/parser2.py", line 60, in parse_subject
verb = parse_verb(word_list)
NameError: global name 'parse_verb' is not defined
Not sure why it's telling me that. Can someone point me in the right direction?
You are confused as to what your code is doing.
This does not assign, a class, it creates an instance, and assigns that instance to a variable:
parser_class = parser2.Parsers()
This calls a method on that instance:
parser_class.parse_subject(word_list, ('noun', 'player'))
The following tells you that there is no global function (or indeed variable of any type) parse_verb:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "ex48/parser2.py", line 60, in parse_subject
verb = parse_verb(word_list)
NameError: global name 'parse_verb' is not defined
I expect that you want to change that to verb = self.parse_verb(word_list), which will call the parse_verb method of the same object (self).
There are two problems here, one mechanical, one conceptual. I'll deal with the latter first.
The whole point of a class is that its instances represent some kind of object in your model of the world. For example, a "sentence" is an object, so it makes sense to have a class called Sentence, but a "parsers" is not an object, so Parsers is probably going to be a confused class. Looking at things less abstractly, a class usually has some kind of state, encapsulated in data members that its methods work on.
What you really have here is a parser function that returns a Sentence object (or raises a ParserError). This function could be a method of Sentence, or it could be a free function; it could also be a method of a Parser class if there were any reason for such an object to exist in your model, but there doesn't seem to be one.
If you're just looking to encapsulate all those helper functions so they don't litter the global namespace (a good idea in itself), you don't need to abuse a class to do that. Just make them local functions inside parse_sentence, or put them in a module called parsers.
Now, on to the mechanical problem: If you have a method in a class, you normally only call it through dot syntax: someObject.methodName(params). When a method calls another method of the same object, you use the special self parameter (which you've correctly listed as the first param of each method) to do that. So:
verb = self.parse_verb(word_list)
obj = self.parse_object(word_list)
… and so on for all the other method calls in your sample.
Not every language requires an explicit self like this. For example, in C++ or related languages (Java, C#, etc.), a method can call another method without specifying a target, and it's implicitly assumed that the target is this (the C++ equivalent of self).
It looks like you are working from a python interactive session.
If you've made any changes to parser2.py after importing parser2, then you have to
reload(parser2)
to make those changes known to the interactive interpreter.

Python: multiple calls to __init__() on the same instance

The __init__() function gets called when object is created.
Is it ok to call an object __init__() function again, after its been created?
instance = cls(p1=1, p2=2)
# some code
instance.__init__(p1=123, p2=234)
# some more code
instance.__init__(p1=23, p2=24)
why would anyone wanna call __init__() on an object that is already created?
good question. i wanna re-initialize the instance's fields.
It's fine to call __init__ more than once on an object, as long as __init__ is coded with the effect you want to obtain (whatever that may be). A typical case where it happens (so you'd better code __init__ appropriately!-) is when your class's __new__ method returns an instance of the class: that does cause __init__ to be called on the returned instance (for what might be the second, or twentieth, time, if you keep "recycling" instances via your __new__!-).
You can, but it's kind of breaking what __init__ is intended to do. A lot of Python is really just convention, so you might as well follow then and expect __init__ to only be called once. I'd recommend creating a function called init or reset or something which sets the instance variables, use that when you want to reset the instance, and have __init__ just call init. This definitely looks more sane:
x = Pt(1,2)
x.set(3,4)
x.set(5,10)
As far as I know, it does not cause any problems (edit: as suggested by the kosher usage of super(...).__init__(...)), but I think having a reset() method and calling it both in __init__() and when you need to reset would be cleaner.

Categories