We Keep Coding

Strange behavior with contextmanager

Take a look at the following example:
from contextlib import AbstractContextManager, contextmanager
class MyClass(AbstractContextManager):
_values = {}
#contextmanager
def using(self, name, value):
print(f'Allocating {name} = {value}')
self._values[name] = value
try:
yield
finally:
print(f'Releasing {name}')
del self._values[name]
def __enter__(self):
return self.using('FOO', 42).__enter__()
def __exit__(self, exc_type, exc_val, exc_tb):
pass
with MyClass():
print('Doing work...')
I would expect the above code to print the following:
Allocating FOO = 42
Doing work...
Releasing FOO
Instead, this is what is being printed:
Allocating FOO = 42
Releasing FOO
Doing work...
Why is FOO getting released eagerly?

You're creating two context managers here. Only one of those context managers is actually implemented correctly.
Your using context manager is fine, but you've also implemented the context manager protocol on MyClass itself, and the implementation on MyClass is broken. MyClass.__enter__ creates a using context manager, enters it, returns what that context manager's __enter__ returns, and then throws the using context manager away.
You don't exit the using context manager when MyClass() is exited. You never exit it at all! You throw the using context manager away. It gets reclaimed, and when it does, the generator gets close called automatically, as part of normal generator cleanup. That throws a GeneratorExit exception into the generator, triggering the finally block.
Python doesn't promise when this cleanup will happen (or indeed, if it will happen at all), but in practice, CPython's reference counting mechanism triggers the cleanup as soon as the using context manager is no longer reachable.
Aside from that, if _values is supposed to be an instance variable, it should be set as self._values = {} inside an __init__ method. Right now, it's a class variable.

Is it possible to raise Exception if function is not called

For some given class in python like:
class Foo:
def __init__(self, ...):
...
pass
def not_raising_1(self, ...):
...
pass
def not_raising_2(self, ...):
...
pass
is it possible to enforce that the user has to call not_raising_1() or not_raising_2() after creating an object of type Foo. So I'm thinking of a behavior:
foo = Foo(...) # will raise a Exception saying you need to call not_raising_1 or not_raising_1
foo = Foo(...).not_raising_1(...) # will NOT raise a Excpetion
foo = Foo(...).not_raising_2(...) # will NOT raise a Excpetion
I know that a pragmatic solution would obviously be to put what ever should happen in not_raising_1() or not_raising_2() with some parameter in the constructor of Foo. But I'm here not asking for a pragmatic solution but am just curios if someone can think of some creative solution to get the described behavior.

First, for the record: if the two methods must be called before the object is ready to be used, that means that calling them is part of the initialization of the object, and so they should be called by __init__ itself:
class Foo:
def __init__(self, ...):
...
self.not_raising_1()
self.not_raising_2()
def not_raising_1(self, ...):
...
def not_raising_2(self, ...):
...
But, moving on to the question as asked...
The problem is not well defined.
Unless you call the methods inside __init__ itself, it is trivially true that neither method has been called the instant __init__ exits.
Further, once __init__ exits, the class Foo has no visibility into what happens outside its own definition. You need some sort of explicit state that maintains what happens after Foo.__init__ exits. Foo.not_raising_1 could examine that state to determine if anything else happened before it was called.
But that raises another problem: who will update that state? Every single bit of code would have to cooperate with Foo. Is this illegal?
x = Foo()
y = 3
x.not_raising_1()
Then how are you going to make Python update your state when it executes y = 3? The hooks just aren't there.
And finally, who is going to raise the exception if x.not_raising_1 is never called?
Refine the problem.
Rather than ask if the functions are never called, you can ensure they are called inside a with statement using an appropriately defined context manager. This context manager can ensure that not_raising_1 and not_raising_2 are called before the with statement completes, as well as ensure that they are only used inside a with statement. You can't enforce that the object is used as a context manager, but you can ensure that it is only used in a with statement.
class Foo:
def __init__(self, ...):
...
self._in_with_statement = False
self._r1_called = False
self._r2_called = False
def not_raising_1(self, ...):
self._r1_called = True
if not self._in_with_statement
raise RuntimeException("Foo instance must be used with context manager")
def not_raising_2(self, ...):
self._r2_called = True
if not self._in_with_statement
raise RuntimeException("Foo instance must be used with context manager")
def something_else(self):
if not self._r1_called or not self._r2_called:
raise RuntimeException("Failed to call not_raising_1 and/or not_raising_2")
...
def __enter__(self):
self._in_with_statement = True
def __exit__(self):
self._in_with_statement = False
if not self._r1_called or not self._r2_called:
raise RuntimeException("Failed to call not_raising_1 and/or not_raising_2")
self._r1_called = False
self._r2_called = False
Here, __init__ sets the condition that neither method has yet been called, nor are we yet executing in a with statement. The instance itself acts as the external state that monitors how the instance is used.
The two required methods require themselves to be executed inside a with statement (by checking if __enter__ has been called).
Every other method can check if the required methods have been called.
The __enter__ method simply marks the object as now being in a with statement, allowing the required methods to be called.
The __exit_ method ensures that the required methods were eventually called, and resets the state of the object as being outside a context manger.
I think this is as strong a guarantee as you can enforce, short of a class that uses the inspect module to examine the script's source code looking for violations.

You could use a classmethod like this:
class Foo:
def __init__(self, flag=True):
if flag:
raise CustomError()
#classmethod
def not_raising_1(cls):
return cls(flag=False)
Thus foo = Foo() or foo = Foo(...).not_raising_1(...) would still raise the exception, but foo = Foo.not_raising_1(...) would work.

It's not possible. You could use a workaround as that one suggested by Ajay Signh Rana or chepner but I would, personally, not recommend it as it is hard to grasp when reading the code.
Your goal should be to increase readability and usability of the class for yourself and other programmers that uses this class. Use well-known patterns and concepts whenever possible and if applicable.
Reading your question, I understand that the object is not ready to use until one of the other methods is invoked. You could consider Julian Fock's answer and use a class method.
Or use any of the other Creational Design Patterns:
https://refactoring.guru/design-patterns/creational-patterns
Depending on the reason why you want to achieve this behaviour, you could consider to implement the Builder pattern:
https://refactoring.guru/design-patterns/builder/python/example
https://stackoverflow.com/a/26193004/42659
Builder pattern equivalent in Python
A third alternative would be, as you mention yourself, that you pass some parameters along when invoking the constructor and call, depending on the parameter, either of the other methods within the constructor.
Which approach is usable and applicable for your situation depends on your needs and bigger picture than the example in your Question. You should choose the approach that suits your needs best and is most readable.

I did get your question but as others suggested it cannot be done. But yeah you wann raise an exception and it should be raised if the function isn't call then you must create another function that checks if the previous functions were called or not and if not you can raise the exception.
I would approach this problem by creating a variable that changes it's value based on the funciton calls and if the functions haven't been called we can determine that as well.
Try:
class SomeError(Exception):
pass
class Foo:
def __init__(self, ...):
self.flag = False # set the flag to false for each object initially
...
pass
def not_raising_1(self, ...):
self.flag = True # set it to true once the function has been called
...
pass
def not_raising_2(self, ...):
self.flag = True # repeat for this on too
...
pass
def raise_exception(self):
if(not self.flag):
raise SomeError
obj1 = Foo()
obj1.not_raising_1()
obj1.raise_exception() # won't do anything
obj2 = Foo()
obj2.raise_exception() # would raise exception as either of the two functions weren't called

As others have suggested, it's not something that you should consider in actual code. But Just as an excercise, I tried doing something similar:
class NoError(Exception):
pass
class Foo:
def __init__(self):
pass
def not_raising_1(self):
raise NoError()
def not_raising_2(self):
raise NoError()
How to use:
try:
Foo()
raise Exception('please use either not_raising_1 or not_raising_2')
except NoError:
print('No error')
# actual code

Is returning a value other than `self` in `__enter__` an anti-pattern?

Following this related question, while there are always examples of some library using a language feature in a unique way, I was wondering whether returning a value other than self in an __enter__ method should be considered an anti-pattern.
The main reason why this seems to me like a bad idea is that it makes wrapping context managers problematic. For example, in Java (also possible in C#), one can wrap an AutoCloseable class in another class which will take care of cleaning up after the inner class, like in the following code snippet:
try (BufferedReader reader =
new BufferedReader(new FileReader("src/main/resources/input.txt"))) {
return readAllLines(reader);
}
Here, BufferedReader wraps FileReader, and calls FileReader's close() method inside its own close() method. However, if this was Python, and FileReader would've returned an object other than self in its __enter__ method, this would make such an arrangement significantly more complicated. The following issues would have to be addressed by the writer of BufferedReader:
When I need to use FileReader for my own methods, do I use FileReader directly or the object returned by its __enter__ method? What methods are even supported by the returned object?
In my __exit__ method, do I need to close only the FileReader object, or the object returned in the __enter__ method?
What happens if __enter__ actually returns a different object on its call? Do I now need to keep a collection of all of the different objects returned by it in case someone calls __enter__ several times on me? How do I know which one to use when I need to use on of these objects?
And the list goes on. One semi-successful solution to all of these problems would be to simply avoid having one context manager class clean up after another context manager class. In my example, that would mean that we would need two nested with blocks - one for the FileReader, and one for the BufferedReader. However, this makes us write more boilerplate code, and seems significantly less elegant.
All in all, these issues lead me to believe that while Python does allow us to return something other than self in the __enter__ method, this behavior should simply be avoided. Is there some official or semi-official remarks about these issues? How should a responsible Python developer write code that addresses these issues?

TLDR: Returning something other than self from __enter__ is perfectly fine and not bad practice.
The introducing PEP 343 and Context Manager specification expressly list this as desired use cases.
An example of a context manager that returns a related object is the
one returned by decimal.localcontext(). These managers set the active
decimal context to a copy of the original decimal context and then
return the copy. This allows changes to be made to the current decimal
context in the body of the with statement without affecting code
outside the with statement.
The standard library has several examples of returning something other than self from __enter__. Notably, much of contextlib matches this pattern.
contextlib.contextmanager produces context managers which cannot return self, because there is no such thing.
contextlib.closing wraps a thing and returns it on __enter__.
contextlib.nullcontext returns a pre-defined constant
threading.Lock returns a boolean
decimal.localcontext returns a copy of its argument
The context manager protocol makes it clear what is the context manager, and who is responsible for cleanup. Most importantly, the return value of __enter__ is inconsequential for the protocol.
A rough paraphrasing of the protocol is this: When something runs cm.__enter__, it is responsible for running cm.__exit__. Notably, whatever code does that has access to cm (the context manager itself); the result of cm.__enter__ is not needed to call cm.__exit__.
In other words, a code that takes (and runs) a ContextManager must run it completely. Any other code does not have to care whether its value comes from a ContextManager or not.
# entering a context manager requires closing it…
def managing(cm: ContextManager):
value = cm.__enter__() # must clean up `cm` after this point
try:
yield from unmanaged(value)
except BaseException as exc:
if not cm.__exit__(type(exc), exc, exc.__traceback__):
raise
else:
cm.__exit__(None, None, None)
# …other code does not need to know where its values come from
def unmanaged(smth: Any):
yield smth
When context managers wrap others, the same rules apply: If the outer context manager calls the inner one's __enter__, it must call its __exit__ as well. If the outer context manager already has the entered inner context manager, it is not responsible for cleanup.
In some cases it is in fact bad practice to return self from __enter__. Returning self from __enter__ should only be done if self is fully initialised beforehand; if __enter__ runs any initialisation code, a separate object should be returned.
class BadContextManager:
"""
Anti Pattern: Context manager is in inconsistent state before ``__enter__``
"""
def __init__(self, path):
self.path = path
self._file = None # BAD: initialisation not complete
def read(self, n: int):
return self._file.read(n) # fails before the context is entered!
def __enter__(self) -> 'BadContextManager':
self._file = open(self.path)
return self # BAD: self was not valid before
def __exit__(self, exc_type, exc_val, tb):
self._file.close()
class GoodContext:
def __init__(self, path):
self.path = path
self._file = None # GOOD: Inconsistent state not visible/used
def __enter__(self) -> TextIO:
if self._file is not None:
raise RuntimeError(f'{self.__class__.__name__} is not re-entrant')
self._file = open(self.path)
return self._file # GOOD: value was not accessible before
def __exit__(self, exc_type, exc_val, tb):
self._file.close()
Notably, even though GoodContext returns a different object, it is still responsible to clean up. Another context manager wrapping GoodContext does not need to close the return value, it just has to call GoodContext.__exit__.

__init__ vs __enter__ in context managers

As far as I understand, __init__() and __enter__() methods of the context manager are called exactly once each, one after another, leaving no chance for any other code to be executed in between. What is the purpose of separating them into two methods, and what should I put into each?
Edit: sorry, wasn't paying attention to the docs.
Edit 2: actually, the reason I got confused is because I was thinking of #contextmanager decorator. A context manager created using #contextmananger can only be used once (the generator will be exhausted after the first use), so often they are written with the constructor call inside with statement; and if that was the only way to use with statement, my question would have made sense. Of course, in reality, context managers are more general than what #contextmanager can create; in particular context managers can, in general, be reused. I hope I got it right this time?

As far as I understand, __init__() and __enter__() methods of the context manager are called exactly once each, one after another, leaving no chance for any other code to be executed in between.
And your understanding is incorrect. __init__ is called when the object is created, __enter__ when it is entered with with statement, and these are 2 quite distinct things. Often it is so that the constructor is directly called in with initialization, with no intervening code, but this doesn't have to be the case.
Consider this example:
class Foo:
def __init__(self):
print('__init__ called')
def __enter__(self):
print('__enter__ called')
return self
def __exit__(self, *a):
print('__exit__ called')
myobj = Foo()
print('\nabout to enter with 1')
with myobj:
print('in with 1')
print('\nabout to enter with 2')
with myobj:
print('in with 2')
myobj can be initialized separately and entered in multiple with blocks:
Output:
__init__ called
about to enter with 1
__enter__ called
in with 1
__exit__ called
about to enter with 2
__enter__ called
in with 2
__exit__ called
Furthermore if __init__ and __enter__ weren't separated, it wouldn't be possible to even use the following:
def open_etc_file(name):
return open(os.path.join('/etc', name))
with open_etc_file('passwd'):
...
since the initialization (within open) is clearly separate from with entry.
The managers created by contextlib.manager are single-entrant, but they again can be constructed outside the with block. Take the example:
from contextlib import contextmanager
#contextmanager
def tag(name):
print("<%s>" % name)
yield
print("</%s>" % name)
you can use this as:
def heading(level=1):
return tag('h{}'.format(level))
my_heading = heading()
print('Below be my heading')
with my_heading:
print('Here be dragons')
output:
Below be my heading
<h1>
Here be dragons
</h1>
However, if you try to reuse my_heading (and, consequently, tag), you will get
RuntimeError: generator didn't yield

Antti Haapalas answer is perfectly fine. I just wanted to elaborate a bit on the usage of arguments (like myClass(* args)) since that was somewhat unclear to me (retrospective I ask myself why....)
Using arguments for initialising your class in a with statement is not different from using the class the usual way.
The calls will happen in the following order:
__init__ (allocation of the class)
__enter__ (enter context)
__exit__ (leaving context)
Simple Example:
class Foo:
def __init__(self, i):
print('__init__ called: {}'.format(i))
self.i = i
def __enter__(self):
print('__enter__ called')
return self
def do_something(self):
print('do something with {}'.format(self.i))
def __exit__(self, *a):
print('__exit__ called')
with Foo(42) as bar:
bar.do_something()
Output:
__init__ called: 42
__enter__ called
do something with 42
__exit__ called
If you want to make sure that your calls can (almost) only be used in a context (e.g. to force the call to __exit__), see the stackoverflow post here. In the comments you will also find a answer to the question how to use arguments even then.

Prevent other classes' methods from calling my constructor

How do I make a python "constructor" "private", so that the objects of its class can only be created by calling static methods? I know there are no C++/Java like private methods in Python, but I'm looking for another way to prevent others from calling my constructor (or other method).
I have something like:
class Response(object):
#staticmethod
def from_xml(source):
ret = Response()
# parse xml into ret
return ret
#staticmethod
def from_json(source):
# parse json
pass
and would like the following behavior:
r = Response() # should fail
r = Response.from_json(source) # should be allowed
The reason for using static methods is that I always forget what arguments my constructors take - say JSON or an already parsed object. Even then, I sometimes forget about the static methods and call the constructor directly (not to mention other people using my code). Documenting this contract won't help with my forgetfulness. I'd rather enforce it with an assertion.
And contrary to some of the commenters, I don't think this is unpythonic - "explicit is better than implicit", and "there should be only one way to do it".
How can I get a gentle reminder when I'm doing it wrong? I'd prefer a solution where I don't have to change the static methods, just a decorator or a single line drop-in for the constructor would be great. A la:
class Response(object):
def __init__(self):
assert not called_from_outside()

I think this is what you're looking for - but it's kind of unpythonic as far as I'm concerned.
class Foo(object):
def __init__(self):
raise NotImplementedError()
def __new__(cls):
bare_instance = object.__new__(cls)
# you may want to have some common initialisation code here
return bare_instance
#classmethod
def from_whatever(cls, arg):
instance = cls.__new__(cls)
instance.arg = arg
return instance
Given your example (from_json and from_xml), I assume you're retrieving attribute values from either a json or xml source. In this case, the pythonic solution would be to have a normal initializer and call it from your alternate constructors, i.e.:
class Foo(object):
def __init__(self, arg):
self.arg = arg
#classmethod
def from_json(cls, source):
arg = get_arg_value_from_json_source(source)
return cls(arg)
#classmethod
def from_xml(cls, source):
arg = get_arg_value_from_xml_source(source)
return cls(arg)
Oh and yes, about the first example: it will prevent your class from being instantiated in the usual way (calling the class), but the client code will still be able to call on Foo.__new__(Foo), so it's really a waste of time. Also it will make unit testing harder if you cannot instantiate your class in the most ordinary way... and quite a few of us will hate you for this.

I'd recommend turning the factory methods into module-level factory functions, then hiding the class itself from users of your module.
def one_constructor(source):
return _Response(...)
def another_constructor(source):
return _Response(...)
class _Response(object):
...
You can see this approach used in modules like re, where match objects are only constructed through functions like match and search, and the documentation doesn't actually name the match object type. (At least, the 3.4 documentation doesn't. The 2.7 documentation incorrectly refers to re.MatchObject, which doesn't exist.) The match object type also resists direct construction:
>>> type(re.match('',''))()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: cannot create '_sre.SRE_Match' instances
but unfortunately, the way it does so relies upon the C API, so it's not available to ordinary Python code.

Good discussion in the comments.
For the minimal use case you describe,
class Response(object):
def __init__(self, construct_info = None):
if construct_info is None: raise ValueError, "must create instance using from_xml or from_json"
# etc
#staticmethod
def from_xml(source):
info = {} # parse info into here
return Response(info)
#staticmethod
def from_json(source):
info = {} # parse info into here
return Response(info)
It can be gotten around by a user who passes in a hand-constructed info, but at that point they'll have to read the code anyway and the static method will provide the path of least resistance. You can't stop them, but you can gently discourage them. It's Python, after all.

This might be achievable through metaclasses, but is heavily discouraged in Python. Python is not Java. There is no first-class notion of public vs private in Python; the idea is that users of the language are "consenting adults" and can use methods however they like. Generally, functions that are intended to be "private" (as in not part of the API) are denoted by a single leading underscore; however, this is mostly just convention and there's nothing stopping a user from using these functions.
In your case, the Pythonic thing to do would be to default the constructor to one of the available from_foo methods, or even to create a "smart constructor" that can find the appropriate parser for most cases. Or, add an optional keyword arg to the __init__ method that determines which parser to use.

An alternative API (and one I've seen far more in Python APIs) if you want to keep it explicit for the user would be to use keyword arguments:
class Foo(object):
def __init__(self, *, xml_source=None, json_source=None):
if xml_source and json_source:
raise ValueError("Only one source can be given.")
elif xml_source:
from_xml(xml_source)
elif json_source:
from_json(json_source)
else:
raise ValueError("One source must be given.")
Here using 3.x's * to signify keyword-only arguments, which helps enforce the explicit API. In 2.x this is recreatable with kwargs.
Naturally, this doesn't scale well to lots of arguments or options, but there are definitely cases where this style makes sense. (I'd argue bruno desthuilliers probably has it right for this case, from what we know, but I'll leave this here as an option for others).

The following is similar to what I ended up doing. It is a bit more general then what was asked in the question.
I made a function called guard_call, that checks if the current method is being called from a method of a certain class.
This has multiple uses. For example, I used the Command Pattern to implement undo and redo, and used this to ensure that my objects were only ever modified by command objects, and not random other code (which would make undo impossible).
In this concrete case, I place a guard in the constructor ensuring only Response methods can call it:
class Response(object):
def __init__(self):
guard_call([Response])
pass
#staticmethod
def from_xml(source):
ret = Response()
# parse xml into ret
return ret
For this specific case, you could probably make this a decorator and remove the argument, but I didn't do that here.
Here is the rest of the code. It's been a long time since I tested it, and can't guarentee that it works in all edge cases, so beware. It is also still Python 2. Another caveat is that it is slow, because it uses inspect. So don't use it in tight loops and when speed is an issue, but it might be useful when correctness is more important than speed.
Some day I might clean this up and release it as a library - I have a couple more of these functions, including one that asserts you are running on a particular thread. You may snear at the hackishness (it is hacky), but I did find this technique useful to smoke out some hard to find bugs, and to ensure my code still behaves during refactorings, for example.
from __future__ import print_function
import inspect
# http://stackoverflow.com/a/2220759/143091
def get_class_from_frame(fr):
args, _, _, value_dict = inspect.getargvalues(fr)
# we check the first parameter for the frame function is
# named 'self'
if len(args) and args[0] == 'self':
# in that case, 'self' will be referenced in value_dict
instance = value_dict.get('self', None)
if instance:
# return its class
return getattr(instance, '__class__', None)
# return None otherwise
return None
def guard_call(allowed_classes, level=1):
stack_info = inspect.stack()[level + 1]
frame = stack_info[0]
method = stack_info[3]
calling_class = get_class_from_frame(frame)
# print ("calling class:", calling_class)
if calling_class:
for klass in allowed_classes:
if issubclass(calling_class, klass):
return
allowed_str = ", ".join(klass.__name__ for klass in allowed_classes)
filename = stack_info[1]
line = stack_info[2]
stack_info_2 = inspect.stack()[level]
protected_method = stack_info_2[3]
protected_frame = stack_info_2[0]
protected_class = get_class_from_frame(protected_frame)
if calling_class:
origin = "%s:%s" % (calling_class.__name__, method)
else:
origin = method
print ()
print ("In %s, line %d:" % (filename, line))
print ("Warning, call to %s:%s was not made from %s, but from %s!" %
(protected_class.__name__, protected_method, allowed_str, origin))
assert False
r = Response() # should fail
r = Response.from_json("...") # should be allowed