Reading the Django source code I came upon this function. It's the implementation for the block tag.
What interests me is that they're setting a variable with two leading underscores (__loaded_blocks) from outside the parser class instance (parser is an instance of the Parser class). A quick grep in the Django source code shows that the string loaded_blocks occurs only here.
Now I've never considered this use of the python name-mangling feature before, but this will in effect hide the __loaded_blocks attribute of parser from itself! To read this attribute from a parser method you have to resort to getattr(self, "__loaded_blocks").
Am I right in thinking this is just an unintended and unused side effect of the chosen attribute name? Or is there a deeper purpose to this?
In general, why would you want to do such a thing?
EDIT: To clarify, I'm fully aware that as long as you don't try to access the __loaded_blocks attribute from a method of parser, it will work just like any other attribute, and that it is in fact not a mangled attribute.
I don't think Name mangling will take place when you add a property prefixed with __ to an instance
from the docs:
Private name mangling: When an identifier that textually occurs in a
class definition begins with two or more underscore characters and
does not end in two or more underscores, it is considered a private
name of that class. Private names are transformed to a longer form
before code is generated for them. The transformation inserts the
class name in front of the name, with leading underscores removed, and
a single underscore inserted in front of the class name. For example,
the identifier __spam occurring in a class named Ham will be
transformed to Ham_spam. This transformation is independent of the
syntactical context in which the identifier is used. If the
transformed name is extremely long (longer than 255 characters),
implementation defined truncation may happen. If the class name
consists only of underscores, no transformation is done.
class Test:
pass
test = Test()
test.__hello = 'hii'
test.__hello # hiii
Although the name is not mangled it still marks this as "private" to the consumer of the code
Related
I would like to have a function in my class, which I am going to use only inside methods of this class. I will not call it outside the implementations of these methods. In C++, I would use a method declared in the private section of the class. What is the best way to implement such a function in Python?
I am thinking of using a static decorator for this case. Can I use a function without any decorators and the self word?
Python doesn't have the concept of private methods or attributes. It's all about how you implement your class. But you can use pseudo-private variables (name mangling); any variable preceded by __(two underscores) becomes a pseudo-private variable.
From the documentation:
Since there is a valid use-case for class-private members (namely to
avoid name clashes of names with names defined by subclasses), there
is limited support for such a mechanism, called name mangling. Any
identifier of the form __spam (at least two leading underscores, at
most one trailing underscore) is textually replaced with
_classname__spam, where classname is the current class name with leading underscore(s) stripped. This mangling is done without regard
to the syntactic position of the identifier, as long as it occurs
within the definition of a class.
class A:
def __private(self):
pass
So __private now actually becomes _A__private.
Example of a static method:
>>> class A:
... #staticmethod # Not required in Python 3.x
... def __private():
... print 'hello'
...
>>> A._A__private()
hello
Python doesn't have the concept of 'private' the way many other languages do. It is built on the consenting adult principle that says that users of your code will use it responsibly. By convention, attributes starting with a single or double leading underscore will be treated as part of the internal implementation, but they are not actually hidden from users. Double underscore will cause name mangling of the attribute name though.
Also, note that self is only special by convention, not by any feature of the language. Instance methods, when called as members of an instance, are implicitly passed the instance as a first argument, but in the implementation of the method itself, that argument can technically be named any arbitrary thing you want. self is just the convention for ease of understanding code. As a result, not including self in the signature of a method has no actual functional effect other than causing the implicit instance argument to be assigned to the next variable name in the signature.
This is of course different for class methods, which receive the instance of the class object itself as an implicit first argument, and static methods, which receive no implicit arguments at all.
Python just doesn't do private. If you like you can follow convention and precede the name with a single underscore, but it's up to other coders to respect that in a gentlemanly† fashion
† or gentlewomanly
There is plenty of great stuff here with obfuscation using leading underscores. Personally, I benefit greatly from the language design decision to make everything public as it reduces the time it takes to understand and use new modules.
However, if you're determined to implement private attributes/methods and you're willing to be unpythonic, you could do something along the lines of:
from pprint import pprint
# CamelCase because it 'acts' like a class
def SneakyCounter():
class SneakyCounterInternal(object):
def __init__(self):
self.counter = 0
def add_two(self):
self.increment()
self.increment()
def increment(self):
self.counter += 1
def reset(self):
print 'count prior to reset: {}'.format(self.counter)
self.counter = 0
sneaky_counter = SneakyCounterInternal()
class SneakyCounterExternal(object):
def add_two(self):
sneaky_counter.add_two()
def reset(self):
sneaky_counter.reset()
return SneakyCounterExternal()
# counter attribute is not accessible from out here
sneaky_counter = SneakyCounter()
sneaky_counter.add_two()
sneaky_counter.add_two()
sneaky_counter.reset()
# `increment` and `counter` not exposed (AFAIK)
pprint(dir(sneaky_counter))
It is hard to imagine a case where you'd want to do this, but it is possible.
You just don't do it:
The Pythonic way is to not document those methods/members using docstrings, only with "real" code comments. And the convention is to append a single or a double underscore to them;
Then you can use double underscores in front of your member, so they are made local to the class (it's mostly name mangling, i.e., the real name of the member outside of the class becomes: instance.__classname_membername). It's useful to avoid conflicts when using inheritance, or create a "private space" between children of a class.
As far as I can tell, it is possible to "hide" variables using metaclasses, but that violates the whole philosophy of Python, so I won't go into details about that.
I know, there are no 'real' private/protected methods in Python. This approach isn't meant to hide anything; I just want to understand what Python does.
class Parent(object):
def _protected(self):
pass
def __private(self):
pass
class Child(Parent):
def foo(self):
self._protected() # This works
def bar(self):
self.__private() # This doesn't work, I get a AttributeError:
# 'Child' object has no attribute '_Child__private'
So, does this behaviour mean, that 'protected' methods will be inherited but 'private' won't at all?
Or did I miss anything?
Python has no privacy model, there are no access modifiers like in C++, C# or Java. There are no truly 'protected' or 'private' attributes.
Names with a leading double underscore and no trailing double underscore are mangled to protect them from clashes when inherited. Subclasses can define their own __private() method and these will not interfere with the same name on the parent class. Such names are considered class private; they are still accessible from outside the class but are far less likely to accidentally clash.
Mangling is done by prepending any such name with an extra underscore and the class name (regardless of how the name is used or if it exists), effectively giving them a namespace. In the Parent class, any __private identifier is replaced (at compilation time) by the name _Parent__private, while in the Child class the identifier is replaced by _Child__private, everywhere in the class definition.
The following will work:
class Child(Parent):
def foo(self):
self._protected()
def bar(self):
self._Parent__private()
See Reserved classes of identifiers in the lexical analysis documentation:
__*
Class-private names. Names in this category, when used within the context of a class definition, are re-written to use a mangled form to help avoid name clashes between “private” attributes of base and derived classes.
and the referenced documentation on names:
Private name mangling: When an identifier that textually occurs in a class definition begins with two or more underscore characters and does not end in two or more underscores, it is considered a private name of that class. Private names are transformed to a longer form before code is generated for them. The transformation inserts the class name, with leading underscores removed and a single underscore inserted, in front of the name. For example, the identifier __spam occurring in a class named Ham will be transformed to _Ham__spam. This transformation is independent of the syntactical context in which the identifier is used.
Don't use class-private names unless you specifically want to avoid having to tell developers that want to subclass your class that they can't use certain names or risk breaking your class. Outside of published frameworks and libraries, there is little use for this feature.
The PEP 8 Python Style Guide has this to say about private name mangling:
If your class is intended to be subclassed, and you have attributes
that you do not want subclasses to use, consider naming them with
double leading underscores and no trailing underscores. This invokes
Python's name mangling algorithm, where the name of the class is
mangled into the attribute name. This helps avoid attribute name
collisions should subclasses inadvertently contain attributes with the
same name.
Note 1: Note that only the simple class name is used in the mangled
name, so if a subclass chooses both the same class name and attribute
name, you can still get name collisions.
Note 2: Name mangling can make certain uses, such as debugging and
__getattr__(), less convenient. However the name mangling algorithm
is well documented and easy to perform manually.
Note 3: Not everyone likes name mangling. Try to balance the need to
avoid accidental name clashes with potential use by advanced callers.
The double __ attribute is changed to _ClassName__method_name which makes it more private than the semantic privacy implied by _method_name.
You can technically still get at it if you'd really like to, but presumably no one is going to do that, so for maintenance of code abstraction reasons, the method might as well be private at that point.
class Parent(object):
def _protected(self):
pass
def __private(self):
print("Is it really private?")
class Child(Parent):
def foo(self):
self._protected()
def bar(self):
self.__private()
c = Child()
c._Parent__private()
This has the additional upside (or some would say primary upside) of allowing a method to not collide with child class method names.
By declaring your data member private :
__private()
you simply can't access it from outside the class
Python supports a technique called name mangling.
This feature turns class member prefixed with two underscores into:
_className.memberName
if you want to access it from Child() you can use: self._Parent__private()
Also PEP8 says
Use one leading underscore only for non-public methods and instance
variables.
To avoid name clashes with subclasses, use two leading underscores to
invoke Python's name mangling rules.
Python mangles these names with the class name: if class Foo has an
attribute named __a, it cannot be accessed by Foo.__a. (An insistent
user could still gain access by calling Foo._Foo__a.) Generally,
double leading underscores should be used only to avoid name conflicts
with attributes in classes designed to be subclassed.
You should stay away from _such_methods too, by convention. I mean you should treat them as private
Although this is an old question, I encountered it and found a nice workaround.
In the case you name mangled on the parent class because you wanted to mimic a protected function, but still wanted to access the function in an easy manner on the child class.
parent_class_private_func_list = [func for func in dir(Child) if func.startswith ('_Parent__')]
for parent_private_func in parent_class_private_func_list:
setattr(self, parent_private_func.replace("_Parent__", "_Child"), getattr(self, parent_private_func))
The idea is manually replacing the parents function name into one fitting to the current namespace.
After adding this in the init function of the child class, you can call the function in an easy manner.
self.__private()
AFAIK, in the second case Python perform "name mangling", so the name of the __private method of the parent class is really:
_Parent__private
And you cannot use it in child in this form neither
I am trying to understand more about the scopes of python variables.
As of now, I do not want to break or violate encapsulation on variables that are declared to be private i.e., "self._variable".
I was wondering whether if it would be breaking encapsulation if a child class directly calls a variable from its parent class. For example:
class Parent:
def __init__():
self._randomVariable = ''
class Child(Parent):
def__init__():
super().__init__()
def doSomething():
self._randomVariable = 'Test'
Does Chid.doSomething() technically break encapsulation for directly calling self._randomVariable in its method even if it is a child class?
I couldn't find anything that was Python specific about encapsulation but rather stuff based on Java. Is it the same idea between Java and Python?
Encapsulation is not as big of a deal in Python as it is in most other languages (Java, C++, et cetera), and you really shouldn't worry about it too much. In the Python community, we have this principle that "we are all consenting adults here".
What this means is that it's on your responsibility if you go and mess around with someone else's code, but also don't prevent others from messing with your code if they really know what they're doing. For this reason, there isn't really private and protected in Python, and you shouldn't worry about them the same way you do in Java.
But as it's become clear by now, there is still some sort of privacy with underscores. So, what are they usually used for?
Single underscore prefixed variables (e.g. self._var)
These are used for both private and protected variables. Prefixing your variable with an underscore is (mostly) a convention, which simply tells the reader that "this variable is used internally by this class, and should not be accessed from the outside". Well, if your subclasses need it, they may still use it. And if you need it from outside of the class, you may still use it. But it's on your responsibility, make sure you don't break anything.
There are some other minor effects too, such as from module import * not importing underscore prefixed variables, but the convention of privacy is the main point.
Double underscore prefixed variables (e.g. self.__var)
Also known as "dunder" (double-under) variables, these are used for name mangling. The idea is that if you have a common variable name and you're afraid that subclasses might use the same variable name for their internal stuff, you can use double underscore prefix to secure your variable from being overwritten accidentally. This way your self.__var becomes self._BaseClassName__var, and your subclass's self.__var becomes self._SubClassName__var. Now they won't overlap.
And while the effect can be used to simulate other languages' private, I recommend you not to. Again, we are all consenting adults here, just mark your variable "private" with a single underscore, and I won't touch it unless I really know what I'm doing.
First of all, let me change your program a little bit to fix some issues :
class Parent:
def __init__(self):
self.__randomVariable = 'Test1'
class Child(Parent):
def __init__(self):
super().__init__()
def doSomething(self):
self.__randomVariable = 'Test2'
I added self to the methods as first arguments since that is required when you define a method.
I changed the assignment in the parent class to 'Test1' instead of '' just for the sake of the example.
I also used __randomVariable with double underscores, since your question is about private variables, and private variables require two underscores. In Python private variables are not really private, but a mechanism of "name mangling" is used to turn them in a variable named _Parent.__randomVariable or _Child.__randomVariable depending on the class in which they are being declared.
Regarding the scope. If you would refer to self.__randomVariable it would refer only to the private variable named __randomVariable as defined in that class. In fact in your case there would be two different such private variables. There would be the one defined in the superclass, which because of the mechanism of "name mangling" in fact gets stored as _Parent.__randomVariable. And there is the one defined in the subclass, which gets stored as _Child.__randomVariable. So in your example, your child class is in fact NOT accessing the variable from its parent class; it instead defines a new private variable for the instance of the child class.
Here is some sample code to illustrate what would happen if you use the above definitions:
c = Child()
print(c._Parent__randomVariable)
# prints Test1, i.e. the value assigned in the parent class
c.doSomething()
# calling doSomething will execute the assignment in the subclass
print(c._Parent__randomVariable)
# still prints Test1, i.e. the private variable in the parent class
# did not get reassigned in the method doSomething
print(c._Child__randomVariable)
# This one prints Test2, so in fact what happened is that a new
# private attribute was created in the subclass.
Hence, in this piece of code there is no "breach of encapsulation" since it creates different private instance variables in the parent and the child class. That being said, private instance variables are not really "encapsulated" since you can always access them (even though you are not supposed to), when you know the mechanism of name mangling. This is what I did in the above code sample: I accessed the private variable __randomVariable of the parent class and of the subclass by writing c._Parent__randomVariablerespectively c._Child__randomVariable. Its just an implementation trick Python uses to simulate private variables.
I'm following this link and trying to make a singleton class using Metaclass. But, I want to make some internal tweaks to this singleton class and want the users to use another class (let's call it MySingleton(__Singleton)). So I decided to make it private but it gives the following error.
My sole purpose is to prevent __Singleton from being used outside. How can I achieve this?
On a separate note, is it a good practice to use double underscore with classes?
Inside the class, the identifier __Singleton is getting mangled. You end up having problems because name mangling only happens inside classes (not outside). So __Singleton as a class name means something different than __Singleton when you are inside a class suite.
Any identifier of the form __spam (at least two leading underscores, at most one trailing underscore) is textually replaced with _classname__spam, where classname is the current class name with leading underscore(s) stripped. This mangling is done without regard to the syntactic position of the identifier, as long as it occurs within the definition of a class.
Note that the primary reason for mangling is because it
... is helpful for letting subclasses override methods without breaking intraclass method calls.
Also:
... to avoid name clashes of names with names defined by subclasses
As such, there really isn't any reason to have a class with leading double underscores in the name (there is no chance of intraclass method calls having conflicts with class names). A single leading underscore is a good enough signal to users that they shouldn't use that class:
... a name prefixed with an underscore (e.g. _spam) should be treated as a non-public part of the API (whether it is a function, a method or a data member). It should be considered an implementation detail and subject to change without notice.
I wouldn't advise it, but if you really want it to work, you can probably use globals to look up the class:
class __Foo(object):
def __init__(self):
super(globals()['__Foo'], self).__init__()
f = __Foo()
print f
Every name inside a class definition with two leading underscores is mangled, so __Singleton becomes _Singleton__Singleton. To make clear, that some class is not supposed to be used publicly use one underscore.
Python does not have private variables; they are all accessible externally.
“Private” instance variables that cannot be accessed except from inside an object don’t exist in Python. However, there is a convention that is followed by most Python code: a name prefixed with an underscore (e.g. _spam) should be treated as a non-public part of the API (whether it is a function, a method or a data member). It should be considered an implementation detail and subject to change without notice.
The Python Cookbook provides a Singleton class that can be inherited by other classes to become Singletons.
I know, there are no 'real' private/protected methods in Python. This approach isn't meant to hide anything; I just want to understand what Python does.
class Parent(object):
def _protected(self):
pass
def __private(self):
pass
class Child(Parent):
def foo(self):
self._protected() # This works
def bar(self):
self.__private() # This doesn't work, I get a AttributeError:
# 'Child' object has no attribute '_Child__private'
So, does this behaviour mean, that 'protected' methods will be inherited but 'private' won't at all?
Or did I miss anything?
Python has no privacy model, there are no access modifiers like in C++, C# or Java. There are no truly 'protected' or 'private' attributes.
Names with a leading double underscore and no trailing double underscore are mangled to protect them from clashes when inherited. Subclasses can define their own __private() method and these will not interfere with the same name on the parent class. Such names are considered class private; they are still accessible from outside the class but are far less likely to accidentally clash.
Mangling is done by prepending any such name with an extra underscore and the class name (regardless of how the name is used or if it exists), effectively giving them a namespace. In the Parent class, any __private identifier is replaced (at compilation time) by the name _Parent__private, while in the Child class the identifier is replaced by _Child__private, everywhere in the class definition.
The following will work:
class Child(Parent):
def foo(self):
self._protected()
def bar(self):
self._Parent__private()
See Reserved classes of identifiers in the lexical analysis documentation:
__*
Class-private names. Names in this category, when used within the context of a class definition, are re-written to use a mangled form to help avoid name clashes between “private” attributes of base and derived classes.
and the referenced documentation on names:
Private name mangling: When an identifier that textually occurs in a class definition begins with two or more underscore characters and does not end in two or more underscores, it is considered a private name of that class. Private names are transformed to a longer form before code is generated for them. The transformation inserts the class name, with leading underscores removed and a single underscore inserted, in front of the name. For example, the identifier __spam occurring in a class named Ham will be transformed to _Ham__spam. This transformation is independent of the syntactical context in which the identifier is used.
Don't use class-private names unless you specifically want to avoid having to tell developers that want to subclass your class that they can't use certain names or risk breaking your class. Outside of published frameworks and libraries, there is little use for this feature.
The PEP 8 Python Style Guide has this to say about private name mangling:
If your class is intended to be subclassed, and you have attributes
that you do not want subclasses to use, consider naming them with
double leading underscores and no trailing underscores. This invokes
Python's name mangling algorithm, where the name of the class is
mangled into the attribute name. This helps avoid attribute name
collisions should subclasses inadvertently contain attributes with the
same name.
Note 1: Note that only the simple class name is used in the mangled
name, so if a subclass chooses both the same class name and attribute
name, you can still get name collisions.
Note 2: Name mangling can make certain uses, such as debugging and
__getattr__(), less convenient. However the name mangling algorithm
is well documented and easy to perform manually.
Note 3: Not everyone likes name mangling. Try to balance the need to
avoid accidental name clashes with potential use by advanced callers.
The double __ attribute is changed to _ClassName__method_name which makes it more private than the semantic privacy implied by _method_name.
You can technically still get at it if you'd really like to, but presumably no one is going to do that, so for maintenance of code abstraction reasons, the method might as well be private at that point.
class Parent(object):
def _protected(self):
pass
def __private(self):
print("Is it really private?")
class Child(Parent):
def foo(self):
self._protected()
def bar(self):
self.__private()
c = Child()
c._Parent__private()
This has the additional upside (or some would say primary upside) of allowing a method to not collide with child class method names.
By declaring your data member private :
__private()
you simply can't access it from outside the class
Python supports a technique called name mangling.
This feature turns class member prefixed with two underscores into:
_className.memberName
if you want to access it from Child() you can use: self._Parent__private()
Also PEP8 says
Use one leading underscore only for non-public methods and instance
variables.
To avoid name clashes with subclasses, use two leading underscores to
invoke Python's name mangling rules.
Python mangles these names with the class name: if class Foo has an
attribute named __a, it cannot be accessed by Foo.__a. (An insistent
user could still gain access by calling Foo._Foo__a.) Generally,
double leading underscores should be used only to avoid name conflicts
with attributes in classes designed to be subclassed.
You should stay away from _such_methods too, by convention. I mean you should treat them as private
Although this is an old question, I encountered it and found a nice workaround.
In the case you name mangled on the parent class because you wanted to mimic a protected function, but still wanted to access the function in an easy manner on the child class.
parent_class_private_func_list = [func for func in dir(Child) if func.startswith ('_Parent__')]
for parent_private_func in parent_class_private_func_list:
setattr(self, parent_private_func.replace("_Parent__", "_Child"), getattr(self, parent_private_func))
The idea is manually replacing the parents function name into one fitting to the current namespace.
After adding this in the init function of the child class, you can call the function in an easy manner.
self.__private()
AFAIK, in the second case Python perform "name mangling", so the name of the __private method of the parent class is really:
_Parent__private
And you cannot use it in child in this form neither