how do you test private static method in module in python - python

we have a module of static methods in our python app. these methods use a lot of private (e.g: "__do_sub_task2(**args)") I would like to write unit tests for these private static methods within this module, but I am getting refernce errors.
is there a way to do this?
update: adding scenario
I have a module file named 'my_module.py'
contents of said file is as follows:
def public_method_foo(my_number):
return __sub_method_bar(my_number * 10)
def __sub_method_bar(other_number)
return other_number + 11
update #2
The reason I am asking this question is because I have a similar scenario as above, but when I add the following reference to my test.py module file:
from my_module import __sub_method_bar
and try to use it in my test, I get the following exception in my test
global name '_MyTests__sub_method_bar' is not defined

What you have are not methods, not private, and not static; they're just plain old public functions in the module. So you call them the same way as any other function. For your example:
>>> my_module.__sub_method_bar(5)
That's it; nothing tricky.*
* Well, there is one tricky thing, but it's probably not going to affect you here: If my_module doesn't have an __all__, and you do from my_module import *, you will not get any of the globals (including functions) whose names start with _. But normally your unit tests are going to import my_module, so this won't be relevant.
Methods are callables that are members of a class. And methods can be private ("private" in this sense means "visible only to this class, not even to super- or sub-classes", so it doesn't make sense for anything but methods). The tutorial chapter on Classes explains how private methods are implemented, with name-mangling. Methods (private or otherwise) can also be static ("static" in this context means "does not take the normal self", so again, it doesn't make sense for anything but methods). Either way, for a private method, you have to manually demangle the name to call it from outside:
>>> thingy = Thingy()
>>> thingy._Thingy__private_method(5)
>>> Thingy._Thingy__private_static_method(5)

Related

single underscore before function name in single-file python script

Should I use underscore before function name in single-file simple python script?
For example, in this script should I add underscore before definition of f and g? This script won't be imported from other files.
def f(x): # or _f(x)?
return x * 2
def g(x): # or _g(x)?
return x ** 2
def main():
x = f(100)
print(g(x))
if __name__ == "__main__":
main()
I read many documents about usage of underscores in python. Many of them says underscores are about OOP-style programming and how import statement works. However, in simple one-file script, I can't find good answer.
What is better pattern?
The most popular programming languages (such as Java and C/C++) have this kind of syntax to declare private attributes and methods of a class instance:
class MyClass {
private:
int _function() {
// Some code here
}
public:
bool _foo() {
// Some code here
}
}
Python doesn't have (and will probably never have) a syntax like this, so they simply create a conventional name to make developers understand that a method is private and should never be accessed from outside the class.
According to PEP-8:
We don't use the term "private" here, since no attribute is really private in Python (without a generally unnecessary amount of work).
According to Python 2 and Python 3 class documentation:
“Private” instance variables that cannot be accessed except from inside an object don’t exist in Python. However, there is a convention that is followed by most Python code: a name prefixed with an underscore (e.g. _spam) should be treated as a non-public part of the API (whether it is a function, a method or a data member).
Using _ may be useful for a tecnique explained below.
According to Python 3 class documentation:
Since there is a valid use-case for class-private members (namely to avoid name clashes of names with names defined by subclasses), there is limited support for such a mechanism, called name mangling.
NOTE: The Python documentation mostly talks about "internal use" referring to classes, but it can be referred to modules too, for example:
# main.py
import myModule
functionThatCanBeImported()
# myModule.py
def functionThatCanBeImported(): pass
def _functionThatShouldNeverBeImported(): pass
If somebody creates a package, they don't have to put private functions in the documentation, since they are for internal scope and explainations about them could be useful only to developers.
_ is used to let developers know that the variables and methods are private and should not be modified externally. So use _ if you want to keep the functionality within the class.
NOTE: using _ will not restrict the use of these methods or variables though, it's just a way of representation.

when to use "_(.)" kind of things in python.? [duplicate]

How can I make methods and data members private in Python? Or doesn't Python support private members?
9.6. Private Variables
“Private” instance variables that
cannot be accessed except from inside
an object, don’t exist in Python.
However, there is a convention that is
followed by most Python code: a name
prefixed with an underscore (e.g.
_spam) should be treated as a non-public part of the API (whether it
is a function, a method or a data
member). It should be considered an
implementation detail and subject to
change without notice.
Since there is a valid use-case for
class-private members (namely to avoid
name clashes of names with names
defined by subclasses), there is
limited support for such a mechanism,
called name mangling. Any identifier
of the form __spam (at least two
leading underscores, at most one
trailing underscore) is textually
replaced with _classname__spam, where
classname is the current class name
with leading underscore(s) stripped.
This mangling is done without regard
to the syntactic position of the
identifier, as long as it occurs
within the definition of a class.
So, for example,
class Test:
def __private_symbol(self):
pass
def normal_symbol(self):
pass
print dir(Test)
will output:
['_Test__private_symbol',
'__doc__',
'__module__',
'normal_symbol']
__private_symbol should be considered a private method, but it would still be accessible through _Test__private_symbol.
The other answers provide the technical details. I'd like to emphasise the difference in philosophy between Python on one hand and languages like C++/Java (which I presume you're familiar with based on your question).
The general attitude in Python (and Perl for that matter) is that the 'privacy' of an attribute is a request to the programmer rather than a barbed wire fence by the compiler/interpreter. The idea is summarised well in this mail and is often referred to as "We're all consenting adults" since it 'assumes' that the programmer is responsible enough to not meddle with the insides. The leading underscores serve as a polite message saying that the attribute is internal.
On the other hand, if you do want to access the internals for some applications (a notable example is documentation generators like pydoc), you're free to do so. Onus is on you as a programmer to know what you're doing and do it properly rather than on the language to force you do to things it's way.
There are no private of any other access protection mechanisms in Python. There is a convention documented in the Python style guide for indicating to the users of your your class that they should not be accessing certain attribute.
_single_leading_underscore: weak "internal use" indicator. E.g. from M import * does not import objects whose name starts with an underscore.
single_trailing_underscore_: used by convention to avoid conflicts with Python keyword, e.g. Tkinter.Toplevel(master, class_='ClassName')
__double_leading_underscore: when naming a class attribute, invokes name mangling (inside class FooBar, __boo becomes _FooBar__boo; see below).
If the name of a Python function,
class method, or attribute starts with
(but doesn't end with) two
underscores, it's private; everything
else is public. Python has no concept
of protected class methods (accessible
only in their own class and descendant
classes). Class methods are either
private (accessible only in their own
class) or public (accessible from
anywhere).
Dive Into Python
Python does not support privacy directly . Programmer need to know when it is safe to modify attribute from outside but anyway with python you can achieve something like private with little tricks.
Now let's see a person can put anything private to it or not.
class Person(object):
def __priva(self):
print "I am Private"
def publ(self):
print " I am public"
def callpriva(self):
self.__priva()
Now When we will execute :
>>> p = Person()
>>> p.publ()
I am public
>>> p.__priva()
Traceback (most recent call last):
File "", line 1, in
p.__priva()
AttributeError: 'Person' object has no attribute '__priva'
​#Explanation : You can see here we are not able to fetch that private method directly.
>>> p.callpriva()
I am Private
#​Explanation : Here we can access private method inside class​
​Then how someone can access that variable ???
You can do like :
>>> p._Person__priva
I am Private
​wow , actually if python is getting any variable starting with double underscore are “translated” by adding a single underscore and the class name to the beginning:
Note : If you do not want this name changing but you still want to send a signal for other objects to stay away, you can use a single initial underscore names with an initial underscore aren’t imported with starred imports (from module import *)
Example :
#test.py
def hello():
print "hello"
def _hello():
print "Hello private"
#----------------------
#test2.py
from test import *
print hello()
print _hello()
output-->
hello
Traceback (most recent call last):
File "", line 1, in
NameError: name '_hello' is not defined
Now if we will call _hello manually .
#test2.py
from test import _hello , hello
print hello()
print _hello()
output-->
hello
hello private
Finally : Python doesn’t really have an equivalent privacy support, although single
and double initial underscores do to some extent give you two levels of privacy
This might work:
import sys, functools
def private(member):
#functools.wraps(member)
def wrapper(*function_args):
myself = member.__name__
caller = sys._getframe(1).f_code.co_name
if (not caller in dir(function_args[0]) and not caller is myself):
raise Exception("%s called by %s is private"%(myself,caller))
return member(*function_args)
return wrapper
class test:
def public_method(self):
print('public method called')
#private
def private_method(self):
print('private method called')
t = test()
t.public_method()
t.private_method()
This is kinda a l-o-n-g answer but I think it gets to the root of the real problem here -- scope of visibility. Just hang in there while I slog through this!
Simply importing a module need not necessarily give the application developer access to all of its classes or methods; if I can't actually SEE the module source code how will I know what's available? Some one (or some THING) has to tell me what I can do and explain how to use those features I'm allowed to use, otherwise the whole thing is useless to me.
Those developing higher-level abstractions based on fundamental classes and methods via imported modules are presented with a specification DOCUMENT -- NOT the actual source code.
The module spec describes all the features intended to be visible to the client developer. When dealing with large projects and software project teams, the actual implementation of a module should ALWAYS remain hidden from those using it -- it's a blackbox with an interface to the outside world. For OOD purists, I believe the techie terms are "decoupling" and "coherence". The module user need only know the interface methods without being burden with the details of implementation.
A module should NEVER be changed without first changing its underlying spec document, which may require review / approval in some organizations prior to changing the code.
As hobby programmer (retired now), I start a new module with the spec doc actually written out as a giant comment block at the top of the module, this will be the part the user actually sees in the spec library. Since it's just me, I've yet to set up a library, but it would be easy enough to do.
Then I begin coding by writing the various classes and methods but without functional bodies -- just null print statements like "print()" -- just enough to allow the module to compile without syntax errors. When this step is complete I compile the completed null-module -- this is my spec. If I were working on a project team, I would present this spec/interface for review & commentary before proceeding with fleshing out the body.
I flesh out the bodies of each method one at a time and compile accordingly, ensuring syntax errors are fixed immediately on-the-fly. This is also a good time to start writing a temporary "main" execution section at the bottom to test each method as you code it. When the coding/testing are complete, all of the test code is commented out until you need it again should updates become necessary.
In a real-world development team, the spec comment block would also appear in a document control library, but that's another story. The point is: you, as the module client, see only this spec and NOT the source code.
PS: long before the beginning of time, I worked in the defense aerospace community and we did some pretty cool stuff, but things like proprietary algorithms and sensitive systems control logic were tightly vaulted and encrypted in super-duper secure software libraries. We had access to module / package interfaces but NOT the blackbox implementation bodies. There was a document management tool that handled all system-level designs, software specs, source code and test records -- it was all synched together. The government had strict requirements software quality assurance standards. Anyone remember a language called "Ada"? That's how old I am!
import inspect
class Number:
def __init__(self, value):
self.my_private = value
def set_private(self, value):
self.my_private = value
def __setattr__(self, my_private, value):
f = inspect.stack()[1][3]
if f not in ['__init__', 'set_private']:
raise Exception("can't access private member-my_private")
# the default behavior
self.__dict__[my_private] = value
def main():
n = Number(2)
print(n.my_private)
n.set_private(3)
print(n.my_private)
if __name__ == '__main__':
main()
I use Python 2.7 and 3.5. I wrote this code:
class MyOBject(object):
def __init__(self):
self.__private_field = 10
my_object = MyOBject()
print(my_object.__private_field)
ran it and got:
AttributeError: 'MyOBject' object has no attribute '__private_field'
Please see:
https://www.tutorialsteacher.com/python/private-and-protected-access-modifiers-in-python

Using module as a singleton in Python - is that ok?

I've got a really complex singleton object. I've decided to modify it, so it'll be a separate module with module--wide global variables that would store data.
Are there some pitfalls of this approach? I just feel, like that's a little bit hacky, and that there may be some problems I cannot see now.
Maybe someone did this or have some opinion :) Thanks in advance for help.
Regards.
// Minimal, Complete, and Verifiable example:
"""
This is __init__.py of the module, that could be used as a singleton:
I need to set and get value of IMPORTANT_VARIABLE from different places in my code.
Folder structure:
--singleton_module
|
-__init__.py
Example of usage:
import singleton_module as my_singleton
my_singleton.set_important_variable(3)
print(my_singleton.get_important_variable())
"""
IMPORTANT_VARIABLE = 0
def set_important_variable(value):
global IMPORTANT_VARIABLE
IMPORTANT_VARIABLE = value
def get_important_variable():
return IMPORTANT_VARIABLE
Technically, Python modules ARE singletons, so from this point of view there's no particular issue (except the usual issues with singletons that is) with your code. I'd just spell the varibale in all_lower (ALL_UPPER denotes a pseudo-constant) and prefix it with either a single ("protected") or double ("really private") leading underscore to make clear it's not part of the public API (standard Python naming convention).
Now whether singletons are a good idea is another debate but that's not the point here...
e.g that in one potential situation I may lost data, or that module could be imported in different places of code two times, so it would not be a singleton if imported inside scope of function or something like that.
A module is only instanciated once per process (the first time it's imported), then subsquent imports will directly get if from sys.modules. The only case where you could have two distinct instances of the same module is when the module is imported by two different path, which can only happens if you have a somewhat broken sys.path ie something like this:
src/
foo/
__init.py
bar/
__init__.py
baaz/
__init__.py
mymodule.py
with both "src" and "foo" in sys.path, then importing mymodule once as from foo.bar.baaz import mymodule and a second time as from bar.baaz import mymodule
Needless to say that it's a degenerate case, but it can happens and lead to hard to diagnose bugs. Note that when you have this case, you do have quite a few other things that breaks, like identity testing anything from mymodule.
Also, I am not sure how would using object instead of module increase security
It doesn't.
And I am just asking, if that's not a bad practice, maybe someone did this and found some problems. This is probably not a popular pattern
Well, quite on the contrary you'll often find advises on using modules as singletons instead of using classes with only staticmethods, classmethods and class attributes (another way of implementing a singleton in Python). This most often concerns stateless classes used as namespaces while your example does have a state, but this doesn't make much practical difference.
Now what you won't get are all the nice OO features like computed attributes, inheritance, magicmethods etc, but I assume you already understood this.
As far as I'm concerned, depending on the context, I might rather use a plain class but only expose one single instance of the class as the module's API ie:
# mymodule.py
__all__ = ["mysingleton"]
class __MySingletonLike(object):
def __init__(self):
self._variable = 42
#property
def variable(self):
return self._variable
#variable.setter
def variable(self, value):
check_value(value) # imaginary validation
self._variable = value
mysingleton = __MySingleton()
but that's only when I have special concerns about the class (implementation reuse, proper testability, other special features requiring a class etc).

Private functions in python

Is it possible to avoid importing a file with from file import function?
Someone told me i would need to put an underscore as prefix, like: _function, but isn't working.
I'm using Python 2.6 because of a legacy code.
There are ways you can prevent the import, but they're generally hacks and you want to avoid them. The normal method is to just use the underscore:
def _function():
pass
Then, when you import,
from my_module import *
You'll notice that _function is not imported because it begins with an underscore. However, you can always do this:
# In Python culture, this is considered rude
from my_module import _function
But you're not supposed to do that. Just don't do that, and you'll be fine. Python's attitude is that we're all adults. There are a lot of other things you're not supposed to do which are far worse, like
import my_module
# Remove the definition for _function!
del my_module._function
There is no privacy in Python. There are only conventions governing what external code should consider publicly accessible and usable.
Importing a module for the first time, triggers the creation of a module object and the execution of all top-level code in the module. The module object contains the global namespace with the result of that code having run.
Because Python is dynamic you can always introspect the module namespace; you can see all names defined, all objects those names reference, and you can access and alter everything. It doesn't matter here if those names start with underscores or not.
So the only reason you use a leading _ underscore for a name, is to document that the name is internal to the implementation of the module, and that external code should not rely on that name existing in a future version. The from module import * syntax will ignore such names for that reason alone. But you can't prevent a determined programmer from accessing such a name anyway. They simply do so at their own risk, it is not your responsibility to keep them from that access.
If you have functions or other objects that are only needed to initialise the module, you are of course free to delete those names at the end.

Python: 'Private' module in a package

I have a package mypack with modules mod_a and mod_b in it. I intend the package itself and mod_a to be imported freely:
import mypack
import mypack.mod_a
However, I'd like to keep mod_b for the exclusive use of mypack. That's because it exists merely to organize the latter's internal code.
My first question is, is it an accepted practice in Python programming to have 'private' modules like this?
If yes, my second question is, what is the best way to convey this intention to the client? Do I prefix the name with an underscore (i.e. _mod_b)? Or would it be a good idea to declare a sub-package private and place all such modules there?
I prefix private modules with an underscore to communicate the intent to the user. In your case, this would be mypack._mod_b
This is in the same spirit (but not completely analogous to) the PEP8 recommendation to name C-extension modules with a leading underscore when it’s wrapped by a Python module; i.e., _socket and socket.
The solution I've settled on is to create a sub-package 'private' and place all the modules I wish to hide in there. This way they stay stowed away, leaving mypack's module list cleaner and easier to parse.
To me, this doesn't look unpythonic either.
While there are not explicit private keywords there is a convention to have put private functions start with a single underscore but a double leading underscore will make it so others cannot easily call the function from outside the module. See the following from PEP 8
- _single_leading_underscore: weak "internal use" indicator. E.g. "from M
import *" does not import objects whose name starts with an underscore.
- single_trailing_underscore_: used by convention to avoid conflicts with
Python keyword, e.g.
Tkinter.Toplevel(master, class_='ClassName')
- __double_leading_underscore: when naming a class attribute, invokes name
mangling (inside class FooBar, __boo becomes _FooBar__boo; see below).
- __double_leading_and_trailing_underscore__: "magic" objects or
attributes that live in user-controlled namespaces. E.g. __init__,
__import__ or __file__. Never invent such names; only use them
as documented.
To make an entire module private, don't include it __init__.py file.
One thing to be aware of in this scenario is indirect imports. If in mypack you
from mypack._mod_b import foo
foo()
Then a user can
from mypack import foo
foo()
and be none the wiser. I recommend importing as
from mypack import _mod_b
_mod_b.foo()
then a user will immediately see a red flag when they try to
from mypack import _mod_b
As for actual directory structure, you could even extend Jeremy's answer into a _package_of_this_kind package, where anything in that can have any 'access modifiers' on it you like - users will know there be dragons
Python doesn't strictly know or support "private" or "protected" methods or classes. There's a convention that methods prefixed with a single underscore aren't part of an official API, but I wouldn't do this on classes or files - it's ugly.
If someone really needs to subclass or access mod_b, why prevent him/her from doing so? You can always supply a preferred API in your documentation and document in your module that you shouldn't access it directly and use mypack in stead.

Categories