We Keep Coding

Does __all__ have to be a list? [duplicate]

In many large projects, even in such as Django, and in official Python documentation use list to list the "available from outside" module components in the __init__.py file:
__all__ = [foo, bar, other]
However, the record
__all__ = (foo, bar, other)
it will also work, and in theory it does not give a significant, but an increase in code performance.
Why, then use it to list?
Maybe there is some magic PEP that I don't know about?

There is no binding reason to use either list or tuple. However, list idiomatically represents a sequence of same kind of items but tuple represents a sequence of different kind of items. This is also encoded by type hints, which have some list[str | int] but positional fields inside tuple[str, int, int].
As such, list more accurately represents "arbitrary sequence of names".
PEP 8 repeatedly makes mention of __all__ being a list:
To better support introspection, modules should explicitly declare the names in their public API using the __all__ attribute. Setting __all__ to an empty list indicates that the module has no public API.
"""This is the example module.
This module does stuff.
"""
...
__all__ = ['a', 'b', 'c']

The language reference says:
The public names defined by a module are determined by checking the module’s namespace for a variable named __all__; if defined, it must be a sequence of strings which are names defined or imported by that module.
A sequence is something that supports iteration (for x in __all__) and access using integer indices (__all__[i]). So it can be a list, or a tuple.

From a practical standpoint, it's somewhat common to add elements to __all__ semi-dynamically. It happens when you want to expose functions defined deep in the package structure at the top level. This is much easier to do with a list.
A couple of examples of modules that do this are numpy and pyserial. I strongly suspect that Django does this too in places, but am not familiar enough with it to know for sure.
The idiom looks something like this in __init__.py:
__all__ = [] # or might have some initial names
from .subpackage import (name1, name2, name3)
__all__.extend(['name1', 'name2', 'name3']) # or append 1-by-1 or +=
I've even seen a slightly sloppier approach, although arguably more maintainable under certain circumstances, that looks like this:
__all__ = []
from .subpackage import *
from .subpackage import __all__ as _all
__all__.extend(_all)
del _all
Clearly this is greatly simplified by having a mutable __all__. There is no substantial benefit to turning it into a tuple after the fact or "appending" to a tuple using +=.
Another way a mutable __all__ is useful is when your API depends on optional external packages. It's much easier to enable or disable names in a list than a tuple.
Here is an example of a module that enables additional functionality if a library called optional_dependency is installed:
# Core API
__all__ = ['name', 'name2', 'name3']
from .sub1 import name1
from .sub2 import name2, name3
try:
import optional_dependency
except ImportError:
# Let it be, it maybe issue a warning
pass
else:
from .opt_support import name4, name5
__all__ += ['name4', 'name5']

Just wanted to document a little error I ran into relevant to this post: note that you need a trailing comma to create a single element tuple. So this:
__all__ = ['foo'] # I am list with one element
Is not the same as this:
__all__ = ('foo') # I am a string
Here's an example of this going wrong. In the second case, if you try to import with the wildcard*:
from mymodule import *
You get the confusing error:
AttributeError: module 'mypackage.mymodule' has no attribute 'f'
What is 'f'?!? Well, it is the first element of __all__, which is pointing to the string 'foo', not the single-element tuple ('foo',).
* Using from x import * is maybe what is more to blame here, as opposed to the tuple vs. list choice. But this still seems to be a relatively common pattern in __init__.py files, and makes me lean towards preferring lists.

when to use "_(.)" kind of things in python.? [duplicate]

How can I make methods and data members private in Python? Or doesn't Python support private members?

9.6. Private Variables
“Private” instance variables that
cannot be accessed except from inside
an object, don’t exist in Python.
However, there is a convention that is
followed by most Python code: a name
prefixed with an underscore (e.g.
_spam) should be treated as a non-public part of the API (whether it
is a function, a method or a data
member). It should be considered an
implementation detail and subject to
change without notice.
Since there is a valid use-case for
class-private members (namely to avoid
name clashes of names with names
defined by subclasses), there is
limited support for such a mechanism,
called name mangling. Any identifier
of the form __spam (at least two
leading underscores, at most one
trailing underscore) is textually
replaced with _classname__spam, where
classname is the current class name
with leading underscore(s) stripped.
This mangling is done without regard
to the syntactic position of the
identifier, as long as it occurs
within the definition of a class.
So, for example,
class Test:
def __private_symbol(self):
pass
def normal_symbol(self):
pass
print dir(Test)
will output:
['_Test__private_symbol',
'__doc__',
'__module__',
'normal_symbol']
__private_symbol should be considered a private method, but it would still be accessible through _Test__private_symbol.

The other answers provide the technical details. I'd like to emphasise the difference in philosophy between Python on one hand and languages like C++/Java (which I presume you're familiar with based on your question).
The general attitude in Python (and Perl for that matter) is that the 'privacy' of an attribute is a request to the programmer rather than a barbed wire fence by the compiler/interpreter. The idea is summarised well in this mail and is often referred to as "We're all consenting adults" since it 'assumes' that the programmer is responsible enough to not meddle with the insides. The leading underscores serve as a polite message saying that the attribute is internal.
On the other hand, if you do want to access the internals for some applications (a notable example is documentation generators like pydoc), you're free to do so. Onus is on you as a programmer to know what you're doing and do it properly rather than on the language to force you do to things it's way.

There are no private of any other access protection mechanisms in Python. There is a convention documented in the Python style guide for indicating to the users of your your class that they should not be accessing certain attribute.
_single_leading_underscore: weak "internal use" indicator. E.g. from M import * does not import objects whose name starts with an underscore.
single_trailing_underscore_: used by convention to avoid conflicts with Python keyword, e.g. Tkinter.Toplevel(master, class_='ClassName')
__double_leading_underscore: when naming a class attribute, invokes name mangling (inside class FooBar, __boo becomes _FooBar__boo; see below).

If the name of a Python function,
class method, or attribute starts with
(but doesn't end with) two
underscores, it's private; everything
else is public. Python has no concept
of protected class methods (accessible
only in their own class and descendant
classes). Class methods are either
private (accessible only in their own
class) or public (accessible from
anywhere).
Dive Into Python

Python does not support privacy directly . Programmer need to know when it is safe to modify attribute from outside but anyway with python you can achieve something like private with little tricks.
Now let's see a person can put anything private to it or not.
class Person(object):
def __priva(self):
print "I am Private"
def publ(self):
print " I am public"
def callpriva(self):
self.__priva()
Now When we will execute :
>>> p = Person()
>>> p.publ()
I am public
>>> p.__priva()
Traceback (most recent call last):
File "", line 1, in
p.__priva()
AttributeError: 'Person' object has no attribute '__priva'
​#Explanation : You can see here we are not able to fetch that private method directly.
>>> p.callpriva()
I am Private
#​Explanation : Here we can access private method inside class​
​Then how someone can access that variable ???
You can do like :
>>> p._Person__priva
I am Private
​wow , actually if python is getting any variable starting with double underscore are “translated” by adding a single underscore and the class name to the beginning:
Note : If you do not want this name changing but you still want to send a signal for other objects to stay away, you can use a single initial underscore names with an initial underscore aren’t imported with starred imports (from module import *)
Example :
#test.py
def hello():
print "hello"
def _hello():
print "Hello private"
#----------------------
#test2.py
from test import *
print hello()
print _hello()
output-->
hello
Traceback (most recent call last):
File "", line 1, in
NameError: name '_hello' is not defined
Now if we will call _hello manually .
#test2.py
from test import _hello , hello
print hello()
print _hello()
output-->
hello
hello private
Finally : Python doesn’t really have an equivalent privacy support, although single
and double initial underscores do to some extent give you two levels of privacy

This might work:
import sys, functools
def private(member):
#functools.wraps(member)
def wrapper(*function_args):
myself = member.__name__
caller = sys._getframe(1).f_code.co_name
if (not caller in dir(function_args[0]) and not caller is myself):
raise Exception("%s called by %s is private"%(myself,caller))
return member(*function_args)
return wrapper
class test:
def public_method(self):
print('public method called')
#private
def private_method(self):
print('private method called')
t = test()
t.public_method()
t.private_method()

This is kinda a l-o-n-g answer but I think it gets to the root of the real problem here -- scope of visibility. Just hang in there while I slog through this!
Simply importing a module need not necessarily give the application developer access to all of its classes or methods; if I can't actually SEE the module source code how will I know what's available? Some one (or some THING) has to tell me what I can do and explain how to use those features I'm allowed to use, otherwise the whole thing is useless to me.
Those developing higher-level abstractions based on fundamental classes and methods via imported modules are presented with a specification DOCUMENT -- NOT the actual source code.
The module spec describes all the features intended to be visible to the client developer. When dealing with large projects and software project teams, the actual implementation of a module should ALWAYS remain hidden from those using it -- it's a blackbox with an interface to the outside world. For OOD purists, I believe the techie terms are "decoupling" and "coherence". The module user need only know the interface methods without being burden with the details of implementation.
A module should NEVER be changed without first changing its underlying spec document, which may require review / approval in some organizations prior to changing the code.
As hobby programmer (retired now), I start a new module with the spec doc actually written out as a giant comment block at the top of the module, this will be the part the user actually sees in the spec library. Since it's just me, I've yet to set up a library, but it would be easy enough to do.
Then I begin coding by writing the various classes and methods but without functional bodies -- just null print statements like "print()" -- just enough to allow the module to compile without syntax errors. When this step is complete I compile the completed null-module -- this is my spec. If I were working on a project team, I would present this spec/interface for review & commentary before proceeding with fleshing out the body.
I flesh out the bodies of each method one at a time and compile accordingly, ensuring syntax errors are fixed immediately on-the-fly. This is also a good time to start writing a temporary "main" execution section at the bottom to test each method as you code it. When the coding/testing are complete, all of the test code is commented out until you need it again should updates become necessary.
In a real-world development team, the spec comment block would also appear in a document control library, but that's another story. The point is: you, as the module client, see only this spec and NOT the source code.
PS: long before the beginning of time, I worked in the defense aerospace community and we did some pretty cool stuff, but things like proprietary algorithms and sensitive systems control logic were tightly vaulted and encrypted in super-duper secure software libraries. We had access to module / package interfaces but NOT the blackbox implementation bodies. There was a document management tool that handled all system-level designs, software specs, source code and test records -- it was all synched together. The government had strict requirements software quality assurance standards. Anyone remember a language called "Ada"? That's how old I am!

import inspect
class Number:
def __init__(self, value):
self.my_private = value
def set_private(self, value):
self.my_private = value
def __setattr__(self, my_private, value):
f = inspect.stack()[1][3]
if f not in ['__init__', 'set_private']:
raise Exception("can't access private member-my_private")
# the default behavior
self.__dict__[my_private] = value
def main():
n = Number(2)
print(n.my_private)
n.set_private(3)
print(n.my_private)
if __name__ == '__main__':
main()

I use Python 2.7 and 3.5. I wrote this code:
class MyOBject(object):
def __init__(self):
self.__private_field = 10
my_object = MyOBject()
print(my_object.__private_field)
ran it and got:
AttributeError: 'MyOBject' object has no attribute '__private_field'
Please see:
https://www.tutorialsteacher.com/python/private-and-protected-access-modifiers-in-python

Private functions in python

Is it possible to avoid importing a file with from file import function?
Someone told me i would need to put an underscore as prefix, like: _function, but isn't working.
I'm using Python 2.6 because of a legacy code.

There are ways you can prevent the import, but they're generally hacks and you want to avoid them. The normal method is to just use the underscore:
def _function():
pass
Then, when you import,
from my_module import *
You'll notice that _function is not imported because it begins with an underscore. However, you can always do this:
# In Python culture, this is considered rude
from my_module import _function
But you're not supposed to do that. Just don't do that, and you'll be fine. Python's attitude is that we're all adults. There are a lot of other things you're not supposed to do which are far worse, like
import my_module
# Remove the definition for _function!
del my_module._function

There is no privacy in Python. There are only conventions governing what external code should consider publicly accessible and usable.
Importing a module for the first time, triggers the creation of a module object and the execution of all top-level code in the module. The module object contains the global namespace with the result of that code having run.
Because Python is dynamic you can always introspect the module namespace; you can see all names defined, all objects those names reference, and you can access and alter everything. It doesn't matter here if those names start with underscores or not.
So the only reason you use a leading _ underscore for a name, is to document that the name is internal to the implementation of the module, and that external code should not rely on that name existing in a future version. The from module import * syntax will ignore such names for that reason alone. But you can't prevent a determined programmer from accessing such a name anyway. They simply do so at their own risk, it is not your responsibility to keep them from that access.
If you have functions or other objects that are only needed to initialise the module, you are of course free to delete those names at the end.

How to find out what methods, properties, etc a python module possesses

Lets say I import a module. In order for me to make the best use of it, I would like to know what properties, methods, etc. that I can use. Is there a way to find that out?
As an example: Determining running programs in Python
In this line:
os.system('WMIC /OUTPUT:C:\ProcessList.txt PROCESS get Caption,Commandline,Processid')
Let's say I wanted to also print out the memory consumed by the processes. How do I find out if that's possible? And what would be the correct 'label' for it? (just as the author uses 'Commandline', 'ProcessId')
Similarly, in this:
import win32com.client
def find_process(name):
objWMIService = win32com.client.Dispatch("WbemScripting.SWbemLocator")
objSWbemServices = objWMIService.ConnectServer(".", "root\cimv2")
colItems = objSWbemServices.ExecQuery(
"Select * from Win32_Process where Caption = '{0}'".format(name))
return len(colItems)
print find_process("SciTE.exe")
How would I make the function also print out the memory consumed, the executable path, etc.?

As for Python modules, you can do
>>> import module
>>> help(module)
and you'll get a list of supported methods (more exactly, you get the docstring, which might not contain every single method). If you want that, you can use
>>> dir(module)
although now you'd just get a long list of all properties, methods, classes etc. in that module.
In your first example, you're calling an external program, though. Of course Python has no idea which features wmic.exe has. How should it?

dir(module) returns the names of the attributes of the module
module.__dict__ is the mapping between the keys and the attributes objects themselves
module.__dict__.keys() and dir(module) are lists having the same elements, though they are not equals because the elements aren't in same order in them
it seems that help(module) iswhat you really need

Python has a build in function called dir(). I'm not sure if this is what you are referring to, but fire up a interactive python console and type:
import datetime
dir(datetime)
This should give you a list of methods, properties and submodules

#ldmvcd
Ok, excuse me, I think you are a beginner and you don't see to what fundamental notions I am refering.
Objects are Python’s abstraction for
data. All data in a Python program is
represented by objects or by relations
between objects.
http://docs.python.org/reference/datamodel.html#the-standard-type-hierarchy
I don't understand why it is called "abstraction": for me an object is something real in the machine, a series of bits organized according certain rules to represent conceptual data or functionning.
Names refer to objects. Names are
introduced by name binding operations.
Each occurrence of a name in the
program text refers to the binding of
that name established in the innermost
function block containing the use.
http://docs.python.org/reference/executionmodel.html#naming-and-binding
.
A namespace is a mapping from names to
objects. Most namespaces are currently
implemented as Python dictionaries,
but that’s normally not noticeable in
any way (except for performance), and
it may change in the future. Examples
of namespaces are: the set of built-in
names (containing functions such as
abs(), and built-in exception names);
the global names in a module; and the
local names in a function invocation.
In a sense the set of attributes of an
object also form a namespace.
http://docs.python.org/tutorial/classes.html#a-word-about-names-and-objects
.
By the way, I use the word attribute
for any name following a dot — for
example, in the expression z.real,
real is an attribute of the object z.
Strictly speaking, references to names
in modules are attribute references:
in the expression modname.funcname,
modname is a module object and
funcname is an attribute of it. In
this case there happens to be a
straightforward mapping between the
module’s attributes and the global
names defined in the module: they
share the same namespace!
http://docs.python.org/tutorial/classes.html#a-word-about-names-and-objects
.
Namespaces are created at different
moments and have different lifetimes.
http://docs.python.org/tutorial/classes.html#a-word-about-names-and-objects
.
The namespace for a module is
automatically created the first time a
module is imported. The main module
for a script is always called
main. http://docs.python.org/reference/executionmodel.html#naming-and-binding
.
Well, a Python programm is a big machine that plays with objects, references to these objects , names of these objects, and namespaces in which are binded the names and the objects , namespaces being implemented as dictionaries.
So, you're right: when I refer to keys , I refer to names being the keys in the diverse namespaces. Names are arbitrary or not , according if the objects they have been created to name are user's objects or built-in objects.
I give advise you to read thoroughly the parts
3.1. Objects , values and types
http://docs.python.org/reference/datamodel.html#the-standard-type-hierarchy
and
4.1. Naming and binding
http://docs.python.org/reference/executionmodel.html#naming-and-binding

two conflicting meanings of builtins in python 3 (python 3.1, python 3k, python3000)

I just posted below query to comp.lang.python, but i feel this kind of question has some kind of right-of-way here on Stack Overflow, too, so be it repeated. the essence: why does ‘builtins’ have two distinct interpretations in Python 3?
I would be very gladly accept any commentaries about what this
sentence, gleaned from http://celabs.com/python-3.1/reference/executionmodel.html,
is meant to mean, or why gods have decided this is the way to go. i
anticipate this guy named Kay Schluehr will have a say on that, or
maybe even the BDFL will care to pronounce __builtins__ the
correct way to his fallovers, followers, and fellownerds::
The built-in namespace associated with the execution of
a code block is actually found by looking up the name
__builtins__ in its global namespace; this should be a
dictionary or a module (in the latter case the module’s
dictionary is used). By default, when in the __main__
module, __builtins__ is the built-in module builtins;
when in any other module, __builtins__ is an alias for
the dictionary of the builtins module itself.
__builtins__ can be set to a user-created dictionary to
create a weak form of restricted execution.
it used to be the case that there were at least two distinct terms,
‘builtin’ (in the singular) and ‘builtins’ (in the plural), some of
which existed both in module and in dict form (?just guessing?). now
there is only builtins, so fortunately the ambivalence between
singular and plural has gone—good riddance.
but why does __builtins__ change its meaning depending on whether
this is the scope of the ‘script’ (i.e. the module whose name was
present, when calling python foobar.py) or whether this is the
scope of a secondary module (imported or executed, directly or
indirectly, by foobar.py)? i cannot understand the reasoning
behind this and find it highly confusing.
rationale: why do i care?—i want to be able to ‘export names to the
global namespace that were not marked private (by an underscore
prefix) in a python module that i execute via exec( compile( get
( locator ), locator, 'exec' ), R ) where R is supposed to going
to hold the private names of said module’. it is a little arcane but
the basic exercise is to by-pass python’s import system and get similr
results... it is all about injecting names into the all-global and the
module-global namespaces.

getattr(__builtins__, '__dict__', __builtins__) should give you the dict that you want to update to "export names to the global namespace", whether __builtins__ is a dict (then it doesn't have a __dict__ attribute so getattr returns the third argument, which is the dict __builtins__ itself) or a module (then it does have that attribute and getattr returns it). This is the workaround. As to why Python's documented to work in a way requiring such a tangled workaround, I'd classify it as an unfortunate case of an implementation issue surfacing to user-visible (and indeed documented) level (sigh). Pity we didn't think of fixing it in the move to Python 3, but it's too late to make backwards-incompatible changes now:-(.