We Keep Coding

Private functions in python

Is it possible to avoid importing a file with from file import function?
Someone told me i would need to put an underscore as prefix, like: _function, but isn't working.
I'm using Python 2.6 because of a legacy code.

There are ways you can prevent the import, but they're generally hacks and you want to avoid them. The normal method is to just use the underscore:
def _function():
pass
Then, when you import,
from my_module import *
You'll notice that _function is not imported because it begins with an underscore. However, you can always do this:
# In Python culture, this is considered rude
from my_module import _function
But you're not supposed to do that. Just don't do that, and you'll be fine. Python's attitude is that we're all adults. There are a lot of other things you're not supposed to do which are far worse, like
import my_module
# Remove the definition for _function!
del my_module._function

There is no privacy in Python. There are only conventions governing what external code should consider publicly accessible and usable.
Importing a module for the first time, triggers the creation of a module object and the execution of all top-level code in the module. The module object contains the global namespace with the result of that code having run.
Because Python is dynamic you can always introspect the module namespace; you can see all names defined, all objects those names reference, and you can access and alter everything. It doesn't matter here if those names start with underscores or not.
So the only reason you use a leading _ underscore for a name, is to document that the name is internal to the implementation of the module, and that external code should not rely on that name existing in a future version. The from module import * syntax will ignore such names for that reason alone. But you can't prevent a determined programmer from accessing such a name anyway. They simply do so at their own risk, it is not your responsibility to keep them from that access.
If you have functions or other objects that are only needed to initialise the module, you are of course free to delete those names at the end.

Ansible: Access host/group vars from within custom module

Is there a way how one can access host/group vars from within a custom written module? I would like to avoid to pass all required vars as module parameters.
My module is written in Python and I use the boilerplate. I checked pretty much all available vars but they are not stored anywhere:
def main():
pprint(dir())
pprint(globals())
pprint(locals())
for name in vars().keys():
print(name)
Now my only hope is they are somehow accessible through the undocumented module utils.
I guess it is not possible, since the module runs on the target machine and probably the facts/host/group vars are not transferred along with the module...
Edit: Found the module utils now and it doesn't look promising.

Is there a way how one can access host/group vars from within a custom
written module?
Not built-in.
You will have to pass them yourself one way or the other:
Module args.
Serialize to local file system (with pickle or yaml.dump() or json or ...) and send the file over.
any other innovative ideas you can come up with.
Unfortunately you can't just send over whole host/groupvar files as-it-is because you would have to implement the variable scope/precedence resolution algorithm of ansible which is undefined (it's not the Zen philosophy of ansible to define such petty things :P ).
--edit--
I see they have some precedence defined now.
Ansible does apply variable precedence, and you might have a use for
it. Here is the order of precedence from least to greatest (the last
listed variables override all other variables):
command line values (for example, -u my_user, these are not variables)
role defaults (defined in role/defaults/main.yml) 1
inventory file or script group vars 2
inventory group_vars/all 3
playbook group_vars/all 3
inventory group_vars/* 3
playbook group_vars/* 3
inventory file or script host vars 2
inventory host_vars/* 3
playbook host_vars/* 3
host facts / cached set_facts 4
play vars
play vars_prompt
play vars_files
role vars (defined in role/vars/main.yml)
block vars (only for tasks in block)
task vars (only for the task)
include_vars
set_facts / registered vars
role (and include_role) params
include params
extra vars (for example, -e "user=my_user")(always win precedence)
In general, Ansible gives precedence to variables that were defined
more recently, more actively, and with more explicit scope. Variables
in the defaults folder inside a role are easily overridden. Anything
in the vars directory of the role overrides previous versions of that
variable in the namespace. Host and/or inventory variables override
role defaults, but explicit includes such as the vars directory or an
include_vars task override inventory variables.
Ansible merges different variables set in inventory so that more
specific settings override more generic settings. For example,
ansible_ssh_user specified as a group_var is overridden by
ansible_user specified as a host_var. For details about the precedence
of variables set in inventory, see How variables are merged.
Footnotes
1 Tasks in each role see their own role’s defaults. Tasks defined
outside of a role see the last role’s defaults.
2(1,2) Variables defined in inventory file or provided by dynamic
inventory.
3(1,2,3,4,5,6) Includes vars added by ‘vars plugins’ as well as
host_vars and group_vars which are added by the default vars plugin
shipped with Ansible.
4 When created with set_facts’s cacheable option, variables have the
high precedence in the play, but are the same as a host facts
precedence when they come from the cache.

As per your suggestion in your answer here, I did manage to read host_vars and local play vars through a custom Action Plugin.
I'm posting this answer for completeness sake and to give an explicit example of how one might go about this method, although you gave this idea originally :)
Note - this example is incomplete in terms of a fully functioning plugin. It just shows the how to access variables.
from ansible.template import is_template
from ansible.plugins.action import ActionBase
class ActionModule(ActionBase):
def run(self, tmp=None, task_vars=None):
# some boilerplate ...
# init
result = super(ActionModule, self).run(tmp, task_vars)
# more boilerplate ...
# check the arguments passed to the task, where if missing, return None
self._task.args.get('<TASK ARGUMENT NAME>', None)
# or
# check if the play has vars defined
task_vars['vars']['<ARGUMENT NAME>']
# or
# check if the host vars has something defined
task_vars['hostvars']['<HOST NAME FORM HOSTVARS>']['<ARGUMENT NAME>']
# again boilerplate...
# build arguments to pass to the module
some_module_args = dict(
arg1=arg1,
arg2=arg2
)
# call the module with the above arguments...
In case you have your playbook variables with jinja 2 templates, you can resolve these templates in the plugin as follows:
from ansible.template import is_template
# check if the variable is a template through 'is_template'
if is_template(var, self._templar.environment):
# access the internal `_templar` object to resolve the template
resolved_arg = self._templar.template(var_arg)
Some words of caution:
If you have a variable defined in your playbook as follows
# things ...
#
vars:
- pkcs12_path: '{{ pkcs12_full_path }}'
- pkcs12_pass: '{{ pkcs12_password }}'
The variable pkcs12_path must not match the host_vars name.
For instance, if you had pkcs12_path: '{{ pkcs12_path }}', then resolving the template with the above code will cause a recursive exception... This might be obvious to some, but for me it was surprising that the host_vars variable and the playbook variable must not be with the same name.
You can also access variables through task_vars['<ARG_NAME>'], but I'm not sure where it's reading this from. Also it's less explicit than taking variables from task_vars['vars']['<ARG_NAME>'] or from the hostvars.
PS - in the time of writing this, the example follows the basic structure of what Ansible consider an Action Plugin. In the future, the run method might change its signature...

I think you pretty much hit the nail on the head with your thinking here:
I guess it is not possible, since the module runs on the target machine and probably the facts/host/group vars are not transferred along with the module...
However, having said that, if you really have a need for this then there might be a slightly messy way of doing it. As of Ansible 1.8 you can set up fact caching, which uses redis to cache facts between runs of plays. Since redis is pretty easy to use and has clients for most popular programming languages, you could have your module query the redis server for any facts you need. It's not exactly the cleanest way to do it, but it just might work.

Reloading a module gives functionality that isn't originally available by importing it. where can i learn more about this? [duplicate]

This question already has answers here:
Why should we NOT use sys.setdefaultencoding("utf-8") in a py script?
(4 answers)
Closed 8 years ago.
This bit of code solves a problem I had. however, the "setdefaultencoding" is not available without reload.
what is this quirk of the language called?
why wasn't i told earlier?
where can i read more about it.
import sys;
reload(sys);
sys.setdefaultencoding("utf8")
FROM
http://mypy.pythonblogs.com/12_mypy/archive/1253_workaround_for_python_bug_ascii_codec_cant_encode_character_uxa0_in_position_111_ordinal_not_in_range128.html

The 'quirk' is the site module deliberately deleting the sys.setdefaultencoding() function:
# Remove sys.setdefaultencoding() so that users cannot change the
# encoding after initialization. The test for presence is needed when
# this module is run as a script, because this code is executed twice.
if hasattr(sys, "setdefaultencoding"):
del sys.setdefaultencoding
You should not use it! Setting the default encoding to UTF-8 is like strapping a stick to your leg after you broke it and walking on instead of having a doctor set the broken bones.
Really, let me make it clear: There is a reason it is removed and the reason is that you'll a) break any module that relies on the normal default and b) you are masking your actual problems, which is handling Unicode correctly by decoding as early as possible and postponing encoding until you need to send the data out again.
That out the way, the way the reload() function works is that it lets you bypass the module cache; import will load a Python module only once; subsequent imports give you the already-loaded module. reload() loads the module a-new as if it was never imported, and merges the new names back into the existing module object (to preserve extra names added later):
Reload a previously imported module. The argument must be a module object, so it must have been successfully imported before. This is useful if you have edited the module source file using an external editor and want to try out the new version without leaving the Python interpreter. The return value is the module object (the same as the module argument).
When reload(module) is executed:
Python modules’ code is recompiled and the module-level code reexecuted, defining a new set of objects which are bound to names in the module’s dictionary. The init function of extension modules is not called a second time.
As with all other objects in Python the old objects are only reclaimed after their reference counts drop to zero.
The names in the module namespace are updated to point to any new or changed objects.
Other references to the old objects (such as names external to the module) are not rebound to refer to the new objects and must be updated in each namespace where they occur if that is desired.
So reload() restores the deleted sys.setdefaultencoding() name into the module.

Should namedtuples follow constant name conventions in python?

I've written a small python module where I use a couple of namedtuples to pass info around because I find them very expressive. I considered these types and named them following the PEP8 convention for class names (CamelCased). However pylint sees the line:
PersonData = collections.namedtuple('PersonData', 'name surname age')
at the module's global scope and goes: Invalid constant name 'PersonData'.
Am I misusing namedtuples? What's the pythonic recommendation? I can only think of suppressing the warning, renaming the structure to PERSON_DATA, or making it a full class. Note that, in my case, it wouldn't make sense for it to have methods though.
If the answer is to suppress the warning. Wouldn't this be a recurring problem with pylint vs named tuples?
(using pylint-0.26.0, python-2.7.4)

This issue has been solved in newer versions of pylint.
My system was picking the version from the ubuntu (13.04) repositories (pylint-0.26.0).
Using a virtualenv I could pip install pylint-1.0.0 which recognizes named tuples and will actually enforce using the same format as classes.
Note that it won't pick it up if you're doing string manipulation on the fields string. For example:
PersonData = collections.namedtuple('PersonData', 'name ' + 'surname age')
will still cause pylint-1.0.0 to spit out the error code for invalid constant name. In this case the only workaround is to disable the warning as per Martijn Pieters suggestion.

You can ignore pylint here, you are using the camel case naming convention exactly right.
You can suppress the warning:
PersonData = collections.namedtuple('PersonData', 'name surname age') # pylint: disable-msg=C0103
namedtuple is a class factory, so use the naming conventions for a class.

This has been fixed in pylint-1.0.0 (see Pylint's Changelog):
For toplevel name assignment, the class name regex will be used if pylint can detect that value on the right-hand side is a class (like collections.namedtuple()).
Note that the namedtuple must be defined on the toplevel, if defined e.g. within a function pylint will still give a invalid-name warning.

in such case pylint should detect the name is assigned to a class an use the class name regexp.
Please submit a ticket on http://bitbucket.org/logilab/pylint

Pylint best practices

Pylint looks like a good tool for running analysis of Python code.
However, our main objective is to catch any potential bugs and not coding conventions. Enabling all Pylint checks seems to generate a lot of noise. What is the set of Pylint features you use and is effective?

You can block any warnings/errors you don't like, via:
pylint --disable=error1,error2
I've blocked the following (description from http://www.logilab.org/card/pylintfeatures):
W0511: Used when a warning note as FIXME or XXX is detected
W0142: Used * or * magic*. Used when a function or method is called using *args or **kwargs to dispatch arguments. This doesn't improve readability and should be used with care.
W0141: Used builtin function %r. Used when a black listed builtin function is used (see the bad-function option). Usual black listed functions are the ones like map, or filter, where Python offers now some cleaner alternative like list comprehension.
R0912: Too many branches (%s/%s). Used when a function or method has too many branches, making it hard to follow.
R0913: Too many arguments (%s/%s). Used when a function or method takes too many arguments.
R0914: Too many local variables (%s/%s). Used when a function or method has too many local variables.
R0903: Too few public methods (%s/%s). Used when class has too few public methods, so be sure it's really worth it.
W0212: Access to a protected member %s of a client class. Used when a protected member (i.e. class member with a name beginning with an underscore) is access outside the class or a descendant of the class where it's defined.
W0312: Found indentation with %ss instead of %ss. Used when there are some mixed tabs and spaces in a module.
C0111: Missing docstring. Used when a module, function, class or method has no docstring. Some special methods like __init__ don't necessarily require a docstring.
C0103: Invalid name "%s" (should match %s). Used when the name doesn't match the regular expression associated to its type (constant, variable, class...).

To persistently disable warnings and conventions:
Create a ~/.pylintrc file by running pylint --generate-rcfile > ~/.pylintrc
Edit ~/.pylintrc
Uncomment disable= and change that line to disable=W,C

Pyflakes should serve your purpose well.

-E will only flag what Pylint thinks is an error (i.e., no warnings, no conventions, etc.)

Using grep like:
pylint my_file.py | grep -v "^C"
Edit :
As mentionned in the question, to remove the conventions advices from pylint output, you remove the lines that start with an uppercase C.
From the doc of pylint, the output consists in lines that fit the format
MESSAGE_TYPE: LINE_NUM:[OBJECT:] MESSAGE
and the message type can be:
[R]efactor for a “good practice” metric violation
[C]onvention for coding standard violation
[W]arning for stylistic problems, or minor programming issues
[E]rror for important programming issues (i.e. most probably bug)
[F]atal for errors which prevented further processing
Only the first letter is displayed, so you can play with grep to select/remove the level of message type you want.
I didn't use Pylint recently, but I would probably use a parameter inside Pylint to do so.