I have created my own module X. At the beginning, I import functions from some other modules (e.g. from math import func). I have noticed that when I create documentation with:
pydoc -w X
the resulting file also contains the imported function function from the math module, which is undesirable (especially if I import many functions from several modules, which is what I do).
It can be avoided by instead calling:
import math
but in this case I have to import all the functions and then call them using math.func instead of just func.
Is there another way with which I can avoid populating my documentation with imported functions while still being able to import functions with from?
Looking inside the source for Pydoc you can see the following comment made:
if all is not None:
# only document that which the programmer exported in __all__
return name in all
meaning that pydoc will look for the __all__ module attribute, and, if defined, will only document the functions defined in it.
So, for you module X, you can define the functions to be exported in __all__ by specifying their names. Only those will get documented in the corresponding Functions section:
__all__ = ['myfunc1', 'myfunc2', ..., 'myfuncN']
Case in point:
Without __all__, the following simple file named mod.py:
from math import cos
def myfunc():
""" documentation"""
pass
Generates a mod.html file that contains the documentation for the user defined myfunc() and for the imported built-in function cos():
By adding __all__ and specifying the function name(s) you'd want to export inside it:
__all__ = ['myfunc'] # visible names
from math import cos
def myfunc():
""" documentation"""
pass
You'll 'filter out' the cos() function and only have documentation for myfunc():
Note: __all__ can contain functions and variable names used inside you script. pydoc will discriminate between these and segregate them in two different groups:
Functions in Functions
Variables in Data.
Related
I'm currently setting up a test suite for a file called main.py. The test file is called test_main.py. Here's an example:
# main.py
def add(a,b):
return a+b
#test_main.py
import pytest
from main import *
def test_add():
assert add(1,2) == 3
For reasons which are outside the scope of this question, I would like to dynamically load the function add in test_main.py as opposed to calling it directly. I'm already aware this is possible using the following
globals or vars
use of importlib
use of eval
However, I'd like to see if there's another option. globals and vars are bad practice. eval is allright, but it doesn't return the function object and I have to do some string manipulation to get the function call, including its arguments right. importlib is by far the best option, but main.py happens to contain functions which I want to import the "normal" way. It feels wrong to import functions in test_main.py using both an import statement and the importlib module.
So, is there a better way? One which is more pythonic?
I have 3 file main.py, a.py, b.py and a function func() belongs to either a.py or b.py but I don't know exactly which. The code in the main.py:
from a import *
from b import *
func()
I want to get the directory of the file containing the func(), here is a.py or b.py. So how can I do?
Thanks in advance.
You can use the following code snippet to get file paths of modules containing the desired functions:
file_path = func.__code__.co_filename
dir_path = os.path.dirname(file_path) # since you wanted directory
where func is the function you imported from one of the modules.
Note: This will work for all function objects that are instances of the python function class. For eg, this won't work for functions in numpy, because they are instances of the numpy.ufunc class.
This is why they tell you not to do import *. As a rule of thumb, import only the names you need from any given import, to avoid confusion of this nature.
What you can do is check, in reverse order, which module contains the function you're concerned about. Later imports should overwrite earlier ones with the same name.
if hasattr(b, 'func') and b.func is func:
print(b.__file__)
elif hasattr(a, 'func') and a.func is func:
print(a.__file__)
else:
print("func came from somewhere else")
You can just look, by listing all the functions in a module with:
print(dir(module))
Eg. print(dir(math)) will list all the functions in the math module
I usually don't think too hard about variable scope in python, but I wanted to see if there's a clean explanation for this. Given two files called main.py and utils.py:
utils.py
def run():
print(L)
main.py
import utils
def run():
print(L)
if __name__ == '__main__':
L = [1,2]
run()
utils.run()
The first run() call in main.py runs fine despite L not being fed into run(), and the utils.run() call raises a NameError. Is L a global variable available to all functions defined in main.py?
If I imported utils with from utils import * instead of import utils, would that change anything?
It's module-level scope. A global variable defined in a module is available to all functions defined in the same module (if it's not overriden). Functions in another module don't have access to another module's variables unless they import them.
About "If I imported utils with from utils import * instead of import utils, would that change anything?":
No. The scope is determined at parsing time.
Check
this
for more information.
Notably:
It is important to realize that scopes are determined textually: the global
scope of a function defined in a module is that module’s namespace, no matter
from where or by what alias the function is called. On the other hand, the
actual search for names is done dynamically, at run time [...]
So the global scopes of both functions for variables defined in a module are the modules they're defined in. For one, its module also later has a definition for a global variable it uses, but not the other module, and when it's time to check for a variable when a function is run, each checks their own module's variables definitions, one finds it, the other does not.
See Python's FAQ. Their implementation of scope is a compromise between convenience and the dangers of globals.
Variables are treated as globals if they're only referenced by a function, and need to be explicitly declared as globals (e.g. global foo ) inside of the function body if you want to edit them. If you edit run() to try and change the value of L, you'll get an error.
What's happening here is that your Python code imports utils, and then runs run(). This function sees that you're looking for a variable named "L," and checks your global namespace.
This question already has answers here:
What can I do about "ImportError: Cannot import name X" or "AttributeError: ... (most likely due to a circular import)"?
(17 answers)
Closed 6 months ago.
I know the issue of circular imports in python has come up many times before and I have read these discussions. The comment that is made repeatedly in these discussions is that a circular import is a sign of a bad design and the code should be reorganised to avoid the circular import.
Could someone tell me how to avoid a circular import in this situation?: I have two classes and I want each class to have a constructor (method) which takes an instance of the other class and returns an instance of the class.
More specifically, one class is mutable and one is immutable. The immutable class is needed
for hashing, comparing and so on. The mutable class is needed to do things too. This is similar to sets and frozensets or to lists and tuples.
I could put both class definitions in the same module. Are there any other suggestions?
A toy example would be class A which has an attribute which is a list and class B which has an attribute which is a tuple. Then class A has a method which takes an instance of class B and returns an instance of class A (by converting the tuple to a list) and similarly class B has a method which takes an instance of class A and returns an instance of class B (by converting the list to a tuple).
Consider the following example python package where a.py and b.py depend on each other:
/package
__init__.py
a.py
b.py
Types of circular import problems
Circular import dependencies typically fall into two categories depending
on what you're trying to import and where you're using it inside each
module. (And whether you're using python 2 or 3).
1. Errors importing modules with circular imports
In some cases, just importing a module with a circular import dependency
can result in errors even if you're not referencing anything from the
imported module.
There are several standard ways to import a module in python
import package.a # (1) Absolute import
import package.a as a_mod # (2) Absolute import bound to different name
from package import a # (3) Alternate absolute import
import a # (4) Implicit relative import (deprecated, python 2 only)
from . import a # (5) Explicit relative import
Unfortunately, only the 1st and 4th options actually work when you
have circular dependencies (the rest all raise ImportError
or AttributeError). In general, you shouldn't be using the
4th syntax, since it only works in python2 and runs the risk of
clashing with other 3rd party modules. So really, only the first
syntax is guaranteed to work.
EDIT: The ImportError and AttributeError issues only occur in
python 2. In python 3 the import machinery has been rewritten and all
of these import statements (with the exception of 4) will work, even with
circular dependencies. While the solutions in this section may help refactoring python 3 code, they are mainly intended
for people using python 2.
Absolute Import
Just use the first import syntax above. The downside to this method is
that the import names can get super long for large packages.
In a.py
import package.b
In b.py
import package.a
Defer import until later
I've seen this method used in lots of packages, but it still feels
hacky to me, and I dislike that I can't look at the top of a module
and see all its dependencies, I have to go searching through all the
functions as well.
In a.py
def func():
from package import b
In b.py
def func():
from package import a
Put all imports in a central module
This also works, but has the same problem as the first method, where
all the package and submodule calls get super long. It also has two
major flaws -- it forces all the submodules to be imported, even if
you're only using one or two, and you still can't look at any of the
submodules and quickly see their dependencies at the top, you have to
go sifting through functions.
In __init__.py
from . import a
from . import b
In a.py
import package
def func():
package.b.some_object()
In b.py
import package
def func():
package.a.some_object()
2. Errors using imported objects with circular dependencies
Now, while you may be able to import a module with a circular import
dependency, you won't be able to import any objects defined in the module
or actually be able to reference that imported module anywhere
in the top level of the module where you're importing it. You can,
however, use the imported module inside functions and code blocks that don't
get run on import.
For example, this will work:
package/a.py
import package.b
def func_a():
return "a"
package/b.py
import package.a
def func_b():
# Notice how package.a is only referenced *inside* a function
# and not the top level of the module.
return package.a.func_a() + "b"
But this won't work
package/a.py
import package.b
class A(object):
pass
package/b.py
import package.a
# package.a is referenced at the top level of the module
class B(package.a.A):
pass
You'll get an exception
AttributeError: module 'package' has no attribute 'a'
Generally, in most valid cases of circular dependencies, it's possible
to refactor or reorganize the code to prevent these errors and move
module references inside a code block.
Only import the module, don't import from the module:
Consider a.py:
import b
class A:
def bar(self):
return b.B()
and b.py:
import a
class B:
def bar(self):
return a.A()
This works perfectly fine.
We do a combination of absolute imports and functions for better reading and shorter access strings.
Advantage: Shorter access strings compared to pure absolute imports
Disadvantage: a bit more overhead due to extra function call
main/sub/a.py
import main.sub.b
b_mod = lambda: main.sub.b
class A():
def __init__(self):
print('in class "A":', b_mod().B.__name__)
main/sub/b.py
import main.sub.a
a_mod = lambda: main.sub.a
class B():
def __init__(self):
print('in class "B":', a_mod().A.__name__)
The following code prints each function's docstring from an imported module. However, the results incude some functions that were not defined within the module, but rather, were imported by the module.
import inspect
import my_module
all_functions = inspect.getmembers(my_module, inspect.isfunction)
for i in all_functions:
print i[0] # print function name
print i[1].__doc__ # print docstring
How can I print only the docstrings of functions defined within the module?
Functions have a __module__ attribute storing the name of the module they were defined in. You can check if that matches the module you're inspecting. Note that this will frequently miss functions that actually are part of a module's API, but were defined in a different module. For example, heapq.heappush.__module__ == '_heapq', because the function is actually defined in a C module _heapq and import *ed into the Python module heapq.