We Keep Coding

pdoc3 or Sphinx for directory with nested module

My code directory looks like below. I need to generate documentation for all the modules like for sub1,sub2,submoduleA1,submoduleB1 and so on.
Also as shown for submoduleB2.py: all the modules imports from other modules/submodules
<workspace>
└── toolbox (main folder)
├── __init__.py
│
├── sub
│ ├── __init__.py
│ ├── sub1.py
│ └── sub2.py
│
├── subpackageA
│ ├── __init__.py
│ ├── submoduleA1.py
│ └── submoduleA2.py
│
└── subpackageB
├── __init__.py
├── submoduleB1.py
└── submoduleB2.py code[from sub import sub1
from subpackageA import submoduleA2 and so on]
code structure for submoduleB2.py
from __future__ import absolute_import, division
import copy
import logging
import numpy as np
import pandas as pd
from dc.dc import DataCleaning
from sub.sub1 import ToolboxLogger
from subpackageA import pan
LOGGER = ToolboxLogger(
"MATH_FUNCTIONS", enableconsolelog=True, enablefilelog=False, loglevel=logging.DEBUG
).logger
"""
Calculations also take into account units of the tags that are passed in
"""
def spread(tag_list):
"""
Returns the spread of a set of actual tag values
:param tag_list: List of tag objects
:type tag_list: list
:return: Pandas Series of spreads
:rtype: Pandas Series
:example:
>>> tag_list = [tp.RH1_ogt_1,
tp.RH1_ogt_2,
tp.RH1_ogt_3,
tp.RH1_ogt_4,
tp.RH1_ogt_5,
tp.RH1_ogt_6]
>>> spread = pan.spread(tag_list)
"""
# use the same units for everything
units_to_use = tag_list[0].units
idxs = tag_list[0].actuals.index
spread_df = pd.DataFrame(index=idxs)
spread_series = spread_df.max(axis=1).copy()
return Q_(spread_series, units_to_use)
I tried to run the pdoc command using anaconda prompt by navigating it to the toolbox folder and executed the below command
pdoc --html --external-links --all-submodules preprocess/toolbox/subpackageA
after executing this command a "subpackageA" folder was created under toolbox with index.html file but it was all blank
Then i tried to generate documentation by providing specific module name
pdoc --html --external-links --all-submodules preprocess/toolbox/submoduleB2.py
but received this below error:
File "C:\Users\preprocess/toolbox/submoduleB2.py", line 16, in
from sub import sub1
ImportError: No module named sub.sub1
Can you please tell me how to generate the documentation using pdoc for complete directory?
Or is there any other package which will auto generate the documentation?
I even tried Sphnix, but faced issues in adding the module/submodule paths in config file

It appears that pdoc3 is throwing that kind of error for a module if it cannot find an import into that module in the python path. One solution is to put
import os, sys
syspath = os.path.dirname(os.path.abspath(__file__))
sys.path.append(path)
into the __init__.py files in each of the subdirectories.

How to check does a file imports from another file in Python

Suppose i have a project structure like this
src
└── app
├── main.py
├── db
│ └── database.py
├── models
│ ├── model_a.py
│ └── model_b.py
└── tests
├── test_x.py
└── test_y.py
I want to check which file uses a class or a function from another file. I have a class called Test in main.py
class Test:
pass
I used that class in model_a,
from ..main import Test
But in model_b i used
from ..main import Test
from ..db.database import Data
I want to to check which file uses another file, just like tree command, just a folder name is enough so i tried an old method but it was inefficient ,dirty and that was not something that i expect. The method was i created a file in src named check.py, i imported all packages
from app.db import database
from app.models import model_a, model_b
from app.tests import test_x, test_y
from app import main
print('__file__={0:<35} | __name__={1:<20} | __package__={2:<20}'.format(__file__,__name__,str(__package__)))
And i added this line in the bottom of all files
print('__file__={0:<35} | __name__={1:<20} | __package__={2:<20}'.format(__file__,__name__,str(__package__)))
So when i run check.py i get this result
__file__=/home/yagiz/Desktop/struct/src/app/main.py | __name__=app.main | __package__=app
__file__=/home/yagiz/Desktop/struct/src/app/db/database.py | __name__=app.db.database | __package__=app.db
__file__=/home/yagiz/Desktop/struct/src/app/models/model_a.py | __name__=app.models.model_a | __package__=app.models
__file__=/home/yagiz/Desktop/struct/src/app/models/model_b.py | __name__=app.models.model_b | __package__=app.models
__file__=/home/yagiz/Desktop/struct/src/app/tests/test_x.py | __name__=app.tests.test_x | __package__=app.tests
__file__=/home/yagiz/Desktop/struct/src/app/tests/test_y.py | __name__=app.tests.test_y | __package__=app.tests
__file__=/home/yagiz/Desktop/struct/src/check.py | __name__=__main__ | __package__=None
The result is dirty and doesn't meet my expectations is there a way to get a output like this?
main.py = app/models/model_a, app/models/model_b # These files imports something from main.py
models_b = None # No file imports from models_b
Update, i tried #Hessam Korki's suggestion it doesn't works.
I looked up the source code of modulefinder and i found it adds a badmodule in every import statement which is not useful for me.
Here is how did it go, first i created a function, also i created an another project structure.
src
├── empty.py
├── __init__.py
├── main.py
├── module_finder.py
├── other
│   └── other.py
├── test
│   └── some_app.py
└── this_imports.py
Here is the module_finder.py that contains my function
from modulefinder import ModuleFinder
file_names = ["this_imports.py", "main.py", "test/some_app.py", "other/other.py", "empty.py"]
def check_imports(file_names):
finder = ModuleFinder()
for file in file_names:
finder.run_script(file)
print("\n", file)
for name, mod in finder.modules.items():
print('%s: ' % name, end='')
print(','.join(list(mod.globalnames.keys())[:3]))
print('\n'.join(finder.badmodules.keys()))
Empty file is empty(as expected), in main.py i have
class Test:
pass
In this_imports.py i only have
from src.main import Test
In other/other.py i have
from src.main import Test
from src.test import DifferentTest
And for the last one in test/some_app.py i have
from src.main import Test
class DifferentTest:
pass
So the result should be:
empty.py = None
main.py = None
other/other.py = src.main , src.test
test/some_app.py = src.main
this_imports.py = src.main
But the function gives a wrong result, here is the output:
Filename: this_imports.py
__main__: Test
src.main
Filename: main.py
__main__: Test,__module__,__qualname__
src.main
Filename: test/some_app.py
__main__: Test,__module__,__qualname__
src.main
Filename: other/other.py
__main__: Test,__module__,__qualname__
src.main
src.test
Filename: empty.py
__main__: Test,__module__,__qualname__
src.main
src.test

I believe python's Modulefinder will effectively solve your problem. There is a key named '__main__' in the Modulefinder().items() which holds the modules that were imported in a python file. After running the script through your project and storing the data in a way that suits your purpose, you should be good to go

What you are looking for is to find import dependencies in your package modules. You can run a static analysis on your package directory and parse the import nodes in the syntax trees (ast), and build a dependency graph. Something like below:
import os
from ast import NodeVisitor, parse
import networkx as nx
class Dependency():
def __init__(self, root):
self.root = root
self.base = os.path.basename(root)
self.dependency = nx.DiGraph()
self.visitor = NodeVisitor()
self.visitor.visit_ImportFrom = self.visit_ImportFrom
self.current_node = None
self.dependency.add_node = self.base
def visit_ImportFrom(self, node):
self.dependency.add_edge(node.module, self.current_node)
self.visitor.generic_visit(node)
def run(self):
for root, dirs, files in os.walk(self.root):
for file in files:
full_path = os.path.join(root+os.sep, file)
loc = full_path.split(self.root+os.sep)[1].replace(os.sep,'.')
self.current_node = self.base+'.'+loc
with open(full_path) as fp:
src = fp.read()
tree = parse(src)
self.visitor.generic_visit(tree)
dependency = {}
for src, target in nx.dfs_edges(self.dependency):
if src in dependency:
dependency[src].add(target)
else:
dependency[src] = set([target])
return dependency
For the root location of any package you want to map the import dependencies, you need to do the following then:
root = "path/to/your/src"
d = Dependency(root)
d.run()
This will return the dependency tree (as a dict). Note, we parsed only ImportFrom, you need to add Import to make it complete. Also, all imports are assumed absolute here (i.e. no .. etc). If required, you can add that too (check the level field of the ImportFrom node to do that).

Python `pkgutil.get_data` disrupts future imports

Consider the following package structure:
.
├── module
│   ├── __init__.py
│   └── submodule
│   ├── attribute.py
│   ├── data.txt
│   └── __init__.py
└── test.py
and the following piece of code:
import pkgutil
data = pkgutil.get_data('module.submodule', 'data.txt')
import module.submodule.attribute
retval = module.submodule.attribute.hello()
Running this will raise the error:
Traceback (most recent call last):
File "test.py", line 7, in <module>
retval = module.submodule.attribute.hello()
AttributeError: module 'module' has no attribute 'submodule'
However, if you run the following:
import pkgutil
import module.submodule.attribute
data = pkgutil.get_data('module.submodule', 'data.txt')
retval = module.submodule.attribute.hello()
or
import pkgutil
import module.submodule.attribute
retval = module.submodule.attribute.hello()
it works fine.
Why does running pkgutil.get_data disrupt the future import?

First of all, this was a great question and a great opportunity to learn something new about python's import system. So let's dig in!
If we look at the implementation of pkgutil.get_data we see something like this:
def get_data(package, resource):
spec = importlib.util.find_spec(package)
if spec is None:
return None
loader = spec.loader
if loader is None or not hasattr(loader, 'get_data'):
return None
# XXX needs test
mod = (sys.modules.get(package) or
importlib._bootstrap._load(spec))
if mod is None or not hasattr(mod, '__file__'):
return None
# Modify the resource name to be compatible with the loader.get_data
# signature - an os.path format "filename" starting with the dirname of
# the package's __file__
parts = resource.split('/')
parts.insert(0, os.path.dirname(mod.__file__))
resource_name = os.path.join(*parts)
return loader.get_data(resource_name)
And the answer to your question is in this part of the code:
mod = (sys.modules.get(package) or
importlib._bootstrap._load(spec))
It looks at the already loaded packages and if the package we're looking for (module.submodule in this example) exists it uses it and if not, then tries to load the package using importlib._bootstrap._load.
So let's look at the implementation of importlib._bootstrap._load to see what's going on.
def _load(spec):
"""Return a new module object, loaded by the spec's loader.
The module is not added to its parent.
If a module is already in sys.modules, that existing module gets
clobbered.
"""
with _ModuleLockManager(spec.name):
return _load_unlocked(spec)
Well, There's right there! The doc says "The module is not added to its parent."
It means the submodule module is loaded but it's not added to the module module. So when we try to access the submodule via module there's no connection, hence the AtrributeError.
It makes sense for the get_data method to use this function as it just wants some other file in the package and there is no need to import the whole package and add it to its parent and its parents' parent and so on.
to see it yourself I suggest using a debugger and setting some breakpoints. Then you can see what happens step by step along the way.

Module imports from another folder

I wanted to write a test program and for this I need to include my DPS310.py file. But somehow it doesn't work
I tried:
• from myPackage import DPS310
• import ..DPS310
• import myPackage.DPS310
My structure:
Projekt
├── myPackage
│   ├── __init__.py
│ └── DPS310.py
├── tests
│ ├── __init__.py
│ └── test_module1.py
├── README.md
├── LICENSE
└── setup.py
test_module1.py File
import myPackage.DPS310
msg = "Hello World"
print(msg)
dps = DPS310()
#y = DPS310.getTemperature
print(dps.getTemperature())
DPS310.py File (Extraction. Just to show that the getTemperature method is in here)
...
class DPS310():
def __init__(self, *args):
self.i2c = smbus2.SMBus(1)
if len(args) > 0:
self.addr = args[0]
print("Man addr set")
else:
# default sensor's address
self.addr = 0x77
def getTemperature(self):
r = self.i2c.read_i2c_block_data(self.addr, DPS310_TMP_B2, 3)
# reads out the temperature that is stored in the register
# r[0]=TMP0-7 r[1]=TMP8-15 r[2]=TMP16-23
temp = (r[2] << 16) | (r[1] << 8) | (r[0]) # r[0] << 0
# deploys this function to 24 bits
return self.twos_comp(temp, 24)
...
If I run the test_module1.py file:
Exception has occurred: ModuleNotFoundError
No module named 'myPackage'
File "C:\Julian\Projects\PhytonLib_Raspberry\DPS310_Library_RPi\tests\test_module1.py", line 1, in <module>
import myPackage.DPS310

No module named 'myPackage' File
Most likely Python doesn't know where to find myPackage, so you could try setting it as a source root (you could look up the relevant info for your IDE). Also, check out this StackOverFlow post.

Module inside module error, python

I created a File named StringPlay.py, and called it upon a file ScanTheFile.py then called a function in ScanTheFile.py in another file named Controller.py. But it raise an error that StringPlay.py does not exist
In ScanTheFile.py
import StringPlay as SP
def TNews(FileFirstLine):
FileLine = SP.RemoveSpases(FileFirstLine)
if True:
Statement
return(FileLine)
In Controller.py
from HelpFiles import StringPlay as SP
from HelpFiles import ScanTheFile as StF
File1 = open("TextFiles\File1.txt")
print(Stf.TNews(File1.readline()))
When this is executed it raises a Message Stating that in ScanTheFile.py in Line 1:
File "C:\Users\***\My Documents\Python\HelpFiles\ScanTheFile.py", line 1, in <module>
import StringPlay as SP
ImportError: No module named 'StringPlay'
But when ScanTheFile.py is being executed there's no problem.
This is my directory structure
My Documents\Python
My Documents\Python\Controller.py
My Documents\Python\HelpFiles
My Documents\Python\HelpFiles\ScanTheFile.py
My Documents\Python\HelpFiles\StringPlay.py
My Documents\Python\TextFiles
My Documents\Python\TextFiles\File1.txt
My Documents\Python\TextFiles\File2.txt

You are having this problem because when you run Controller.py you are running it in say C:\...\My Documents when you call from HelpFiles import ScanTheFile as StF you are telling python to look in C:\...\My Documents\HelpFiles for a file named ScanTheFile.py.
It finds this file and runs it in C:\...\My Documents so when it hits the line in ScanTheFile.py:
import StringPlay as SP
it looks in C:\...\My Documents for StringPlay.py which from what I can gather exists in C:\...\My Documents\HelpFiles.
Since you are already importing StringPlay.py explicitly in Controller.py I would say just change ScanTheFile.py to only import StringPlay.py when it is the file being run directly, and not imported:
ScanTheFile.py
if __name__ == "__main__":
import StringPlay as SP
def TNews(FileFirstLine):
FileLine = SP.RemoveSpases(FileFirstLine)
if True:
Statement
return(FileLine)

You are importing from HelpFiles which is basically a folder not a file. To put it simply, in python if you want to import something from a folder, it has to be a package unless you add the path to the folder to python path explicitly.
So, to solve the problem you are having make HelpFiles a package by putting an empty file with name __init__.py in that folder which should look something like this
.
└── My Document
└── Python
├── Controller.py
├── HelpFiles
│   ├── __init__.py
│   ├── ScanTheFile.py
│   └── StringPlay.py
└── TextFiles
├── File1.txt
└── File2.txt