We Keep Coding

pdoc3 or Sphinx for directory with nested module

My code directory looks like below. I need to generate documentation for all the modules like for sub1,sub2,submoduleA1,submoduleB1 and so on.
Also as shown for submoduleB2.py: all the modules imports from other modules/submodules
<workspace>
└── toolbox (main folder)
├── __init__.py
│
├── sub
│ ├── __init__.py
│ ├── sub1.py
│ └── sub2.py
│
├── subpackageA
│ ├── __init__.py
│ ├── submoduleA1.py
│ └── submoduleA2.py
│
└── subpackageB
├── __init__.py
├── submoduleB1.py
└── submoduleB2.py code[from sub import sub1
from subpackageA import submoduleA2 and so on]
code structure for submoduleB2.py
from __future__ import absolute_import, division
import copy
import logging
import numpy as np
import pandas as pd
from dc.dc import DataCleaning
from sub.sub1 import ToolboxLogger
from subpackageA import pan
LOGGER = ToolboxLogger(
"MATH_FUNCTIONS", enableconsolelog=True, enablefilelog=False, loglevel=logging.DEBUG
).logger
"""
Calculations also take into account units of the tags that are passed in
"""
def spread(tag_list):
"""
Returns the spread of a set of actual tag values
:param tag_list: List of tag objects
:type tag_list: list
:return: Pandas Series of spreads
:rtype: Pandas Series
:example:
>>> tag_list = [tp.RH1_ogt_1,
tp.RH1_ogt_2,
tp.RH1_ogt_3,
tp.RH1_ogt_4,
tp.RH1_ogt_5,
tp.RH1_ogt_6]
>>> spread = pan.spread(tag_list)
"""
# use the same units for everything
units_to_use = tag_list[0].units
idxs = tag_list[0].actuals.index
spread_df = pd.DataFrame(index=idxs)
spread_series = spread_df.max(axis=1).copy()
return Q_(spread_series, units_to_use)
I tried to run the pdoc command using anaconda prompt by navigating it to the toolbox folder and executed the below command
pdoc --html --external-links --all-submodules preprocess/toolbox/subpackageA
after executing this command a "subpackageA" folder was created under toolbox with index.html file but it was all blank
Then i tried to generate documentation by providing specific module name
pdoc --html --external-links --all-submodules preprocess/toolbox/submoduleB2.py
but received this below error:
File "C:\Users\preprocess/toolbox/submoduleB2.py", line 16, in
from sub import sub1
ImportError: No module named sub.sub1
Can you please tell me how to generate the documentation using pdoc for complete directory?
Or is there any other package which will auto generate the documentation?
I even tried Sphnix, but faced issues in adding the module/submodule paths in config file

It appears that pdoc3 is throwing that kind of error for a module if it cannot find an import into that module in the python path. One solution is to put
import os, sys
syspath = os.path.dirname(os.path.abspath(__file__))
sys.path.append(path)
into the __init__.py files in each of the subdirectories.

Pytest and submodules

I am trying to run pytest tests on my python modules but am running into an error. It looks like it the main script ircFriend.py can't find the modules I import inside of it. This is the error I get. I get this error on every test.
______________________________________________ ERROR collecting test/configuration_test.py ____________________________________
ImportError while importing test module 'C:\Users\munded\Desktop\irc-friend\test\configuration_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
..\..\appdata\local\programs\python\python38\lib\importlib\__init__.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
test\configuration_test.py:2: in <module>
from src import ircFriend
E ModuleNotFoundError: No module named 'configuration'
This is the file structure I am using for my tests. The __init__.py files are empty.
├───src
│ │ configuration.py
│ │ ircFriend.py
│ │ ircSync.py
│ │ logbook.py
│ │ networkdrive.py
│ │ server.py
│ │ tree.py
│ │ workspace.py
│ │ __init__.py
└───test
│ configuration_test.py
│ fileIO_test.py
│ sandbox_test.py
│ server_test.py
│ sync_test.py
│ __init__.py
If we look at the imports in ircFriend.py they look like this.
import sys
import getopt
import logging
from configuration import Configuration
from logbook import LogBook
from networkdrive import NetworkDrive
from ircSync import IRCSync
from workspace import Workspace
from server import Server
Finaly thees are what my tests look like.
from src import ircFriend
from unittest import mock
from src import configuration
from src import server
#mock.patch('builtins.input', side_effect=['X'])
def testPropertiesFileExists(mockInput):
conf = Configuration()
assert conf.propertiesFileExists() is True
#mock.patch('builtins.input', side_effect=['X'])
def testIrcConfigExists(mockInput):
conf = Configuration()
assert conf.ircConfigExists() is True
#mock.patch('builtins.input', side_effect=['devsite.dev', 'user'])
#mock.patch('src.ircFriend.getpass.getpass', return_value="IDK")
def testServerCreation(mock_input, mock_getpass):
dev = Server()
if isinstance(dev, ircFriend.Server):
assert True
else:
assert False
Any guidence on this subject would do me a world of good.
Best Regards,
E

You should not make both src/__init__.py and test/__init__.py files because these src and test are not packages. These are just root directories for source and test codes.
In test codes, You should remove from src because src is not a package.
Finally, run pytest adding src to PYTHONPATH otherwise pytest can't find modules under the src directory.
$ PYTHONPATH=src pytest test
Or, You can make src/conftest.py, this is a special file for pytest.
I checked these codes.
# test/conftest.py
import sys
sys.path.append("./src")
# src/a.py
from b import say
def func():
return say()
# src/b.py
def say():
return "Hello"
# test/test_a.py
import a
def test_a():
assert a.func() == "Hello"
$ pytest test

Module imports from another folder

I wanted to write a test program and for this I need to include my DPS310.py file. But somehow it doesn't work
I tried:
• from myPackage import DPS310
• import ..DPS310
• import myPackage.DPS310
My structure:
Projekt
├── myPackage
│   ├── __init__.py
│ └── DPS310.py
├── tests
│ ├── __init__.py
│ └── test_module1.py
├── README.md
├── LICENSE
└── setup.py
test_module1.py File
import myPackage.DPS310
msg = "Hello World"
print(msg)
dps = DPS310()
#y = DPS310.getTemperature
print(dps.getTemperature())
DPS310.py File (Extraction. Just to show that the getTemperature method is in here)
...
class DPS310():
def __init__(self, *args):
self.i2c = smbus2.SMBus(1)
if len(args) > 0:
self.addr = args[0]
print("Man addr set")
else:
# default sensor's address
self.addr = 0x77
def getTemperature(self):
r = self.i2c.read_i2c_block_data(self.addr, DPS310_TMP_B2, 3)
# reads out the temperature that is stored in the register
# r[0]=TMP0-7 r[1]=TMP8-15 r[2]=TMP16-23
temp = (r[2] << 16) | (r[1] << 8) | (r[0]) # r[0] << 0
# deploys this function to 24 bits
return self.twos_comp(temp, 24)
...
If I run the test_module1.py file:
Exception has occurred: ModuleNotFoundError
No module named 'myPackage'
File "C:\Julian\Projects\PhytonLib_Raspberry\DPS310_Library_RPi\tests\test_module1.py", line 1, in <module>
import myPackage.DPS310

No module named 'myPackage' File
Most likely Python doesn't know where to find myPackage, so you could try setting it as a source root (you could look up the relevant info for your IDE). Also, check out this StackOverFlow post.

Python can't find my packages:

I have been able to run a web app I wrote in bottle from my IDE. It works well, and when I tried to move it to a new system, it broke (couldn't find the files I wrote in the lib folder).
How do I get it to find my lib files? I found this for bottle:
bottle.TEMPLATE_PATH.insert(0,'/absolut/path/to/templates/')
And I think that will help with the missing .tpl files, but how do I do this for python in general? Can I add some sort of python.PATH.insert() to my main.py?
Here is my dir structure:
DEV
├───.idea
│ ├───inspectionProfiles
│ └───scopes
└───myProject *(Also has a .py file I need)*
├───output
│ └───processed
└───webapp
├───src
│ ├───lib
│ │ └───*(All my .py files I need)*
│ ├───static
│ │ ├───css
│ │ ├───files
│ │ ├───img
│ │ └───js
│ └───views *(All the .tpl files I need)*
│ ├───main
│ ├───popup
│ └───reports
└───main.py *(The file that acesses the .py files)*
Relevant Code:
import threading
import time
import datetime
import crud_ops
from helper_functions import load_config_file, dt_ona_sum_format
from server import run_bottle
from myProject import ona_sum_tool #THIS LINE HERE
...
def run_queue(col):
while(1):
if not col:
print "Unable to connect to db."
else:
my_limit = 10
processing_cursor = col.queue.find({"status":"processing"}).limit(my_limit)
if not processing_cursor.count():
queued_cursor = col.queue.find({"status":"queued"}).limit(my_limit)
if queued_cursor.count():
for doc in queued_cursor:
print col.queue.update({"_id":doc['_id']},{"$set":{"status":"processing"}} )
# print col.queue.update({"status":"queued"}, {"$set":{"status":"processing"}}).limit(10)
processing_cursor = col.queue.find({"status":"processing"})
if processing_cursor.count():
time.sleep(1)
for doc in processing_cursor:
############################# THIS LINE HERE ######################
new_file_name = ona_sum_tool.run_summary(dt_ona_sum_format(doc['start']), dt_ona_sum_format(doc['end']))
# print "new_file_name: ", new_file_name
old_id = doc['_id']
# print old_id
doc['_id'] = str(new_file_name)
doc['status'] = 'done'
insert_result = col.queue.insert(doc)
if(insert_result):
col.queue.remove({"_id":old_id})
Error:
Traceback (most recent call last):
File "main.py", line 5, in <module>
from lib.threads import ConnectToDBThread, StartBottleThread, ProcessOutputController, \
File "C:\dev\myProject\myProject\webapp\src\lib\threads.py", line 10, in <module>
from myProject import ona_sum_tool
ImportError: No module named onager

Perhaps you have forgotten to put the the __init__.py file in your lib folder. Here's why you need the __init__.py file:
The init.py files are required to make Python treat the directories as containing packages; this is done to prevent directories with a common name, such as string, from unintentionally hiding valid modules that occur later on the module search path. In the simplest case, init.py can just be an empty file, but it can also execute initialization code for the package or set the all variable, described later.

How to load module with same name as other module in Python?

Let me explain problem - we have such project:
model/__init__.py
model/abstract.py
task/__init__.py
task/model.py
How to load into task/model.py model.abstract what is the syntax for it?
# task/model.py
import model # it loads task/model.py not model
from model.abstract import test # there is no test exception
# model/abstract.py
test = 1
How to do such import?
Requested more details.
Google App Engine application:
- main is main.py
Directory structure:
└───src
│ app.yaml
│ index.yaml
│ main.html
│ main.py
│ task_master_api.py
│
├───circle
│ model.py
│ __init__.py
│
├───model
│ abstract.py
│ xxx.py
│ __init__.py
│
├───task
│ model.py
│ __init__.py
│
├───user
│ model.py
│ __init__.py
Exception (see task.model not model in root):
from .. import model
logging.critical((type(model), model.__name__))
from model.abstract import AbstractNamed, AbstractForgetable
-
CRITICAL 2014-02-17 21:23:36,828 model.py:8] (<type 'module'>, 'task.model')
from model.abstract import AbstractNamed, AbstractForgetable
ImportError: No module named abstract
Much more related to answer.
from .. import model
Gives exception.
ValueError: Attempted relative import beyond toplevel package

While the relative imports in ndpu's answer should work, the answer to this question that is burning in my mind is simply this: change the name of your files to avoid this error.
If you have model.py inside the circle directory, how about changing the name to circle_model.py?
Then, you should be able to import modules without any of the relative import .. business.
Edit - knowing now that you don't want to rename
Make sure you have an __init__.py file in your src directory, then try the relative import from .model.abstract import test

Relative import given in the other answer should just work fine. But it is not working because you have a name conflict. You have both a package and module named model. try to use another name either for your package or module.

I found two tricks to force load modele name into module name:
First forcing only absolute loading:
from __future__ import absolute_import
import name
Second is like previous but more code and more local impact:
save_path = sys.path[:]
sys.path.remove('')
import name
sys.path = save_path