We Keep Coding

python pdoc the whole project

lately i've started using pdoc3 to document my whole project,
right now i have a problem generating a HTML that contain the whole project in 1 command, and not going every.py file one by one
the project structure is like this:
--pythonProject
| file_a.py
| file_b.py
--folder
| file_c.py
| file_d.py
--docs
i tried the following command : pdoc --html .\file_a.py --output-dir docs --force and it genetrates the following file in docs folder -> file_a.html
and when using this pdoc --html .\ --output-dir docs --force i receive error for pytest library that not even in the project
i wanted to know if there is a command to generate the whole project (im using PyCharm)
thanks alot :)

MonkeyType: No traces found for module src.projectname.__main__

Context
While trying to generate type hints for all python files in some project named projectname, I am experiencing some difficulties applying the stubs. The project directory is:
projectname/
|-- src/
| |-- projectname/__main__.py
| |-- projectname/some_folder/some_script.py
|-- tests/
| |-- projectname/test_something.py
| |-- projectname/another_folder/test_another_thing.py
|
|-- setup.py
|-- README.md
The main code is executed with:
python -m src.projectname
and the tests are executed with:
python -m pytest
Output Attempt I
Based on this issue, I tried running:
monkeytype run src/snncompare/__main__.py
Which generates the monkeytype.sqlite3 file in the root directory of the project. However, when I try to apply the generated type hints with:
monkeytype stub src.snncompare.__main__.py
monkeytype apply src.snncompare.__main__.py
or:
monkeytype stub src.snncompare.__main__
monkeytype apply src.snncompare.__main__
I get:
No traces found for module src.projectname.__main__
No traces found for module src.projectname.__main__.py
respectively. And for:
monkeytype apply src.snncompare
It says:
No traces found for module src.snncompare
Output Attempt II
Based on this issue, in which it says one should apply the stub to the "modulename" I tried running:
monkeytype run src/projectname/some_folder/some_script.py
monkeytype run src/projectname.some_folder.some_script.py
monkeytype run src/projectname.some_folder.some_script
monkeytype run some_script.py
monkeytype run some_script
from the root directory of the script, and they all produce the same error. I do not yet know exactly how to determine what the "modulename" of my project is.
Attempt III
Based on this answer, I think I should create an additional file that calls the __main__.py file and executes its code, to be able to generate stubs for it, or the modules imported by it. However, that would seem in conflict with the quote:
We don't collect traces for main because it could be different on subsequent runs and it would be confusing to look up such traces.
Question
How can I apply the generated stubs to all the (touched) .py files in the project?

Misunderstanding
I think the key misunderstanding I had can be clarified with:
MonkeyType currently does not automatically create typings for a Python project.
Issue
As described in the issues and the readme, monkey type only generates type hints for the some.py files that a main.py file imports and uses. Even though I am relatively confident that a large fraction of my files are used when I run __main__.py, most of them are not directly imported by __main__.py. Instead, they are imported by the code that is imported by my __main__.py.
Manual-"Solution"
To "automatically"/semi-manually generate the type hints, you need to write another_python.py file that:
imports each other.py file for which you want to automatically create type-hints.
Calls each function in every other.py file for which you want to automatically create type-hints.
Bit of automation
Once could at least automate applying the stubs that you did find by walking over the directories and trying to apply the stubs if they exists:
monkeytype run src/projectname/__main__.py
# List all .py files in the project:
for f in $(find src/ -name '*.py'); do echo $f; done
# List all .py files in the project with `.` instead of `/`:
for f in $(find src/ -name '*.py'); do echo ${f//\//.}; done
# Apply monkeytype type hints to each file for which they were found.
for f in $(find src/ -name '*.py'); do monkeytype apply ${f//\//.}; done
Additional Issues
I had a file src/projectname/some_file.py which contained some_function(): which was definitely called by __main__.py yet no stubs like: some_function() -> None: were generated for this file. In total 1 out of 50 files were changed. I retried auto generating the stubs using the tests I wrote, with pyannotate by dropbox and that worked more effectively.

Sphinx: the simplest usage scenario, how to? [duplicate]

I am running Sphinx on a rst file containing automodule but it does not seem to have any effect.
Here are the details: I have a Python project with a file agent.py containing a class Agent in it. I also have a subdirectory apidoc with a file agent.rst in it (generated by sphinx-apidoc):
agent module
============
.. automodule:: agent
:members:
:undoc-members:
:show-inheritance:
I run sphinx with sphinx-build -b html apidoc apidoc/_build with the project's directory as the current working directory.
To make sure the Python files are found, I've included the following in apidoc/conf.py:
import os
import sys
sys.path.insert(0, os.path.abspath('.'))
It runs without errors but when I open the resulting HTML file it only shows "agent module" and everything is blank. Why isn't it showing the class Agent and its members?
Update: the original problem was likely caused by the fact that I had not included sphinx.ext.autodoc in conf.py. Now that I did, though, I get warnings like:
WARNING: invalid signature for automodule ('My Project.agent')
WARNING: don't know which module to import for autodocumenting 'My Project.agent' (try placing a "module" or "currentmodule" directive in the document, or giving an explicit module name)
WARNING: autodoc: failed to import module 'agent'; the following exception was raised:
No module named 'agent'

I'll try answering by putting the "canonical" approach side-by-side with your case.
The usual "getting started approach" follows these steps:
create a doc directory in your project directory (it's from this directory the commands in the following steps are executed).
sphinx-quickstart (choosing separate source from build).
sphinx-apidoc -o ./source ..
make html
This would yield the following structure:
C:\Project
|
| agent.py
|
|---docs
| | make.bat
| | Makefile
| |
| |---build
| |
| |---source
| | conf.py
| | agent.rst
| | index.rst
| | modules.rst
In your conf.py you'd add (after step 2):
sys.path.insert(0, os.path.abspath(os.path.join('..', '..')))
and in index.rst you'd link modules.rst:
Welcome to Project's documentation!
================================
.. toctree::
:maxdepth: 2
:caption: Contents:
modules
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
Now compare the above with what you have - from what you shared in your question:
C:\Project
|
| agent.py
|
|---apidoc
| | agent.rst
| | conf.py
| |
| |-- _build
You ran:
sphinx-build -b html apidoc apidoc/_build
and in your conf.py:
sys.path.insert(0, os.path.abspath('.'))
Your error stacktrace says it couldn't find the module agent. That's probably because you didn't go 1 level down in your conf.py (it's pointing to the path with .rst, not the path with .py), this should work:
sys.path.insert(0, os.path.abspath('..')). Also, if you didn't manually edit/connect your modules.rst in your index.rst you are likely to only see that module.
You may care to notice the signatures of the sphinx commands at play:
sphinx-apidoc [OPTIONS] -o <OUTPUT_PATH> <MODULE_PATH>
sphinx-build [options] <sourcedir> <outputdir> [filenames …]
<sourcedir> refers to where .rst are, and <MODULE_PATH> to where .py are. <OUTPUT_PATH> to where .rst are placed, and <outputdir> to where .html are placed.
Please also notice, you mentioned: "the project's directory as the current working directory." I've seen "working directory" mentioned in sphinx threads on stackoverflow, interchangeably as both the Project base directory, or the docs directory. However, if you search the Sphinx documentation for "working directory" you'll find no mention of it.
Finally, there is an advantage to using the file/directory structure of the "getting started approach". It basically "puts you on the same page" with most threads on the Sphinx tag, and that way alleviates the mental work of mapping the cases to different directory/file structures.
I hope this helps.

Sphinx index.rst inside directory

The path to the master doc of a Sphinx build can be specified in conf.py. However, this directory path is reflected in the generated HTML, and shows for example in Read The Docs as a missing index. I'd like to use this alternative path to structure my project cleanly - to have configuration at the top level and documentation inside src, but have the build essentially get rid of it.
So here's what I had previously:
docs
conf.py
index.rst
things
doc1.rst
doc2.rst
This works, but when lots of files are added to the top level, it gets messy with Sphinx's makefiles etc. I'd like to have this instead:
docs
conf.py
src
index.rst
things
doc1.rst
doc2.rst
Which builds, but index.html is in build/html/src instead of build/html. I was surprised to find no information on this, other than the fact that master_doc in conf.py controls the location and name of the main file. How could I get my documentation to be built to build/html?

There is an option for specifying the location of the configuration file: -c.
# conf.py
master_doc = 'index'
# structure
docs
conf.py
src
index.rst
...
Then run sphinx-build -b html -c . src build/html. However, this solution indeed needs control over the build command, which is not available in Read The Docs. And it seems that -c doesn't work in Sphinx's own makefiles either (with -M instead of -b).

How to run python test files

I am developing python in eclipse. As a result, python src files and test files are in different directories.
Question is: How do we run on command line specific test files in the test folder? These obviously depend on files in the src folder.
Cheers
Edit: if I run
python test/myTestFile.py
I get dependency errors, eg. ImportError: No module named SrcFile1

You need to make sure your PYTHONPATH is set correctly so the command-line interpreter can find your packages, or run your test cases from within Eclipse Pydev. Update: Another option: running your tests using nose might make things a bit easier, since it can auto-discover packages and test cases.
If your project is laid out like so:
/home/user/dev/
src/pkg1/
mod1.py
test/
mod1_test.py
Use: PYTHONPATH=$HOME/dev/src python test/mod1_test.py. I'd also recommend using distribute and virtualenv to set up your project for development.
Updated in response to question in comments:
This shows how the PYTHONPATH environment variable extends Python's package sear
ch path:
% PYTHONPATH=foo:bar python -c 'import sys; print sys.path[:3]'
['', '/home/user/foo', '/home/user/bar']
# exporting the variable makes it sticky for your current session. you can
# add this to your shell's resource file (e.g. ~/.profile) or source
# it from a textfile to save typing:
% export PYTHONPATH=bar:baz
% python -c 'import sys; print sys.path[:3]'
['', '/home/user/foo', '/home/user/bar']
% python -c 'import sys; print sys.path[:3]'
['', '/home/user/foo', '/home/user/bar']
The above should get you going in the short term. Using distribute and
virtualenv have a higher one-time setup cost but you get longer-term benefits
from using them. When you get a chance, read some of the many tutorials on SO for setting these up to see if they're a good fit for your project.

There are 2 principal solutions to this. Either, you need to use e.g. PYTHONPATH environment variable to tell the tests where the source is, or you need to make tests and production code part of the same module tree by inserting the relevant __init__.py files. In the latter approach, the tree may look something like this:
|-- qbit
| |-- __init__.py
| |-- master.py
| |-- policy.py
| |-- pool.py
| |-- synchronize.py
| `-- worker.py
`-- test
|-- __init__.py
|-- support.py
|-- test_policy.py
|-- test_synchronize.py
`-- test_worker.py
__init__.py can be an empty file.