Documentation generation with sphinx

documentation-generation-with-sphinx

Summary

Generate browsable HTML API documentation from Python source code using Sphinx and sphinx-apidoc. This skill automates the extraction of docstrings and module structure into reStructuredText stubs, then compiles them into a static HTML documentation site.

When to use

You have a Python package with docstrings and want to produce publicly browsable API documentation. Typical triggers: (1) package is being released or deployed; (2) you need to host documentation on ReadTheDocs or similar; (3) you want to generate API reference alongside hand-written guides (tutorials, methods). This is most useful when documentation source files (conf.py, index.rst, tutorials) already exist in a docs/ directory and you need to regenerate or update the API stubs from source code changes.

When NOT to use

• Your package uses a different documentation framework (e.g., MkDocs, Jupyter Book, pdoc) — Sphinx is not universal.
• You need dynamic, runtime-generated documentation (e.g., API introspection on-the-fly) — Sphinx generates static HTML only.
• Your source code lacks docstrings or uses non-standard docstring formats that Sphinx autodoc does not parse correctly.

Inputs

• Python source package directory (e.g., ../dreams)
• Sphinx configuration file (conf.py) with theme, extensions, and metadata
• requirements.txt listing Sphinx, sphinx-apidoc, and other build dependencies
• Hand-written reStructuredText files (index.rst, tutorials, guides) in docs/
• External tutorial or documentation folders to be linked (optional)

Outputs

• Generated API documentation .rst stub files in docs/ (one per module/class hierarchy)
• Compiled HTML site in _build/html/ with index.html as entry point
• Cross-referenced API reference with searchable module and function indices
• Static files (CSS, JavaScript, images) for browser navigation

How to apply

First, ensure Python dependencies (including Sphinx and sphinx-apidoc) are installed via pip install -r requirements.txt in the docs directory. Create a symbolic link from external tutorial folders into the docs directory (e.g., ln -s ../tutorials tutorials) so they appear in the build. Run sphinx-apidoc -o . ../dreams to scan the source module (e.g., ../dreams) and generate .rst stub files in the current docs directory; this introspects all modules, classes, and functions and writes their signatures and docstrings into reStructuredText format. Finally, execute make html to invoke Sphinx's build system, which processes all .rst files (both generated stubs and hand-written pages) and outputs a complete, cross-linked HTML site to _build/html/. The build is idempotent and will only rebuild changed files.

Related tools

• Sphinx (Documentation generator; processes .rst source files and themes to produce HTML, PDF, or other outputs. Invoked via make html after sphinx-apidoc generates stubs.) — https://www.sphinx-doc.org/
• sphinx-apidoc (Scans Python source directory, introspects modules and docstrings, and auto-generates reStructuredText stub files for Sphinx to consume.) — https://www.sphinx-doc.org/en/master/man/sphinx-apidoc.html
• pip (Package manager; installs Sphinx, sphinx-apidoc, and other build dependencies from requirements.txt.) — https://pip.pypa.io/

Examples

cd docs && pip install -r requirements.txt && ln -s ../tutorials tutorials && sphinx-apidoc -o . ../dreams && make html && open _build/html/index.html

Evaluation signals

• Verify that _build/html/index.html exists and opens in a browser without 404 errors.
• Check that the API reference index (e.g., _build/html/modules.html) lists all expected modules and classes from the source package.
• Confirm that cross-references between modules (e.g., class inheritance, function parameter types) are rendered as hyperlinks in the HTML.
• Validate that docstrings from source code appear verbatim in the corresponding .rst stubs and are rendered in the final HTML.
• Ensure that hand-written tutorial content (e.g., ../tutorials linked via ln -s) appears in the generated HTML navigation and is searchable.

Limitations

• Sphinx autodoc relies on importable, syntactically correct Python code; broken imports or syntax errors in source modules will cause sphinx-apidoc to fail or skip those modules.
• Docstring extraction quality depends on compliance with a standard format (NumPy, Google, or reStructuredText); non-standard or missing docstrings will appear incomplete or blank in the HTML output.
• Symbolic links (ln -s) for tutorial folders may not be portable across operating systems or file systems; absolute paths in docs configuration may be more robust.
• Large source packages with hundreds of modules can produce very large HTML sites; consider filtering or excluding modules via sphinx-apidoc options (-e, --exclude-patterns) to keep builds fast.

Evidence

• [other] Generate API documentation stubs by running sphinx-apidoc to scan the ../dreams module and output documentation source files.: "Generate API documentation stubs by running sphinx-apidoc to scan the ../dreams module and output documentation source files."
• [other] Build HTML documentation using make html, which processes Sphinx source files and generates the browsable output in _build/html/.: "Build HTML documentation using make html, which processes Sphinx source files and generates the browsable output in _build/html/."
• [methods] sphinx-apidoc -o . ../dreams && make html: "sphinx-apidoc -o . ../dreams && make html"
• [other] Install Python dependencies from requirements.txt using pip.: "Install Python dependencies from requirements.txt using pip."
• [other] Create a symbolic link from the tutorials folder to the current documentation directory using ln -s.: "Create a symbolic link from the tutorials folder to the current documentation directory using ln -s."