Pytest output interpretation

pytest-output-interpretation

Summary

Interpret pytest test execution output to determine whether a Python package's bundled test suite passes or fails, and extract diagnostic information from the test report. This skill is essential for validating package correctness and identifying test failures during development or continuous integration workflows.

When to use

After running pytest tests/ on a Python package (especially one with dev extras installed), you need to determine if the full test suite passed, identify which tests failed, and extract error messages or stack traces for debugging. Use this skill when you have concrete pytest console output or a test report file and need to synthesize a pass/fail verdict and diagnostic summary.

When NOT to use

• The test suite has not yet been run; you need to execute pytest first.
• You are trying to modify or write new tests rather than interpret existing results.
• The output is from a different test runner (e.g., unittest, nose, tox) — this skill is pytest-specific.

Inputs

• pytest console output (stdout/stderr)
• pytest test report file (e.g., JUnit XML, JSON, or terminal summary)
• test result summary line from pytest

Outputs

• pass/fail verdict (boolean or enum)
• total test count (integer)
• passed test count (integer)
• failed test count (integer)
• skipped test count (integer)
• list of failed test names and error messages (structured)
• diagnostic summary (string)

How to apply

Execute the test suite using pytest tests/ from the package root (after installing in development mode with pip install -e .[dev]), then parse the final summary line and any FAILED markers in the output. Look for the summary statement that reports the total number of passed, failed, skipped, or errored tests (e.g., '42 passed', '2 failed'). If the exit code is 0 and the summary shows no failures, the test suite passed. If failures or errors are reported, read the FAILED section and the traceback for each failing test to extract the assertion error or exception type. Record the specific test names, error messages, and any relevant stack trace context to guide remediation. The test pass/fail status directly validates package correctness and reproducibility.

Related tools

• pytest (test runner; executes the bundled test suite and produces structured output with pass/fail status and failure diagnostics) — https://pytest.readthedocs.io/
• pip (package manager; installs the package in development mode with test dependencies before pytest execution) — https://pip.pypa.io/
• Python (runtime environment; executes pytest and the test suite) — https://www.python.org/

Examples

pytest tests/

Evaluation signals

• Exit code is 0 (success) or non-zero (failure) — the process exit code reliably indicates pass/fail
• The final pytest summary line is present and parseable (e.g., 'X passed', 'X failed', 'X error') — absence suggests incomplete output
• All test node IDs in the FAILED section match patterns in the tests/ directory (tests/test_*.py::TestClass::test_method) — validates test identification
• Error messages and stack traces are present for each failed test and contain recognizable Python exception types (AssertionError, ImportError, etc.) — confirms diagnostic completeness
• Test counts are consistent: passed + failed + skipped ≤ total tests run — validates internal consistency of the report

Limitations

• pytest output format may vary by version, verbosity flag, or plugin configuration (e.g., -v, -vv, --tb=short), requiring robust parsing.
• Flaky or nondeterministic tests may pass or fail on successive runs, making a single test run an unreliable validator of package state.
• Pytest may be interrupted or killed before completion, producing truncated output that cannot be fully interpreted.
• Conditional test skips (marked with @pytest.mark.skip or platform/dependency conditions) mean the absence of a test result does not imply failure.

Evidence

• [other] The biosynfoni repository displays a Tests badge linked to a GitHub Actions workflow (test-biosynfoni.yml) that validates the package's test suite status.: "The biosynfoni repository displays a Tests badge linked to a GitHub Actions workflow (test-biosynfoni.yml) that validates the package's test suite status."
• [other] Run the full test suite using pytest on the tests/ directory (pytest tests/). Verify that all tests pass and report the test outcome.: "Run the full test suite using pytest on the tests/ directory (pytest tests/). Verify that all tests pass and report the test outcome."
• [other] Install the package in editable/development mode using pip with the dev extras (pip install -e .[dev]).: "Install the package in editable/development mode using pip with the dev extras (pip install -e .[dev])."
• [other] You can also run the tests locally with the following command: pytest tests/: "You can also run the tests locally with the following command: pytest tests/"