Pytest test suite execution and result interpretation

pytest-test-suite-execution-and-result-interpretation

Summary

Execute a pytest test suite via Hatch to validate code correctness and identify test failures in a development environment. This skill ensures CI/CD reproducibility by running all configured tests and interpreting exit codes and failure reports.

When to use

After forking and cloning a repository (e.g., scverse/scanpy) to verify that the development environment is correctly configured, or after implementing a feature or bugfix to ensure no regressions were introduced. Use this skill whenever you need to reproduce the CI workflow locally or validate that a code change does not break existing functionality.

When NOT to use

• When the development environment is not yet created or Hatch is not installed — first ensure Hatch is available and hatch.toml exists in the repository root.
• When only a subset of tests is needed and the full test suite would be prohibitively slow — use -k pattern matching or specify individual test files instead of running the full suite.
• When testing against a specific older version of a dependency that conflicts with the Hatch environment specification — use a manual virtual environment or a custom pytest invocation instead.

Inputs

• cloned repository with hatch.toml configuration
• pytest test suite in scanpy/tests directory
• matplotlib reference images (for plot comparison tests)

Outputs

• test execution report with pass/fail counts
• exit code (0 for success, non-zero for failure)
• pytest log with detailed failure messages and tracebacks
• image comparison results (for plot-based tests)

How to apply

Clone the repository and navigate to the root directory. Invoke hatch test to automatically create the predefined Hatch environment specified in hatch.toml and execute the full pytest test suite on all tests in the scanpy/tests directory with the configured pytest settings. Monitor the exit code (0 indicates all non-skipped tests passed). Inspect the test output for failure counts and error messages. For rapid iteration during development, run a subset of tests using -k patterns or specific test file names (e.g., hatch test test_plotting.py). For visual regression testing, verify that any custom matplotlib plot reference images match expected outputs via the image_comparer fixture. A test run is considered successful when it completes with exit code 0 and reports zero non-skipped test failures.

Related tools

• pytest (Test framework that discovers and executes unit and integration tests; configured via pytest settings in hatch.toml)
• Hatch (Project environment and dependency manager that creates isolated Python environments and provides the hatch test command to run the test suite with predefined configuration)
• git (Version control tool used to clone the repository and manage branches during development)
• matplotlib (Plotting library; its image_comparer fixture validates that generated plots match reference images)
• Scanpy (The target package being tested; test suite validates preprocessing, visualization, clustering, trajectory inference, and differential expression functionality) — https://github.com/scverse/scanpy

Examples

hatch test

Evaluation signals

• Test execution completes with exit code 0 and reports zero non-skipped test failures.
• All test files in the scanpy/tests directory are discovered and executed by pytest without import or collection errors.
• Custom matplotlib plot reference images generated during test execution match the expected outputs as verified by the image_comparer fixture.
• No unexpected warnings or deprecation notices appear in the test output that would indicate breaking changes in dependencies.
• Optional: subset tests using -k patterns execute faster and still report correct pass/fail status for the filtered test set.

Limitations

• The test suite execution time may be substantial for large codebases; use -k pattern matching or individual test files for faster feedback during rapid development iteration.
• Image comparison tests (matplotlib plots) are sensitive to font rendering, backend configuration, and system-level graphics settings; the image_comparer fixture may produce false negatives on different operating systems or graphics environments.
• Exit code 0 confirms that tests passed but does not guarantee code correctness or comprehensive coverage; review test coverage reports and manually validate critical functionality.
• Hatch environment isolation may mask dependency conflicts that would appear in production environments; consider running tests in multiple isolated environments or with different dependency versions for robustness.

Evidence

• [other] Scanpy uses pytest for testing and provides a Hatch environment configuration to execute the test suite via the hatch test command.: "Scanpy uses pytest for testing and provides a Hatch environment configuration to execute the test suite via the hatch test command."
• [other] Execute the full pytest test suite via hatch test, which runs all tests in the scanpy/tests directory with the configured pytest settings.: "Execute the full pytest test suite via hatch test, which runs all tests in the scanpy/tests directory with the configured pytest settings."
• [other] Verify the test run completes with exit code 0 and reports zero non-skipped test failures.: "Verify the test run completes with exit code 0 and reports zero non-skipped test failures."
• [other] Run a subset of tests using -k patterns or specific test files (e.g., hatch test test_plotting.py) for faster iteration during development.: "Run a subset of tests using -k patterns or specific test files (e.g., hatch test test_plotting.py) for faster iteration during development."
• [other] Check that any custom matplotlib plot reference images match expected outputs via the image_comparer fixture.: "Check that any custom matplotlib plot reference images match expected outputs via the image_comparer fixture."
• [other] We use pytest to test scanpy. To run the tests, simply run hatch test: "We use pytest to test scanpy. To run the tests, simply run hatch test"
• [other] Create a development environment using Hatch by running hatch test from the repository root, which automatically creates the predefined Hatch environment specified in hatch.toml.: "Create a development environment using Hatch by running hatch test from the repository root, which automatically creates the predefined Hatch environment specified in hatch.toml."