agentsclimarketplace

Qa e2e tests reporting

Skill testomatio/skills/skills/qa-e2e-tests-reporting

Install and configure Testomat.io reporting in a test automation project when a user needs to integrate their tests with Testomat.io. Detect the testing framework (e.g., Playwright, CodeceptJS, Jest, Mocha, WebdriverIO, JUnit, Pytest, XML, etc) and automatically set up the appropriate reporter so test results are sent to the Testomat.io TMS.From its SKILL.md

Install
npx -y skills add testomatio/skills --skill qa-e2e-tests-reporting

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

  • no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
  • 19 stars19 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its file declares

Copied from the file, not written here

The file declares its own license as MIT. That is the author’s claim about this one file, and it is not the same thing as the license GitHub reports for the repository, which is listed with the other numbers below.

SKILL.md

11.1 KB, as published. Nobody here has run it

Testomat.io Reporter Setup

Install and configure the Testomat.io reporter in an automation project, then push test results to the Testomat.io TMS.

Step 1: Detect Language & Framework

  • Detect the language from the project manifest (package.json, pyproject.toml/requirements.txt, pom.xml/build.gradle, *.csproj, composer.json, Gemfile).
  • Detect the framework from its config file: playwright.config.*, codecept.conf.js, wdio.conf.*, jest.config.*, .mocharc.*, pytest.ini/conftest.py, *.robot/robot.toml, testng.xml/JUnit deps in pom.xml.
  • No native reporter for the stack (C#, PHP, Ruby, Go, etc.): use JUnit XML import (Step 4).

Step 2: Configure Credentials

  • Check .env for a TESTOMATIO API key. If present, use it.
  • If missing, add a placeholder line to .env:
TESTOMATIO=tstmt_xxxxx
  • Ask the user to replace the placeholder with their "Project Reporting API key" themselves — never ask them to paste the key into chat.

Message to the user:

✅ Added `TESTOMATIO=tstmt_xxxxx` placeholder to .env

To activate reporting:
1. Go to Testomat.io → your project → Settings → Project
   (https://app.testomat.io/projects/<project-id>/settings/project)
2. Copy "Project Reporting API key"
3. Replace `tstmt_xxxxx` in .env with your actual key
4. Run tests: TESTOMATIO=tstmt_xxxxx npx <test-command>

Step 3: Install & Configure Reporter

Pick the section for the detected language and framework.

JavaScript / TypeScript

Use @testomatio/reporter:

npm install @testomatio/reporter --save-dev

Playwright — playwright.config.ts or playwright.config.js:

reporter: [
  ['@testomatio/reporter/playwright'],
  // other reporters...
],

CodeceptJS — codecept.conf.js or codecept.config.js:

plugins: {
  testomatio: {
    enabled: true,
    require: '@testomatio/reporter/codecept',
  }
}

WebdriverIO — wdio.conf.js:

import testomatio from '@testomatio/reporter/webdriver';

exports.config = {
  // ...
  reporters: [
    [testomatio, { apiKey: process.env.TESTOMATIO }],
  ],
};

Jest — jest.config.js or jest.config.ts:

module.exports = {
  reporters: ['default', ['@testomatio/reporter/jest', { apiKey: process.env.TESTOMATIO }]],
};

Mocha — run with CLI flags:

mocha --reporter @testomatio/reporter/mocha --reporter-options apiKey=tstmt_xxx

Python

Use pytestomatio for pytest, robot-framework-reporter for Robot Framework.

  • Use pip3 if Python 2.x and 3.x coexist.

pytest:

pip install pytestomatio

# Run and report tests
pytest --testomatio report

Robot Framework:

pip install robot-framework-reporter

# Run and report tests
robot --listener Testomatio.Report path/to/tests

Docs: Testomat.io Python Reporting

Java (JUnit, TestNG, Cucumber, Karate)

Use the io.testomat Java reporters.

  • Ask the user to set the API key before running tests, then verify it is set and starts with tstmt_:
export TESTOMATIO=<tstmt_api_key>
  • Add the dependency for the framework in use:
<dependency>
  <groupId>io.testomat</groupId>
  <artifactId>java-reporter-junit</artifactId>
  <version>{LATEST_STABLE_VERSION}</version>
</dependency>

Artifact IDs: java-reporter-junit, java-reporter-testng, java-reporter-cucumber, java-reporter-karate.

  • Run tests:
mvn clean test
  • Scan the output for a run ID. If found, show the link:
View results: https://app.testomat.io/projects/{project-id}/runs/{run-id}

Docs: Testomat.io Java/JUnit Reporting

Step 4: Import Automated Tests to Testomat.io TMS

Import test source code into Testomat.io to see test structure and enable codebase-TMS synchronization.

Docs: Testomat.io Import Overview

JavaScript / TypeScript

Use the check-tests CLI:

npx check-tests@latest <framework> "<glob-pattern>" [options]
FrameworkExample
Playwrightnpx check-tests@latest Playwright "tests/**/*.spec.js"
CodeceptJSnpx check-tests@latest CodeceptJS "tests/**_test.js"
Cypressnpx check-tests@latest cypress "cypress/e2e/**/*.js"
Jestnpx check-tests@latest Jest "tests/**/*.test.js"
Mochanpx check-tests@latest mocha "test/**/*_test.js"
WebdriverIOnpx check-tests@latest webdriverio "test/**/*.js"

Options:

  • --typescript — for TypeScript projects: npx check-tests@latest Playwright "tests/**/*.spec.ts" --typescript
  • --update-ids — sync and auto-assign test IDs in source code: npx check-tests@latest CodeceptJS "tests/**_test.js" --update-ids

Python

pytest — sync tests with Testomat.io:

pytest --testomatio sync

Robot Framework — import tests:

robot --listener Testomatio.Import path/to/tests

Java (JUnit, TestNG)

Use the testomatio.jar CLI. Re-run after every test execution.

  • Run only from the project root:
cd <project-root>

export TESTOMATIO=tstmt_xxxxx && \
curl -L -O https://github.com/testomatio/java-check-tests/releases/latest/download/testomatio.jar && \
java -jar testomatio.jar sync

Commands:

  • import — import test code to Testomat.io (dry-run without API key).
  • clean-ids — import tests without system ids.
  • sync — import tests and pull IDs into source code (alias: update-ids).

JUnit XML (C#, PHP, Ruby, Go, etc.)

For stacks without a native reporter, generate a JUnit XML report, then import it.

  • NUnit (C#): dotnet test --logger:"trx;LogFileName=results.xml" — convert to JUnit XML if needed.
  • PHPUnit (PHP): ./vendor/bin/phpunit --log-junit=results.xml
  • RSpec (Ruby): rspec --format json --out results.json — then convert to JUnit XML.

Import to Testomat.io:

npx check-tests@latest junit "path/to/results.xml"

# Multiple XML files
npx check-tests@latest junit "results/*.xml"

Step 5: Verify Setup

Run the tests with the reporter enabled:

TESTOMATIO=tstmt_xxxxx npx <your-test-command>

Via Testomat.io MCP (preferred, if enabled)

  • After the run, fetch the latest run via MCP to confirm data was received.
  • Success indicators: a new run ID appears in the logs; run status shows passed/failed as expected; the run is retrievable by its ID.
  • Setup is complete when the new run is fetchable via MCP.
  • MCP tools: Testomat.io MCP Run Management

Debug Mode (alternative)

  • Capture reporter output locally for only 1-2 tests to verify data is generated correctly before sending:
DEBUG=@testomatio/reporter:pipe:testomatio npx <your-test-command>

Step 6: Configure Artifacts (only on explicit request)

Store screenshots, videos, traces, and logs in S3-compatible storage alongside test results.

  • Only perform this step if the user explicitly asks for artifacts configuration (or mentions saving screenshots, videos, traces, logs).
  • Prerequisite: first test report executed successfully.
  • Check .env for existing S3 credentials (S3_ACCESS_KEY_ID, S3_BUCKET, S3_REGION). If present, skip to verification.

Create S3 Bucket (if needed)

  • Recommended: one bucket per project.
  • Guide the user through their provider, based on official documentation:
ProviderLink
AWS S3https://s3.console.aws.amazon.com/s3
DigitalOceanhttps://cloud.digitalocean.com/spaces
Google Cloud Storagehttps://console.cloud.google.com/storage
Cloudflare R2https://dash.cloudflare.com/
MinioSelf-hosted or cloud instance
  • The user obtains the Access Key ID and Secret Access Key from the provider.

Option A: Testomat.io UI (recommended)

  • The agent cannot access Testomat.io project settings — the user configures this manually:
    1. Open https://app.testomat.io/projects/<project-id>/settings/artifacts
    2. Enable the "Share credentials" toggle.
    3. Enter Access Key ID, Secret Access Key, bucket name, region, and endpoint (endpoint only for non-AWS providers).

Option B: Environment Variables

Add S3 credentials to .env or the CI pipeline:

# Required for all providers
S3_ACCESS_KEY_ID=your_access_key
S3_SECRET_ACCESS_KEY=your_secret_key
S3_BUCKET=your_bucket_name
S3_REGION=us-west-1

# Optional: for non-AWS providers (DigitalOcean, Minio, Cloudflare R2, GCS)
S3_ENDPOINT=https://your-endpoint-url

# Optional: private access mode (recommended)
TESTOMATIO_PRIVATE_ARTIFACTS=1

Cloudflare R2 example:

S3_ACCESS_KEY_ID=xxx
S3_SECRET_ACCESS_KEY=xxx
S3_BUCKET=testomatio-artifacts
S3_REGION=auto
S3_ENDPOINT=https://<account-id>.r2.cloudflarestorage.com
S3_FORCE_PATH_STYLE=true

Verify Artifacts

  • Run tests that produce screenshots, videos, or traces: TESTOMATIO=tstmt_xxxxx npx <your-test-command>
  • Ask the user to open https://app.testomat.io/projects/<project-id>/runs and click a test with artifacts (failed tests typically have screenshots).
  • Success: artifacts appear in the test results ("Artifacts" section or attachment icon) and are clickable; Playwright traces open in the trace viewer.

References

DescriptionLink
Getting startedhttps://docs.testomat.io/getting-started/
Reporter configurationhttps://docs.testomat.io/test-reporting/frameworks/
Framework config snippets./references/TESTOMATIO_REPORTERS_CONFIG.md
Python test reportinghttps://docs.testomat.io/test-reporting/python/
Python Robot Frameworkhttps://github.com/testomatio/robot-framework-reporter/blob/master/README.md
Java/JUnit reportinghttps://docs.testomat.io/test-reporting/junit/
HTML pipe./references/TESTOMATIO_HTML_REPORT.md
Debug pipeTESTOMATIO_DEBUG=1 <test-command>
Artifacts (S3)./references/TESTOMATIO_ARTIFACTS.md

Keep looking

Skills are one crate of 610,677. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.