Validation

Overview

This skill is MANDATORY after creating or modifying any Honeydew object.

Validation ensures:

• The object compiles and executes without errors
• Results are returned (not NULL or empty)
• Values make business sense
• Related objects are consistent with each other

---

Validation by Object Type

Metrics

Step 1: Execute the metric

Call get_data_from_fields with:

• metrics: ["<entity>.<metric_name>"]

If the tool call fails (API error, permission denied, timeout), report the error to the user before proceeding. Do not confuse a tool error with suspicious data.

Step 2: Sanity checks

Check	What to Look For	Action if Failed
Returns data	Not NULL, not empty	Check SQL syntax, entity references
Magnitude	Reasonable for business context	Verify calculation logic
Sign	Positive for revenue/counts (usually)	Check for inverted logic
Related metrics	Parts sum to whole	Query related metrics together and verify consistency
Ratios	Between 0-100% (usually)	Check numerator and denominator metrics independently

Step 3: Cross-validation

If the new metric is a filtered subset of an existing metric, or a ratio of existing metrics, query them together to verify consistency:

• For filtered metrics: query the filtered metric alongside the unfiltered total — the filtered value should be less than or equal to the total.
• For ratio/derived metrics: query the numerator and denominator independently to confirm they return sensible values before checking the ratio.

Call get_data_from_fields with both metrics:

• metrics: ["<entity>.<filtered_metric>", "<entity>.<total_metric>"]

Alert user if:

• Metric returns $0 or NULL unexpectedly
• Revenue/count is negative
• Ratio exceeds 100% or is negative (unless expected)
• Magnitude seems off by orders of magnitude

---

Attributes

Step 1: Execute the attribute

If the attribute references a related entity (multi-entity attribute), first verify the relation exists using get_entity on the source entity. If the relation is missing, report that before attempting to query the attribute.

Sample rows — call get_data_from_fields with:

• attributes: ["<entity>.<attribute_name>"]

For boolean attributes, check distribution — call get_data_from_fields with:

• attributes: ["<entity>.<boolean_attribute>"]
• metrics: ["<entity>.count"]

Step 2: Sanity checks by attribute type

Attribute Type	Valid Range	Red Flags
Age/Duration	0 to reasonable max	Negative values, >150 years
Boolean	TRUE/FALSE mix	All TRUE or all FALSE
Percentage	0-100 (usually)	Negative, >100 (unless growth rate)
Date	Past to near future	Year 1900, year 2099, all NULLs
Category/Bucket	Expected labels	95%+ in "Unknown" or NULL
Rank	Starts at 1, consecutive	Starts at 0, gaps, all same value
Running total	Monotonically increasing	Decreases, resets unexpectedly
Any type	Mix of values	>50% NULLs warrants investigation

Alert user if:

• All values are NULL or >50% NULLs
• Boolean is 100% one value
• Dates are invalid or in wrong century
• Buckets are mostly "Unknown"

---

Entities

Step 1: Verify entity was created

Use list_entities and filter results for the new entity name.

Step 2: Verify data flows

Call get_data_from_fields with:

• metrics: ["<entity>.count"]

Also call with a list of attributes to verify they are accessible:

• attributes: ["<entity>.<attribute1>", "<entity>.<attribute2>", "<entity>.<attribute3>"]

Step 3: Sanity checks

Check	What to Look For	Action if Failed
Entity exists	Shows in list_entities results	Check create_entity call
Has rows	Count > 0	Verify source table path
Key is unique	Count = count distinct of key	Fix key or add composite key
Attributes accessible	Can query attributes	Check dataset definition

Alert user if:

• Entity not found after creation
• Zero rows returned
• Key column has duplicates (for non-fact tables)

Validating Relations (part of entity validation)

Relations are not standalone objects — they are defined within an entity's YAML. Validate them as part of the entity that contains them.

Step 1: Verify relation exists

Use get_entity on the source entity and check its relations list for the new relation.

Step 2: Test the join works

Call get_data_from_fields with a cross-entity query:

• attributes: ["<target_entity>.<attribute>"]
• metrics: ["<source_entity>.<metric>"]

Step 3: Sanity checks

Check	What to Look For	Action if Failed
Relation exists	Shows in get_entity results	Check update_object call
Join works	Cross-entity query succeeds	Verify join keys match
No fan-out	Counts don't explode	Check cardinality (many-to-one vs many-to-many)
NULLs reasonable	Some NULLs OK for left join	Too many NULLs = bad join key

Alert user if:

• Relation not found on the entity
• Cross-entity query fails
• Row count explodes (indicates wrong cardinality)
• All joined values are NULL (bad join condition)

---

Domains

Step 1: Verify domain exists

Use search_model (with search_mode: EXACT) to find the new domain by name.

Step 2: Test with a scoped query

Call get_data_from_fields with the domain parameter to verify entities are accessible and filters apply:

• metrics: ["<entity>.count"]
• domain: "<domain_name>"

Step 3: Verify filters apply

If the domain has semantic or source filters, compare results with and without the domain to confirm filters reduce the data as expected:

• Query a metric with the domain set — note the result.
• Query the same metric without the domain — note the result.
• The domain-scoped result should be less than or equal to the unscoped result (for filters that restrict rows).

Step 4: Sanity checks

Check	What to Look For	Action if Failed
Domain exists	Found via search_model	Check create_object call and YAML syntax
Entities accessible	Scoped query returns data	Verify entity names match existing entities
Filters apply	Scoped count <= unscoped count	Check filter SQL and entity.field references
Field selectors work	Excluded fields not returned in query	Verify selector patterns and order
No errors	Query executes without compilation errors	Check filter SQL syntax, fully qualified refs

Alert user if:

• Domain not found after creation
• Scoped query returns an error (likely bad filter SQL or missing entity)
• Filters have no effect (scoped count equals unscoped count when a filter is expected to reduce rows)
• Excluded fields are still accessible (field selector not applied correctly)

---

Validating Updates (update_object)

When modifying an existing object, compare before and after:

1. Before altering, query the object and note the current result.
2. After altering, query again and compare.
3. Report the difference to the user: "The metric revenue changed from $1,234,567 to $1,198,432 after applying the discount logic."

This confirms the change had the intended effect and helps catch unintended regressions.

---

Error Handling

If a tool call fails, distinguish between:

Situation	Meaning	Action
Tool returns an error	API error, permission denied, timeout, invalid YAML	Report the error message to the user. Do not treat as a validation failure — the object may not have been created.
Tool returns empty/NULL data	Object was created but returns no results	Proceed with sanity checks — likely a data or SQL issue.
Tool returns unexpected values	Object works but results look wrong	Report findings and ask user before attempting a fix.

To investigate a query that failed earlier — or to see how the same object behaves across other clients (BI tools, scheduled jobs) — use list_query_history to pull the failed execution's error, semantic YAML, and compiled SQL. See the query-debugging skill.

---

Alerting Guidelines

When results seem wrong, report to the user with:

1. The actual value returned by the query
2. Why it looks suspicious (e.g., negative revenue, 100% NULL, row count doubled)
3. A suggested next step (e.g., "Should I check the SQL expression?" or "Should I verify the join keys?")

Do not silently fix issues — always surface findings and ask before making changes.

---

Quick Reference

Object	Execute With	Key Checks
Metric	get_data_from_fields with metrics list	Value, magnitude, sign, consistency
Attribute	get_data_from_fields with attributes list	Range, distribution, NULLs
Entity	list_entities + get_data_from_fields	Exists, has rows, key unique
Relation	get_entity + cross-entity field query	Exists, joins work, no fan-out (validated as part of entity)
Domain	search_model + get_data_from_fields with domain	Exists, filters apply, fields scoped