Rudder data catalog

RudderStack Data Catalog Management

This skill teaches how to create and manage the building blocks of instrumentation: events, properties, categories, and custom types.

Recommended Workflow

When adding or editing catalog resources, author bottom-up (dependencies first) then validate and apply. The referencing order is strict — an event can't reference a property URN until that property exists.

digraph data_catalog_workflow {
    rankdir=TB;
    "1. Custom types (reusable shapes)" [shape=box];
    "2. Properties (vocabulary)" [shape=box];
    "3. Categories (grouping)" [shape=box];
    "4. Events (reference all of the above)" [shape=box];
    "rudder-cli validate -l ./" [shape=box];
    "Errors?" [shape=diamond];
    "Fix references / URNs / type config" [shape=box];
    "rudder-cli apply --dry-run -l ./" [shape=box];
    "Diff matches intent?" [shape=diamond];
    "rudder-cli apply -l ./" [shape=box];
    "Done" [shape=doublecircle];

    "1. Custom types (reusable shapes)" -> "2. Properties (vocabulary)";
    "2. Properties (vocabulary)" -> "3. Categories (grouping)";
    "3. Categories (grouping)" -> "4. Events (reference all of the above)";
    "4. Events (reference all of the above)" -> "rudder-cli validate -l ./";
    "rudder-cli validate -l ./" -> "Errors?";
    "Errors?" -> "Fix references / URNs / type config" [label="yes"];
    "Fix references / URNs / type config" -> "rudder-cli validate -l ./";
    "Errors?" -> "rudder-cli apply --dry-run -l ./" [label="no"];
    "rudder-cli apply --dry-run -l ./" -> "Diff matches intent?";
    "Diff matches intent?" -> "Fix references / URNs / type config" [label="no"];
    "Diff matches intent?" -> "rudder-cli apply -l ./" [label="yes"];
    "rudder-cli apply -l ./" -> "Done";
}

Why bottom-up: properties reference custom types; events reference properties, categories, and custom types. Creating in the reverse order means every intermediate validate fails on missing references. For the validate → dry-run → apply details (error formats, diff reading, auth prereqs), see the rudder-cli-workflow skill.

Core Concepts

Concept	Purpose	Example
Events	What happened	"Product Viewed", "Order Completed"
Properties	Attributes of events	product_id, price, quantity
Categories	Organize events	"Ecommerce", "User Lifecycle"
Custom Types	Reusable validation patterns	ProductType, AddressType, Currency

Before Creating: Check Existing Catalog

Before creating new events or properties, check what already exists to prevent duplicates and ensure consistency.

Why Check First?

• Prevents duplicate events with different names ("Product Viewed" vs "ProductView")
• Ensures warehouse consistency — same data, same column names
• Reuses existing custom types — don't reinvent AddressType
• Maintains naming conventions — follow established patterns

How to Check

Using Rudder CLI:

# List existing events
rudder-cli get events

# List existing properties
rudder-cli get properties

# List custom types
rudder-cli get custom-types

Using MCP:

Tool: list_data_catalog_events
Search for events matching your proposed name

Tool: list_data_catalog_properties
Check if property already exists

Naming Convention Validation

Before proposing new resources, verify they follow conventions:

Resource	Convention	Example	Anti-Example
Events	Title Case with spaces	Product Viewed	productViewed, product_viewed
Properties	snake_case	product_id	productId, ProductId
Categories	kebab-case	user-lifecycle	userLifecycle, user_lifecycle
Custom Types	PascalCase	ProductType	product_type, productType

Check for Similar Events

If proposing "Transformation Created", search for:

• Existing "Transformation Created"
• Similar: "Transformation Added", "Create Transformation"
• Related: other transformation events

rudder-cli get events | grep -i transform

Directory Structure

data-catalog/
├── events/
│   ├── ecommerce.yaml        # Product Viewed, Order Completed, etc.
│   └── user-lifecycle.yaml   # Signed Up, Logged In, etc.
├── properties/
│   ├── product-properties.yaml
│   ├── customer-properties.yaml
│   └── address-properties.yaml
├── categories/
│   └── categories.yaml
└── custom-types/
    ├── product-type.yaml
    └── address-type.yaml

YAML Schemas

Event Definition

version: "rudder/v1"
kind: "event"
metadata:
  name: "events"
spec:
  name: "Product Viewed"
  description: "User viewed a product detail page"
  category: "urn:rudder:category/ecommerce"
  rules:
    - property: "urn:rudder:property/product_id"
      required: true
    - property: "urn:rudder:property/product_name"
      required: true
    - property: "urn:rudder:property/product_price"
      required: true
    - property: "urn:rudder:property/product_category"
    - property: "urn:rudder:property/page_url"

Property Definition

version: "rudder/v1"
kind: "property"
metadata:
  name: "properties"
spec:
  name: "product_id"
  type: "string"
  description: "Unique product identifier"
  config:
    minLength: 3
    maxLength: 50

Category Definition

version: "rudder/v1"
kind: "category"
metadata:
  name: "categories"
spec:
  name: "ecommerce"
  description: "Events related to product discovery, cart, and purchase"

Custom Type Definition

version: "rudder/v1"
kind: "custom-type"
metadata:
  name: "custom-types"
spec:
  name: "ProductType"
  type: "object"
  description: "Consolidated product information"
  config:
    properties:
      - property: "urn:rudder:property/product_id"
        required: true
      - property: "urn:rudder:property/product_sku"
        required: true
      - property: "urn:rudder:property/product_name"
        required: true
      - property: "urn:rudder:property/product_category"
        required: true
      - property: "urn:rudder:property/product_price"
        required: true
      - property: "urn:rudder:property/product_msrp"
        required: false

URN Reference System

Resources reference each other using URNs (Uniform Resource Names):

Resource Type	URN Pattern	Example
Event	urn:rudder:event/<name>	urn:rudder:event/product-viewed
Property	urn:rudder:property/<name>	urn:rudder:property/product_id
Category	urn:rudder:category/<name>	urn:rudder:category/ecommerce
Custom Type	urn:rudder:custom-type/<name>	urn:rudder:custom-type/product-type

Important: URN names are kebab-case versions of the resource name.

Property Type Configuration

String Type

spec:
  name: "customer_email"
  type: "string"
  config:
    minLength: 5
    maxLength: 255
    pattern: "^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$"

Config Option	Description
minLength	Minimum string length
maxLength	Maximum string length
pattern	Regex pattern for validation
format	Built-in format (date-time, email, uri)
enum	Array of allowed values

Number Type

spec:
  name: "product_price"
  type: "number"
  description: "Product price in USD"
  config:
    minimum: 0
    exclusiveMinimum: true

Config Option	Description
minimum	Minimum value (inclusive)
maximum	Maximum value (inclusive)
exclusiveMinimum	Minimum is exclusive
exclusiveMaximum	Maximum is exclusive

Integer Type

spec:
  name: "quantity"
  type: "integer"
  description: "Product quantity in cart"
  config:
    minimum: 1
    maximum: 100

Array Type

spec:
  name: "products"
  type: "array"
  description: "List of products in order"
  config:
    items:
      customType: "urn:rudder:custom-type/product-type"
    minItems: 1

Config Option	Description
items.type	Type of array items (string, number, etc.)
items.customType	Custom type for array items
minItems	Minimum array length
maxItems	Maximum array length

Enum (Fixed Values)

spec:
  name: "product_category"
  type: "string"
  description: "Product category"
  config:
    enum:
      - "Footwear"
      - "Clothing"
      - "Accessories"

Real-World Example

See references/ecommerce-example.md for a complete e-commerce data catalog with custom types (ProductType, AddressType), properties, and events showing how these components work together.

Why Custom Types Matter

Custom types let you define reusable validation patterns:

• ProductType → used by Product Viewed, Product Added to Cart
• AddressType → used by shipping_address AND billing_address

Benefits: single source of truth, change in one place, cleaner event definitions.

Creating Properties from Code Types

When your codebase already has domain types, derive properties from them to ensure alignment.

Enum to Property

// Code
enum BillingPlan {
  FREE = 'free',
  STARTER = 'starter',
  GROWTH = 'growth',
  ENTERPRISE = 'enterprise',
}

# Property - values must match exactly
spec:
  name: "billing_plan"
  type: "string"
  config:
    enum:
      - "free"        # Matches BillingPlan.FREE
      - "starter"
      - "growth"
      - "enterprise"

String Union to Property

// Code
type TransformationLanguage = 'javascript' | 'python';

# Property
spec:
  name: "transformation_language"
  type: "string"
  config:
    enum:
      - "javascript"
      - "python"

Critical: Use Exact Values

The tracking plan must use the exact string values from code:

// If code uses lowercase
enum Region {
  US = 'us',    // lowercase
  EU = 'eu',
}

# YAML must match exactly
config:
  enum:
    - "us"      # NOT "US" or "Us"
    - "eu"      # NOT "EU" or "Eu"

For the full code-first workflow, see rudder-code-first-instrumentation skill.

Validation Commands

# Validate all resources
rudder-cli validate -l ./

# Validate specific directory
rudder-cli validate -l ./data-catalog/events/

# Preview changes before applying
rudder-cli apply --dry-run -l ./

# Apply to workspace
rudder-cli apply -l ./

Common Patterns

Pattern: Monetary Values

Use number type with separate currency property:

# Price property
spec:
  name: "order_total"
  type: "number"
  config:
    minimum: 0

# Currency property
spec:
  name: "currency"
  type: "string"
  config:
    pattern: "^[A-Z]{3}$"  # ISO 4217
    enum: ["USD", "EUR", "GBP"]

Pattern: Timestamps

Use string with date-time format:

spec:
  name: "created_at"
  type: "string"
  config:
    format: "date-time"  # ISO 8601

Pattern: Optional with Default Context

Include context properties for attribution:

# Always include for funnel analysis
- property: "urn:rudder:property/page_url"
- property: "urn:rudder:property/referrer_url"
- property: "urn:rudder:property/session_id"

Common Mistakes

Mistake	Problem	Fix
Missing property definition	URN reference fails	Create property YAML first
Wrong URN format	Reference not found	Use kebab-case: product-id not product_id
Type mismatch	Validation fails	Match property type to expected data
Circular custom type	Infinite loop	Custom types cannot reference themselves
Wrong config for type	Config ignored	Use minLength for strings, minimum for numbers