Kafka clusters

kafka-clusters

When to use this skill

• User asks to create, list, describe, or delete Kafka Instance or KafkaCluster resources.
• User mentions kafkacluster, Kafka RTU / throughput unit, clusterProfile, Kafka gateways, Kafka mTLS, Kafka RBAC, or Kafka cluster Schema Registry endpoint.
• User wants snctl context use / service-context setup for KafkaCluster data-plane commands.
• User wants dedicated, dedicated-pro, BYOC, or BYOC Pro Kafka deployments. KafkaCluster does not support serverless today.

Execution base

Prefer StreamNative Cloud remote MCP when available: use sncloud_clusters_read / sncloud_clusters_write for Instance, KafkaCluster, and controller-managed cluster resources visible in catalog; use sncloud_resource_catalog / sncloud_resource_schema before writes. Remote MCP is public preview and disabled by default; newly created clusters have cluster-level MCP disabled until user enables it in StreamNative Cloud UI, so data-plane follow-up through sncloud_context_use_cluster may not work immediately after create; when it does succeed, output MCP_TOOLS_REFRESH_REQUIRED instance=<instanceName> cluster=<clusterName> and stop so the runner/runtime can refresh tools before data-plane calls. See ../cloud-core/references/remote-mcp.md. Use snctl fallback when MCP tools are unavailable or for CLI-specific examples below.

Prerequisites check

Remote MCP route: start with sncloud_context_whoami, then catalog/schema for clusters domain. snctl route: run standard checks (snctl version --client, snctl auth whoami, snctl config get). Require snctl >= v1.7.0 for KafkaCluster-related workflows; upgrade before create/list/describe/delete/context actions. Defer auth/install failures to cloud-core.

KafkaCluster-specific control-plane prerequisites:

• Organization must have Instance enabled (cloud.streamnative.io/instance-enabled=true).
• Organization must have AuthV2 enabled (CloudFeature AuthV2=true).
• snctl create kafkacluster fails early if either feature is missing.
• For BYOC, cloud connection + cloud environment + pool/member must already exist; switch to byoc-infrastructure if missing.

Mental model

Two layered kinds:

• Instance — top-level container; picks deployment type and infrastructure pool. New shared kind used by Kafka and new AuthV2 service contexts. Do not use PulsarInstance for KafkaCluster.
• KafkaCluster — Kafka runtime bound to an Instance. Controller provisions Kafka brokers plus optional Kafka REST Proxy and Schema Registry endpoint.

KafkaCluster important fields:

• spec.instanceName — required parent Instance.
• spec.location — required location/region.
• spec.poolMemberRef — required by admission on create; snctl create may resolve and populate it from --location, but raw manifests must set both namespace and name.
• spec.throughputUnit — requested Kafka broker capacity in RTU; defaults to 1.
• spec.clusterProfile — latency-optimized (default in cloud-manager UI) or cost-optimized.
• spec.endpointAccess[].gateway — gateway names used to expose Kafka endpoints.
• spec.config.custom — Kafka broker config key/value overrides.
• spec.mtls.enabled — enable client-to-broker mTLS.
• spec.rbac.enabled — admission defaults this to true on create when omitted; set false via --disable-rbac / manifest when Kafka RBAC should be off.
• metadata.annotations.cloud.streamnative.io/enable-schema-registry: "true" — requests controller-managed SnSchemaRegistry + backend config.
• metadata.annotations.cloud.streamnative.io/schemaregistry — controller writes associated Schema Registry name, normally same as KafkaCluster.
• metadata.annotations.cloud.streamnative.io/auth: "v2" — controller-managed resources use AuthV2; service context expects AuthV2 token flow.

Deployment-type decision tree

Ask deployment type before create action. If unclear:

1. StreamNative managed cloud or customer cloud account?
2. StreamNative managed cloud: dedicated or dedicated-pro? KafkaCluster does not support serverless today.
3. Customer cloud: BYOC or BYOC Pro, and is cloud connection/environment/pool Ready?

Supported Instance.spec.type for KafkaCluster: dedicated | dedicated-pro | byoc | byoc-pro.

Primary workflows

For remote MCP writes, compose JSON-string manifests and call sncloud_clusters_write with operation=apply, dry_run=true; only repeat with dry_run=false after validation. Use sncloud_clusters_read for list/get. Never use generic sncloud_resources_*.

W1. Create dedicated Instance + KafkaCluster

snctl create instance my-instance -n my-org \
  --pool streamnative/shared-aws \
  --type dedicated

snctl create kafkacluster my-kafka -n my-org \
  --instance-name my-instance \
  --location us-east-1 \
  --throughput-unit 2 \
  --cluster-profile latency-optimized
# snctl resolves --location to exact PoolMember and sends spec.poolMemberRef.

# Enable Schema Registry when using manifests. `snctl create kafkacluster` does not expose
# a schema-registry flag in every version; prefer manifest when endpoint is required.

Manifest path:

snctl apply -f ${CLAUDE_PLUGIN_ROOT}/skills/kafka-clusters/assets/manifests/instance-dedicated.yaml
snctl apply -f ${CLAUDE_PLUGIN_ROOT}/skills/kafka-clusters/assets/manifests/kafkacluster-dedicated.yaml

W2. Create BYOC / BYOC Pro Instance + KafkaCluster

Prerequisite: run BYOC substrate workflows from byoc-infrastructure first.

snctl get cloudconnection -n my-org -o name
snctl get cloudenvironment -n my-org -o name
snctl get poolmember -n my-org

snctl create instance my-instance -n my-org \
  --pool my-org/my-pool \
  --type byoc

snctl create kafkacluster my-kafka -n my-org \
  --instance-name my-instance \
  --pool-member-name my-org/my-pool-member \
  --throughput-unit 2 \
  --cluster-profile latency-optimized

Use --type byoc-pro for BYOC Pro pools.

W3. Create with Schema Registry / REST Proxy endpoints

cloud-manager creates KafkaCluster with cloud.streamnative.io/enable-schema-registry: "true" so controller provisions same-named SnSchemaRegistry and records cloud.streamnative.io/schemaregistry on KafkaCluster.

snctl apply -f ${CLAUDE_PLUGIN_ROOT}/skills/kafka-clusters/assets/manifests/kafkacluster-dedicated.yaml
snctl get snschemaregistry -n my-org
snctl get snschemaregistry my-kafka-dedicated -n my-org -o yaml

Endpoint sources:

• Kafka bootstrap: KafkaCluster.status.serviceEndpoints, prefer type: external.
• Kafka REST Proxy: KafkaCluster.status.kafkaRestProxy.serviceEndpoint.
• Schema Registry: associated SnSchemaRegistry.status.serviceEndpoints, prefer type: external.

Direct KafkaCluster does not imply Kafka Connect endpoint; Kafka Connect remains PulsarCluster/KSN path unless backend adds separate support.

W4. List / describe / delete

snctl get instance -n my-org
snctl get kafkacluster -n my-org
snctl get kafkacluster my-kafka -n my-org -o yaml
snctl describe kafkacluster my-kafka -n my-org

# Delete child before parent. Use --wait=false for async cloud teardown.
snctl delete kafkacluster my-kafka -n my-org --wait=false
snctl delete instance my-instance -n my-org --wait=false
snctl get kafkacluster my-kafka -n my-org --ignore-not-found

W5. Update scale/profile/config

Preferred: edit manifest and snctl apply -f from source control.

Interactive edit:

snctl edit kafkacluster my-kafka

For config, prefer manifest because manual nested-map edits are easy to get wrong.

W6. Inspect readiness and endpoints

snctl get kafkacluster my-kafka -n my-org -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'
snctl describe kafkacluster my-kafka -n my-org
snctl get kafkacluster my-kafka -n my-org -o jsonpath='{.status.serviceEndpoints}'
snctl get kafkacluster my-kafka -n my-org -o jsonpath='{.status.kafkaRestProxy.serviceEndpoint}'

Status endpoints:

• status.serviceEndpoints[type=external].dnsName — preferred Kafka bootstrap endpoint.
• status.serviceEndpoints[type=internal].dnsName — fallback/internal bootstrap endpoint.
• status.kafkaRestProxy.serviceEndpoint — Kafka REST Proxy URL host.
• Associated SnSchemaRegistry is usually named same as KafkaCluster. Trust annotation cloud.streamnative.io/schemaregistry first; fallback to same-name SnSchemaRegistry.

W7. Use KafkaCluster as service context

Remote MCP route: use fixed Kafka cluster endpoint, or organization-level sncloud_context_available_clusters → sncloud_context_use_cluster; after a successful org-level cluster switch, output MCP_TOOLS_REFRESH_REQUIRED instance=<instanceName> cluster=<clusterName> and stop so the runner/runtime refreshes MCP tools; after resume, use feature-gated Kafka tools such as kafka_admin_topics_read/write, kafka_admin_groups_read/write, kafka_admin_sr_read/write, kafka_admin_connect_read/write, kafka_client_produce, and kafka_client_consume. Both require cluster-level MCP enabled for the target cluster in StreamNative Cloud UI; newly created clusters default to disabled.

snctl route: KafkaCluster context uses Instance/AuthV2 flow. It exchanges current user or selected service account grant, then writes Kafka client config.

snctl context use -n my-org --instance my-instance --kafka-cluster my-kafka
snctl kafka admin topics list
snctl kafka admin groups list

Important auth behavior:

• Broker SASL user is extracted from AuthV2 access token username.
• Broker password is raw access token (not token:<jwt>) for KafkaCluster contexts.
• Schema Registry and REST Proxy use same AuthV2 token.
• snctl kafka connect ... can be unavailable for direct KafkaCluster context when no Kafka Connect URL exists; use PulsarCluster/KSN context for Connect.
• If prompted, select service account with required RBAC. For automation, activate SA key first or pass service-context SA options supported by your snctl version.

Common flags and output conventions

Standard rules from cloud-core apply. KafkaCluster-specific:

• metadata.name / CLI cluster name for KafkaCluster must be 20 characters or fewer. Longer names fail server-side with name is longer than 20 characters.
• --instance-name required.
• --location or --pool-member-name required for snctl create; direct manifests must include spec.poolMemberRef.name and spec.poolMemberRef.namespace.
• --throughput-unit must be >= 1; default 1.
• --cluster-profile cost-optimized|latency-optimized.
• --gateways <name> repeatable.
• --config key=value repeatable; becomes spec.config.custom.
• --catalog <name> associates one catalog for compaction/lakehouse workflows.
• --mtls-enabled sets spec.mtls.enabled=true.
• --disable-rbac sets spec.rbac.enabled=false; leaving it unset enables RBAC because admission defaults it to true on create.
• Schema Registry enablement is annotation-driven (cloud.streamnative.io/enable-schema-registry: "true"); use manifest if current snctl create kafkacluster --help lacks a flag.

Reference index

• references/deployment-types.md — Instance deployment types and pool mapping.
• references/manifest-schemas.md — Instance/KafkaCluster/SnSchemaRegistry schemas and snippets.
• references/regions-clouds.md — region/location examples.
• references/authv2-service-context.md — KafkaCluster context auth and endpoint mapping.

Pitfalls and gotchas

• Remote MCP sncloud_clusters_write manifest is JSON string, not YAML/object. Omit status and read-only metadata copied from read output.
• Creating a cluster does not enable cluster-level MCP. Ask user to enable cluster-level MCP in StreamNative Cloud UI before using sncloud_context_use_cluster or Kafka data-plane MCP tools on that cluster. After sncloud_context_use_cluster succeeds, emit MCP_TOOLS_REFRESH_REQUIRED instance=<instanceName> cluster=<clusterName> and wait for runner/runtime refresh because selected cluster context changes the session-scoped tool surface. ToolSearch is not refresh.
• KafkaCluster names must be <=20 characters; use short smoke-test names.
• KafkaCluster uses Instance, not PulsarInstance.
• KafkaCluster does not support serverless Instance today; use dedicated, dedicated-pro, BYOC, or BYOC Pro instead.
• KafkaCluster create requires both Instance and AuthV2 org features. Missing feature errors are expected guardrails, not CLI bugs.
• --pool-member-name must belong to the Instance pool; snctl rejects mismatched pool member vs Instance.spec.poolRef.
• --location must resolve to exactly one matching pool member. Ambiguous location requires explicit --pool-member-name.
• throughputUnit is RTU capacity, not broker replica count. Controller translates RTU to resources unless backend config overrides.
• clusterProfile only accepts cost-optimized or latency-optimized.
• KafkaCluster service context auth differs from legacy Pulsar+KoP context: broker password is raw AuthV2 access token, and username comes from token claims.
• Associated Schema Registry may not exist immediately; poll KafkaCluster Ready and list snschemaregistry if endpoint missing.
• Do not read Schema Registry from KafkaCluster.status.serviceEndpoints; cloud-manager may expose confusing generated fields, but authoritative source is SnSchemaRegistry.status.serviceEndpoints.
• Direct KafkaCluster contexts do not expose Kafka Connect; absence of KafkaConnectUrl is expected.
• snctl delete defaults to --wait=true and can block during cloud teardown. Use --wait=false, then poll.