Confluent kafka developer

Confluent / Kafka Developer

For cluster provisioning, RBAC/networking, or ops — use confluent-kafka-admin instead. This skill is for designing, explaining, diagramming, planning, and reviewing Kafka application work.

Research workflow

Kafka/Confluent APIs, client configs, and best practices change across versions (KRaft, tiered storage, exactly-once semantics, new client configs) — don't answer from pretrained memory alone for anything version-specific.

1. Official docs first: start from docs.confluent.io/llms.txt — Confluent's LLM-oriented doc index (~150 links organized by product/topic: Cloud, Platform, Flink, connectors, clients, security, CLI, etc.) — to find the right page fast, then fetch that specific page for exact details. Use llms-full.txt for grounding on terminology/glossary questions, not as a substitute for the actual page. Also check kafka.apache.org/documentation for OSS-specific behavior, developer.confluent.io for patterns/courses/event-driven design guides, Apache Flink docs for engine-level Flink behavior (see the Flink section for which docs answer what), and the relevant client library docs (Java, Python, Go, .NET, librdkafka).
2. context7 for library and API surfaces — exact client config property names, method signatures, Flink/Streams DSL operators. It is faster than fetching doc pages and never 403s. resolve-library-id first, then query-docs; one concept per query, three calls max. Known-good IDs: /confluentinc/librdkafka (and the C/Python/Go/.NET clients built on it), /apache/kafka (broker/topic configs), /apache/flink, and the Java client javadoc, which is indexed per release as /websites/javadoc_io_doc_org_apache_kafka_kafka-clients_<version>. Pin the version deliberately. resolve-library-id ranks by documentation coverage, not recency — asking it for Apache Flink returns the 1.19 docs site above the 2.x ones. Read the Versions: list and the version suffix in the ID, match it to what the project actually runs (pom.xml/build.gradle, requirements.txt, the Cloud Flink version), and pass /org/project/version when it matters. A 1.19 answer applied to a 2.x job is a wrong answer that looks sourced.
3. Community sources when docs don't settle it: a specific error message, an edge case, or "does X actually behave like Y in practice" — search the Confluent Community Forum, Stack Overflow (apache-kafka, confluent-platform tags), and relevant GitHub issues/discussions (apache/kafka, confluentinc/*).
4. Cite what you find and flag conflicts: if community info contradicts the docs or looks version-specific/stale, say so rather than silently picking one. Name the version a context7 answer came from when the behaviour is version-specific.

Task modes

Explain/understand: ground the explanation in retrieved docs; call out version-specific behavior explicitly (e.g., pre- vs. post-KRaft, exactly-once semantics changes across broker/client versions).

Design: cover topic naming and partitioning strategy, key design, schema format (Avro/Protobuf/JSON Schema) and Schema Registry compatibility mode, delivery semantics (at-least-once vs. exactly-once), consumer group strategy, event-vs-command modeling, ordering guarantees, and retention/compaction choice. Weigh 2-3 approaches with trade-offs and commit to a recommendation with a one-line rationale, then confirm scope with the user before finalizing — don't just present a menu.

Diagram: default to Mermaid (renders natively in Claude Code/artifacts). Use flowcharts for producer→topic→consumer data flow, sequence diagrams for request/response or Streams processing order, and component diagrams for multi-service/cluster architecture. Show partition/consumer-group relationships explicitly when they're the point of the diagram.

Plan tasks: break Kafka feature work into concrete implementation tasks (e.g., "define Avro schema + compatibility mode", "configure idempotent producer", "write Streams topology + tests", "add DLQ handling for poison messages"). For a large multi-task build the user wants driven end to end, hand off to the autopilot skill (Claude Code only — elsewhere, just present the task list).

Implement: write the code or config, don't stop at a plan. Retrieval still applies — verify config keys, SQL syntax, and API surfaces against docs before writing, because a plausible-but-wrong property name fails at runtime, not at compile time, and often only under load. Match the project's existing client library, serialization format, and config style rather than introducing a second way of doing it. What "done" means per surface:

• Producer: acks/enable.idempotence/retry config matching the delivery guarantee you claim, serializer + Schema Registry config, a deliberate key strategy (partitioning consequences stated), and a send error path — not fire-and-forget with an ignored future/callback.
• Consumer: group id, offset-commit strategy matched to the processing guarantee (commit-after-process for at-least-once), poll-loop timing within max.poll.interval.ms, poison-message/DLQ path, and defined rebalance behavior.
• Connect: right source/sink connector for the job (on Cloud, a fully-managed one if it exists), converters matching what's actually on the topic, SMT chain in applied order, errors.tolerance + DLQ topic, and exactly-once claimed only if that specific connector supports it.
• ksqlDB: statements with explicit key/value formats and partition counts, co-partitioning verified for joins, and a CTAS-materialized table where pull queries are needed.
• Flink: every Flink pitfall below addressed explicitly — watermarks and lateness, changelog mode vs. sink, state bounds, checkpointing.

Leave one runnable check behind — a test against an embedded broker or Testcontainers, Connect's /connector-plugins/{class}/config/validate endpoint, EXPLAIN on a Flink or ksqlDB statement, or at minimum a produce/consume round trip. Config that has never been executed is a hypothesis.

Review: check for the common pitfalls below, and for anything config-specific, verify current defaults/behavior against docs rather than assuming.

Confluent Cloud (developer-facing)

For provisioning/RBAC/networking use confluent-kafka-admin — this is the "how does my application code talk to Confluent Cloud" side.

• Client connection config: bootstrap.servers (cloud endpoint), security.protocol=SASL_SSL, sasl.mechanism=PLAIN, and API key/secret as sasl.jaas.config (or the client library's equivalent cloud-config helper, e.g. confluent-kafka-python's cloud config block). Never hardcode API keys/secrets in source — use env vars or a secrets manager, same as any credential.
• Schema Registry: Confluent Cloud Schema Registry has its own endpoint and API key/secret pair, separate from the Kafka cluster's — a common source of "works for produce, fails for schema registration" bugs when the two get conflated.
• Local dev/testing: confluent kafka topic produce/consume for quick manual testing against a Cloud topic, confluent flink shell for interactive Flink SQL against Cloud compute pools, and confluent local for spinning up an ephemeral local broker when you want to test without touching Cloud at all.
• Fully-managed services: prefer a Confluent Cloud fully-managed connector, Cloud ksqlDB app, or Cloud Flink compute pool over self-hosting the equivalent, unless there's a specific reason (unsupported connector, custom SMT, cost) — check current Cloud connector and Cloud Flink docs for what's supported before assuming self-managed is required.
• Cluster Linking: if the design spans multiple Cloud clusters/regions or a hybrid Cloud+self-managed setup, that's a confluent-kafka-admin concern for the linking setup itself, but affects application design (topic availability, read-your-writes across linked clusters) — flag it rather than silently assuming a single cluster.

Flink, ksqlDB, and Kafka Connect

Application-level design/review for these three, same research-first rule as everything else — SQL syntax, connector configs, and operators change across versions.

ksqlDB: WITH clause config (key format, value format, partitions), windowed aggregations (tumbling/hopping/session) and their retention, and the fact that a persistent query is an underlying Kafka Streams app — so every Streams pitfall below applies to it. Retrieve current ksqlDB syntax reference before writing non-trivial statements.

Flink (Confluent Cloud for Apache Flink / self-managed Flink): event-time vs. processing-time choice, and Table API vs. DataStream API vs. Flink SQL fit for the task. Correctness knobs — watermarks, changelog mode, state bounds, checkpointing — are in the pitfalls below.

Which docs: Confluent Cloud Flink for the managed surface — catalogs, compute pools, statements, what's supported — via Cloud Flink docs; Apache Flink docs for engine semantics (watermarks, state, joins, functions) and anything self-managed — it has no llms.txt, so either fetch the specific page or query context7, which carries the Flink docs site per release and is the easier route to an exact operator or connector option. Engine behavior is shared, the managed surface is not — don't cite a Cloud-only feature for a self-managed deployment, or an Apache connector/UDF as available on Cloud, without checking. flink-docs-stable tracks whatever the current release is; name the version when a behavior is version-specific.

Confluent Cloud specifics worth designing around: the catalog model maps environment → catalog and Kafka cluster → database, so every topic is already queryable as a table with its Schema Registry schema attached — no DDL to read an existing topic. Statements run on a compute pool (the billing/scaling unit) and long-running ones keep running after you disconnect. Verify current support (Table API, UDFs, connectors to non-Kafka systems, private networking) against the docs rather than assuming parity with open-source Flink.

Pitfalls specific to these

• ksqlDB: pull query against a plain CREATE TABLE table fails — only tables materialized via CREATE TABLE AS SELECT support pull queries. Pull queries on streams are allowed but full-scan the stream: fine for debugging, wrong as a serving path.
• ksqlDB/Streams joins on mismatched keys/partition counts (not co-partitioned) — join silently returns no results instead of erroring.
• Flink: watermark/lateness misconfigured — late events silently dropped instead of handled, or windows never fire.
• Flink: checkpointing disabled or interval too large for the exactly-once guarantee the pipeline claims to provide.
• Flink SQL: changelog mode mismatched with the sink topic — a query producing updates/retractions written as append-only (or an upsert sink with no primary key declared) silently produces a topic that doesn't say what the pipeline thinks it says.
• Flink: unbounded state from a join or aggregation with no TTL/window — state grows forever, and on Cloud that's a bill as well as a failure.
• Connect: SMT chain ordering wrong (e.g., a field-rename after a filter that already needed the original name) — SMTs apply in the order listed.
• Connect: no errors.tolerance=all + DLQ configured — one bad record can halt the whole connector task.
• Connect: assuming a connector supports exactly-once without checking — it's connector-specific (verify against that connector's docs).

Common pitfalls to check for

• Missing acks=all + min.insync.replicas for durability, or missing enable.idempotence where exactly-once producer semantics are needed.
• Partition key choices that create hot partitions (skewed load across consumers).
• Short session.timeout.ms/max.poll.interval.ms or missing group.instance.id (static membership) causing rebalance storms.
• No dead-letter-queue or poison-message handling in consumers/Streams processors — one bad record blocking the whole partition.
• Schema Registry compatibility mode mismatched with the actual evolution pattern needed (e.g., BACKWARD chosen but producers need to add required fields).
• Kafka Streams: unsized/unbounded state stores, missing standby replicas for HA, or exactly_once_v2 not set where required.
• Manual vs. auto offset commit mismatched with the processing guarantee the code actually needs (commit-before-process vs. commit-after-process).
• Partition count treated as easily changeable — it's a hard ceiling on consumer parallelism and effectively can't be shrunk.

Output defaults

• Design: recommended approach + rationale, topic/schema/partitioning specifics, and open decisions for the user.
• Diagrams: Mermaid source, scoped to the question asked (don't diagram the whole system when only one flow was asked about).
• Implement: the code/config, the check that proves it runs, and any setting deliberately left at its default called out — silence about a durability or ordering config reads as a decision nobody made.
• Review: findings framed as root cause + fix location, not just symptoms — same discipline as autopilot's review phase.
• Explanations: cite the specific doc section retrieved, and flag version-dependent behavior.