agentsclimarketplace

Kafka batch consumer partition tuning

Skill kjuhwa/skills-hub/skills/backend/kafka-batch-consumer-partition-tuning

Self-correcting knowledge corpus for Claude Code — 9 stable shape clusters, bias-correction pipeline baked into contribution flow. 47 papers, 45 techniques, 1.1k skills.

Install
npx -y skills add kjuhwa/skills-hub --skill kafka-batch-consumer-partition-tuning

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Size Kafka partition count + max-poll-records per topic volume so heavy-processing batch consumers avoid max.poll.interval.ms breaches and maintain throughput via per-partition concurrency.

SKILL.md

2.1 KB, as published. Nobody here has run it

Kafka Batch Consumer Partition Tuning

Shape

Two knobs control whether a batch listener survives heavy load:

  1. Partition count per topic — scaled by volume; upper bound on listener concurrency.
  2. max-poll-records — scaled down for heavy work so poll() cadence stays well inside max.poll.interval.ms.

Steps

  1. Classify topics by volume: high-volume (measurements, metrics) vs. low-volume (config changes, admin).
  2. Provision partition count accordingly (e.g. 9 for high, 3 for low).
  3. Listener factory: setBatchListener(true) + setConcurrency(numPartitions) — one thread per partition.
  4. Lower max-poll-records for heavy consumers (e.g. 10). Per-poll overhead is tiny in Spring Kafka; more frequent polls beat fatter batches.
  5. AckMode.BATCH — offsets commit after each batch, not per record.
  6. setAutoStartup(false) — start containers programmatically after service-dependency checks.
  7. Wrap key/value deserializers with ErrorHandlingDeserializer and attach a DeadLetterPublishingRecoverer.
  8. Expose queue/lag metrics via Actuator/Prometheus.

Counter / Caveats

  • Don't over-partition low-volume topics — metadata + replication overhead with no benefit.
  • Partition count is costly to shrink; size deliberately (~3× current peak, not 10×).
  • I/O-bound workloads are capped at one-thread-per-partition — extra concurrency wastes threads.
  • See linked knowledge entries for the project-specific rationale on batch-size and partition topology.

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.