Quantization deployment

Quantization deployment

Quantization maps floating-point weights and activations to low-bit integers, cutting memory 2x to 4x and letting integer tensor cores do the math faster. The catch is that it is lossy: pick the wrong ranges or scheme and accuracy falls off a cliff, often on exactly the inputs you never tested. The discipline is to calibrate ranges on real data, choose the scheme per tensor, and refuse to ship until a quality gate passes.

Method

1. Start with post-training quantization, escalate to QAT only on failure. Post-training quantization (PTQ) needs no retraining: convert an existing FP model and calibrate. If PTQ misses the budget, quantization-aware training (QAT) simulates the rounding during fine-tuning and usually recovers most of the gap, at the cost of a training run. Do the cheap thing first.
2. Calibrate activation ranges on representative data. Weights have fixed values, but activation ranges depend on inputs. Push a few hundred representative samples through the model to record per-tensor histograms, then set clipping thresholds. Entropy or percentile calibration (clipping the top 0.01 percent of outliers) beats plain min/max, which a single outlier ruins.
3. Tame activation outliers before they wreck the scale. In transformers a few channels carry huge activation magnitudes that stretch the range and crush everything else. SmoothQuant shifts that difficulty from activations into weights so a per-tensor activation scale still fits. Apply it before choosing thresholds, not after.
4. Match the scheme to the tensor. Use per-channel (per-output-channel) scales for weights and per-tensor scales for activations; per-channel weight quantization alone recovers much of the loss. Prefer symmetric quantization for weights and asymmetric with a nonzero zero-point for one-sided activations like post-ReLU outputs.
5. Choose bit width against the model's tolerance. INT8 is the safe default and often loses under 1 percent accuracy. INT4 doubles the saving again but needs group-wise scales (for example groups of 128 weights) and a salient-weight scheme like GPTQ or AWQ to stay usable. Do not reach for INT4 before INT8 is validated.
6. Keep sensitive layers in higher precision. The first and last layers, and any layer with wide activation ranges, often dominate the error. Run a sensitivity sweep that quantizes one layer at a time and measures the drop, then leave the worst offenders in INT8 or FP16. Mixed precision trades a little speed for a lot of accuracy.
7. Gate the release on a measured regression. Fix the budget before you quantize (top-1 within 1 percent, or perplexity increase under 2 percent) and evaluate on a held-out set separate from calibration data. Block the deploy if the gate fails, and record before/after numbers and the scheme in the release notes.

Checks

• Does the quantized model meet the pre-stated budget on a held-out set separate from calibration data?
• Were activation ranges calibrated on representative inputs with outlier clipping, not raw min/max?
• Are weights per-channel and activations per-tensor, with symmetric or asymmetric chosen to fit each?
• Did a per-layer sensitivity sweep justify which layers stay higher precision?

Boundaries

This targets integer quantization for inference serving. Floating-point half formats (FP16, BF16, FP8) and their loss scaling are mixed-precision-deployment. Making the resulting INT8 matmuls actually hit the integer tensor cores is tensor-core-utilization. The accuracy budget itself is a product decision: quantization measures the cost, the owner decides if it is acceptable.