agentsclimarketplace

Training debug

Skill RobotSail/rhai-post-training-plugin/skills/training-debug

Debug training issues including OOM, NaN losses, and distributed training problemsFrom its SKILL.md

Install
npx -y skills add RobotSail/rhai-post-training-plugin --skill training-debug

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

  • no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
  • 4 stars4 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

5.3 KB, as published. Nobody here has run it

Training Debug

Diagnose and fix common training issues in training_hub.

Out of Memory (OOM) Issues

Symptoms

CUDA out of memory. Tried to allocate X GiB
RuntimeError: CUDA error: out of memory

Diagnosis Steps

  1. Check GPU memory usage
nvidia-smi -l 1  # Monitor GPU memory in real-time
  1. Estimate memory requirements
from training_hub.utils import estimate_memory

estimate_memory(
    model_path="meta-llama/Llama-3.1-8B",
    batch_size=4,
    seq_length=4096,
    precision="bf16",
)
# Output: Estimated memory: 24.5 GB per GPU

Solutions

Reduce batch size

sft(
    per_device_batch_size=2,  # Reduce from 4
    gradient_accumulation_steps=16,  # Increase to maintain effective batch
)

Enable gradient checkpointing

sft(
    gradient_checkpointing=True,  # Trades compute for memory
)

Use memory-efficient optimizers

sft(
    optim="adamw_8bit",  # 8-bit Adam saves ~50% optimizer memory
    # OR
    optim="paged_adamw_32bit",  # Pages optimizer states to CPU
)

Reduce sequence length

sft(
    max_seq_length=2048,  # Reduce from 4096
)

Use FSDP for large models

sft(
    fsdp=True,
    fsdp_config={"sharding_strategy": "FULL_SHARD"},
)

NaN/Inf Loss Issues

Symptoms

Loss: nan
Loss: inf
Training loss spiked to very large values

Diagnosis

# Enable anomaly detection
import torch
torch.autograd.set_detect_anomaly(True)

# Check for NaN in inputs
from training_hub.utils import check_data_quality

issues = check_data_quality("./data/train.jsonl")
print(issues)  # Reports NaN, inf, empty examples

Solutions

Fix learning rate

sft(
    learning_rate=1e-5,  # Lower from 2e-5
    warmup_ratio=0.1,     # Ensure warmup
)

Use loss scaling for fp16

sft(
    fp16=True,
    fp16_opt_level="O1",
    loss_scale=128.0,  # Or "dynamic"
)

Prefer bf16 over fp16

sft(
    bf16=True,  # More stable than fp16, no loss scaling needed
)

Gradient clipping

sft(
    max_grad_norm=1.0,  # Clip gradients
)

Check and clean data

# Remove problematic examples
from datasets import load_dataset

dataset = load_dataset("json", data_files="train.jsonl", split="train")
dataset = dataset.filter(lambda x: len(x["messages"]) > 0)
dataset = dataset.filter(lambda x: all(
    isinstance(m.get("content", ""), str) for m in x["messages"]
))

Distributed Training Issues

NCCL Timeout

RuntimeError: NCCL timeout
Watchdog caught collective operation timeout

Solutions:

# Increase timeout
export NCCL_TIMEOUT=1800  # 30 minutes

# Debug NCCL
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=ALL

GPU Mismatch

CUDA error: invalid device ordinal
RuntimeError: Expected all tensors to be on the same device

Solutions:

# Verify GPU visibility
export CUDA_VISIBLE_DEVICES=0,1,2,3  # Explicitly set GPUs

# Check GPU topology
nvidia-smi topo -m

Data Loading Bottleneck

# Symptoms: Low GPU utilization, CPUs maxed out

sft(
    dataloader_num_workers=8,  # Increase workers
    dataloader_pin_memory=True,
    dataloader_prefetch_factor=4,
)

FSDP-Specific Issues

Checkpoint Loading Failures

# Use FSDP-compatible checkpoint saving
sft(
    fsdp=True,
    fsdp_config={
        "state_dict_type": "SHARDED_STATE_DICT",  # For large models
    },
)

Memory Fragmentation

sft(
    fsdp=True,
    fsdp_config={
        "limit_all_gathers": True,
        "forward_prefetch": True,
    },
)

Data Issues

Tokenization Errors

from training_hub.utils import validate_data

# Check if data tokenizes correctly
errors = validate_data(
    data_path="./data/train.jsonl",
    model_path="meta-llama/Llama-3.1-8B",
)

for error in errors:
    print(f"Row {error['index']}: {error['message']}")

Sequence Length Distribution

from training_hub.utils import analyze_data

stats = analyze_data(
    data_path="./data/train.jsonl",
    model_path="meta-llama/Llama-3.1-8B",
)

print(f"Mean length: {stats['mean_tokens']}")
print(f"Max length: {stats['max_tokens']}")
print(f"Examples > 4096: {stats['exceeds_4096']}")

Debugging Workflow

  1. Start with minimal config
sft(
    model_path="...",
    data_path="...",
    per_device_batch_size=1,
    max_steps=10,
    bf16=True,
)
  1. Gradually increase complexity
# Step 1: Increase batch size until OOM
# Step 2: Add gradient checkpointing if needed
# Step 3: Add evaluation
# Step 4: Full training run
  1. Enable logging
import logging
logging.basicConfig(level=logging.DEBUG)

sft(
    logging_steps=1,
    logging_first_step=True,
)

Diagnostic Commands

# GPU status
nvidia-smi

# Check CUDA version
nvcc --version
python -c "import torch; print(torch.cuda.is_available())"

# Check distributed setup
python -c "import torch.distributed as dist; print(dist.is_available())"

# Memory profiling
python -c "import torch; torch.cuda.memory_summary()"

Related Skills

  • /training-configure - Configure training runs
  • /pipeline-design - Design end-to-end pipelines

Keep looking

Skills are one crate of 610,045. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.