Kernel validation with corrupted cache detection

Kernel Validation With Corrupted-Cache Detection

What / Why

A JIT cache with an empty entry (compilation interrupted, SIGKILL, out-of-disk) is worse than no cache: the load path silently succeeds with garbage and fails deep in cuModuleLoad with an opaque error. Validate the cache-directory contents before trying to use them, and fail fast with a message that tells the user exactly how to recover.

Procedure

1. Define expected artifacts per compile path:
   • NVCC path: kernel.cu, kernel.cubin (plus optional kernel.ptx).
   • NVRTC path: kernel.cu, kernel.cubin from in-memory compile.
2. Validate at load.

   static bool check_validity(const std::filesystem::path& dir) {
       if (!std::filesystem::exists(dir)) return false;
       if (!std::filesystem::exists(dir / "kernel.cu"))    return false;
       if (!std::filesystem::exists(dir / "kernel.cubin")) return false;
       return true;
   }
   

3. On validation failure inside the runtime constructor (not check_validity, which just returns bool): abort with a specific message:

   Corrupted JIT cache directory (missing kernel.cu or kernel.cubin): <path>,
   please run `rm -rf <path>` and restart your task.
   

4. Fast-path hit uses check_validity to decide whether to skip the compile; this also means a half-populated dir forces recompilation, healing itself.

Key design points

• Include the directory path in the error — grep-able and directly actionable.
• Point to rm -rf explicitly; don't make the user guess whether it's safe to delete.
• Don't auto-delete corrupt caches without user confirmation — a disk fault could mask a deeper issue.

References

• csrc/jit/kernel_runtime.hpp — KernelRuntime::check_validity and the assert site.