Data loading
Skill thada2402/AutoResearchClaw/researchclaw/skills/builtin/tooling/data-loading
Optimize data loading pipeline to prevent GPU starvation. Use when setting up DataLoader or data preprocessing.From its SKILL.md
npx -y skills add thada2402/AutoResearchClaw --skill data-loadingAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- 1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
SKILL.md
0.8 KB, 105 tokens by cl100k_base, as published. Nobody here has run it
Efficient Data Loading Best Practice
- Use num_workers = min(8, os.cpu_count()) for DataLoader
- Enable pin_memory=True when using GPU
- Use persistent_workers=True to avoid re-spawning
- Pre-compute and cache transformations when possible
- For image data: use torchvision.transforms.v2 (faster)
- For large datasets: consider memory-mapped files or WebDataset
- Profile with torch.utils.bottleneck to find I/O bottlenecks
What ships with it
Read from the repository
Just SKILL.md. No reference files, no scripts.