Ort crate onnx runtime session configuration
Skill kjuhwa/skills-hub/skills/inference/ort-crate-onnx-runtime-session-configuration
Configure ONNX Runtime via the ort crate's session builder: inter/intra threads, optimization level, parallel execution, embedded model bytes.From its SKILL.md
npx -y skills add kjuhwa/skills-hub --skill ort-crate-onnx-runtime-session-configurationAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- 0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
SKILL.md
1.8 KB, 292 tokens by cl100k_base, as published. Nobody here has run it
Ort Crate Onnx Runtime Session Configuration
Trigger: Tuning Rust ONNX Runtime inference for your throughput / latency profile, especially when the model is embedded with include_bytes!().
Steps
- Use ort::session::Session::builder() and configure with_intra_threads, with_inter_threads.
- Set optimization level via GraphOptimizationLevel (NoOptimization → Level3) based on startup-vs-throughput tradeoff.
- Embed the model with commit_from_memory(include_bytes!(...)) for single-binary distribution.
- Expose intra/inter thread counts as constructor parameters; respect ORT_INTRA_THREADS / ORT_INTER_THREADS env vars.
- Default thread counts to auto (let ORT pick); only override if you've benchmarked.
- Benchmark your specific model — small models are hurt by high thread counts due to scheduler overhead.
Counter / Caveats
- Too many threads = slower for small models due to scheduler overhead.
- GPU flags are ignored by CPU-only ORT builds.
- Graph optimization is slow on first load; cache the optimized graph or skip it for cold-start-sensitive paths.
- Different models prefer different settings; never assume one configuration fits all.
Source
Extracted from magika (https://github.com/google/magika.git @ main).
Files of interest:
rust/lib/src/builder.rs:20-76rust/lib/src/session.rs:20-36
What ships with it: 1 file
897 B alongside SKILL.md
- content.md897 B