Seed Parameter Support
Added seed parameter to OpenAI-compatible endpoints to enable deterministic generation.
What Changed
- Added seed parameter support to OpenAI-compatible chat/completion endpoints
- Enabled deterministic generation guaranteeing identical output for the same seed input
Why It Matters
Reproducible results for identical inputs were essential for model evaluation, regression testing, and quality comparison experiments. Without seed support, the same prompt produced different outputs each time, making quantitative comparison difficult. Reproducible generation was also a core prerequisite when LoRA Land (Zhao et al., 2024) evaluated 310 models × 31 tasks.
Technical Detail
Implemented passing the OpenAI API spec's seed parameter from LoRAX's Rust router to the sampling layer. On the same day, added seed: 42-based deterministic_test.py/json to internal repo to verify consistency between original vs improved endpoints.
Related Internal Work
On the same day (2024-04-03), added deterministic tests to internal repo, completing both upstream feature and internal verification simultaneously. Subsequently used seed-based reproducible generation in the model evaluation pipeline.
View related project →