Chat Completion Stream Fix & API Improvements
Fixed the last delta serialization in chat completion stream to match OpenAI client standards, added a /tokenize endpoint, and improved Swagger documentation.
What Changed
- Fixed last delta serialization in chat completion streaming to be OpenAI client-compatible
- Added /tokenize endpoint for token counting functionality
- Improved Swagger/OpenAPI documentation
Why It Matters
OpenAI-compatible client libraries (e.g., Python openai SDK) encountered parsing errors at stream termination when communicating with LoRAX servers. This fix stabilized configurations where OpenAI SDK is used as-is with only the backend swapped to LoRAX.
Technical Detail
Fixed the Rust-based router layer (router/src/lib.rs, router/src/server.rs) where the last chunk of streaming chat completion was serialized differently from OpenAI's [DONE] protocol. First fixed in internal fork (repo:cbf2b99b35c1#6) on 2024-03-21, then submitted the same change upstream as a PR. LoRAX is a production framework implementing multi-LoRA serving research from S-LoRA (Sheng et al., MLSys 2024) and Punica (Chen et al., MLSys 2024).
Related Internal Work
Fixed in internal fork first (2024-03-21) then contributed upstream (2024-03-27). This was a prerequisite for using OpenAI-compatible clients with the internal LoRAX serving environment.
View related project →