Structured Output Support
Added support for text, json_object, and json_schema via the response_format parameter, strengthening compatibility with OpenAI's structured output interface.
What Changed
- Added text, json_object, json_schema type support to the response_format parameter
- Reduced provider switching costs through enhanced OpenAI structured output API compatibility
Why It Matters
Structured output forces LLM responses to conform to a JSON schema — essential for tasks requiring structured results like contract classification and checklist generation. Supporting this in LoRAX enabled switching between OpenAI and self-hosted serving without code changes. This implemented constrained decoding techniques theoretically analyzed in Grammar-Aligned Decoding (Park et al., 2024) at the serving interface level.
Technical Detail
Parsed the OpenAI API spec's response_format field in the LoRAX router and implemented guided decoding according to the provided schema in json_schema mode. The day after merge (2024-10-17), updated the LoRAX image in the internal registry to the new version and added image repository selection functionality.
Related Internal Work
Internally applied structured output to contract type classification, playbook generation, legal document drafting, and widget responses. After this PR, work proceeded to unify the response_format interface across OpenAI/Azure/LoRAX in the internal middleware.
View related project →