Transformers backend integration in SGLang

SGLang now supports Hugging Face transformers as a backend, enabling high-throughput, low-latency inference for any transformers-compatible model. It can auto-fall back to transformers or be forced with impl='transformers', and adds features like RadixAttention for efficiency. The article includes practical code snippets and deployment patterns (offline engine, server, and OpenAI-compatible API).

Source

HuggingFace Blog

Ingéré

23 JUIN 2025 · 19:10

Score édito

4.0 / 5