New in llama.cpp: Model Management

The llama.cpp server now ships a router mode that can load, unload, and switch between multiple models without a restart. It runs models in separate processes, offers auto-discovery of GGUF models, on-demand loading, and LRU eviction (default 4). API endpoints and a web UI let you load/unload models and switch contexts, enabling simple A/B testing and multi-tenant deployments.

Source

HuggingFace Blog

Ingéré

11 DÉC. 2025 · 19:10

Score édito

3.0 / 5