Massive breakthrough here! Self-hosting LLMs just got ~75% cheaper: Most agent pipelines now run 4-5 small models under the hood: an embedder for retrieval, a reranker for precision, an extractor for entities, and often a small LLM for routine generation. The standard way to serve them is one server per model. vLLM holds the LLM, TEI holds the embedder, and everything else gets a custom FastAPI wrapper. Each server reserves its own slice of GPU memory and holds it whether traffic arrives or not. GPUs are billed by the hour, so idle time costs the same as busy time. This is why switching to small models rarely reduces the bill. The cost was never in the calls but rather in the servers. The structural fix is serving every model from one process that loads and evicts models based on traffic. Superlinked just open-sourced exactly that. SIE (Superlinked Inference Engine) is an Apache 2.0 server that runs 85+ models behind one API. Four calls cover the whole pipeline: - encode() returns vectors - score() returns relevance scores - extract() returns entity spans - and generate() runs small open LLMs. Models load on first request and are evicted least-recently-used, so one GPU serves a rotating set of models instead of sitting siloed behind one. It runs anywhere from a laptop to a Kubernetes cluster, and it plugs into Qdrant, Weaviate, Chroma, LanceDB, LangChain, and LlamaIndex. You can find the repo here: github.com/superlinked/sie (don't forget to star 🌟) I also wrote a hands-on guide to using SIE. It walks through a real-world example where we deploy and manage five models on a single GPU, including a small language model, an OCR model, an NER model, a reranker, and an object detector. The article is quoted below.