Once you host LLM inference on more than one machine, three boring problems show up. Nobody knows what each box is actually doing for you. Every client has to know which server serves which model today. And every time your fleet changes — a box is added or dropped, or its network address moves, as with a VPN when you work from another location — you end up updating the model provider settings in every agent and harness.
El Pulpo is a proxy that sits between your clients and your inference boxes and solves all three: your clients point at one stable endpoint and one model catalogue that do not change when the topology underneath does, it registers the usage of every request and serves a web dashboard to monitor token usage and savings, and it keeps the configuration of many LLM servers in one place. A fleet change is now a single edit instead of a hunt through every client. Everything is configurable through that web dashboard and takes effect without a restart. The whole thing is deliberately lightweight: a single static binary with no CGO requirement, and a 12.7 MB multi‑arch container image.
docker run -d --name elpulpo -p 8080:8080 \
-v $PWD/config:/etc/elpulpo -v elpulpo-data:/var/lib/elpulpo \
ghcr.io/claw-destine/elpulpo
Open the dashboard, add a host, done — a missing config file is a valid empty configuration, and the
first save creates it. Clients address El Pulpo with the OpenAI chat API; it rewrites the model
field to the base name the serving process actually understands, forwards the request, and streams
the answer through unbuffered.
Configure your servers in one place
Add a host, give it one or more servers — Ollama or any OpenAI‑compatible endpoint such as
vLLM or llama.cpp server — and El Pulpo probes them and publishes their models as ids like
qwen3.8:27b-ollama@minion1. GET /v1/models lists exactly what you may request, so your agents and
apps point at one proxy instead of at N boxes. The configuration is a single YAML file, but the
dashboard is the editing surface: add, edit and delete servers, export the YAML, or import it back
with an added/removed/changed preview. Hand edits on the file are picked up within seconds; invalid
ones are refused with the error shown on screen and the live configuration left intact.
El pulpo adapts when some of your servers are down or change addresses
The same machine is often reachable under several addresses — LAN first, VPN as a fallback — so you configure each host with an ordered address list. Every server keeps the first reachable address as its active one and moves to the next when it stops answering; the dashboard shows the active address and flags it when it is a fallback. Three consecutive probe failures mark a server down and withdraw its models from the catalogue; the first success revives it. There is no fail‑back, so traffic does not flap when a VPN address comes back on its own. Models loaded or unloaded upstream are picked up too — every health probe refreshes the server's model list.
Load balancer / aliases
A route puts one requestable name in front of several machines: the same model on two boxes,
named qwen, is what your clients ask for. Each member carries a cost — what one in‑flight
connection to that box is worth, small on a fast machine and large on a slow one — and El Pulpo aims
every request at the member with the least in‑flight cost, so the machines converge on equal load.
With costs 1 and 3, the fast box absorbs three times as many connections before the loads meet. A
member that cannot answer steps aside without a retry, and usage is recorded against the model that
actually served the request, so statistics stay honest. The dashboard shows live in‑flight counts per
member and which member a request arriving now would go to.
Check the usage statistics
Every request ends up as a row: tokens in, out, cached and reasoning, latency, time‑to‑first‑token and status. The Statistics screen filters and groups them by model, host, server or day, with p50/p95 latencies, a totals row and CSV exports whose numbers always match what the screen shows. This is the answer to the question "is self‑hosting worth it, and where?"
Set the prices
Declare input and output prices per million tokens and El Pulpo computes the cost of every request and your savings against the cloud reference prices. A bundled catalogue of ~24 cloud models offers one‑click Apply for model names you have usage for but no price set. Models without a price are flagged rather than silently estimated — better to show the hole in the picture than to invent numbers for it.
Contact us to discuss your GenAI strategy.