Inference handoff — Northwind Apps (demo)
What you own
- The LLM on your machine — GPU if you have one, CPU if you do not.
- A private generate API in front of it. Same shape if you later swap vLLM in.
- Bearer token in your secrets store — not in chat, not in git.
- Process name (example):
inference.service.
Endpoints
GET https://llm.northwind.internal/health— no token. Load balancer hits this.-
POST https://llm.northwind.internal/v1/generate—Authorization: Bearer …
HTTPS
The API binds on the machine. Caddy or nginx sits in front with your
cert and forwards Authorization. This demo is HTTP.
A paid install documents the proxy you already run — it does not
fake a certificate.
Probe
curl -sf https://llm.northwind.internal/health should
return {"status":"ok"}. If this fails, check the
process before you touch the model.
401 means the token is wrong. 502 means
the model process is down. Your product should show that, not a
blank chat.
Rotate the token
- Mint a new secret (this UI on your network, or
openssl rand -hex 24). - Set
API_TOKEN=in the env file on the machine. - Restart the process. The previous token dies immediately.
- Paste the new value into the product secret. Do not email it.
Start, stop, logs
systemctl start inferencesystemctl stop inference— no hosted vendor to turn off. The machine is yours.journalctl -u inference -f— or the log path on your OS.
Out of this handoff
Product UI, staff logins to the model, training, and 24/7 on-call. Your app keeps the screen. This document is the generate path only.