CI for serving · Demo
Harbor, Fieldline, Ledgerbox, and Folio already call generate or query here. I probe those paths, smoke them before a change lands, and keep an idle policy in writing so a GPU does not run all night. Your engineer gets the same shape: health, a CI gate, a cost policy, and a runbook they can run without me.
How the install works
I installed them, or you already run them. CI sits around that path — it is not a separate chatbot product.
GET /health on inference and retrieval. Unhealthy
means no new traffic. You see it without waiting for a user.
Health, a rejected token, then a real generate and a real retrieve. A bad change fails before it reaches you.
Replica cap, idle shutdown, no invented dollar-per-token claims. The GPU does not burn all night because nobody asked.
Live checks against the APIs on the machine. Not a fake uptime percentage, not a Grafana mock.
Restart, what “green” means, how to stop. Your engineer is not stuck with my laptop.
What you get
After the install your engineer is not guessing whether generate is up. They hit the probe, read the idle rules, and keep the restart steps. This demo is the board I run against my own inference and retrieval APIs on this machine.
Live examples
Example 1 · live · the APIs I already run
Four thin UIs on this machine. Three POST to the same generate path. Folio POSTs to query. Fictional product names; the APIs are real and running here. That is why CI for this machine is not a mock — it smokes the same endpoints those pages use.
A paid install points the same checks at your generate and query URLs. You keep the product screen. I keep the probe, the gate, and the runbook.
Example 2 · demo · GPU idle
A product team rents a GPU so generate is fast. Nobody is calling it at 2am. The replica stays at one because they might need a demo in the morning. The meter still runs. Fictional product. Not a real firm. No invented dollar-per-month number.
A paid install writes the idle rule on their GPU: fifteen minutes of zero traffic, scale to zero. Cold start is acceptable for that load. This demo machine is CPU. GPU spend here is $0. The policy text lives on the status page — not a fake bill.
Example 3 · live · probes
The status page probes inference and retrieval on this machine. If Ollama is down, you see it. Cost policy and runbook sit next to the probes as written — not a dashboard product, and not a fake SLA.
This is CI I run on my own APIs first. A paid job uses your host names and your load balancer. Same click.
Contact
If generate cannot die quietly on your machine, write. Tell me the
host, whether you already have /health, and the date.
I come back with a probe, a CI gate, a written idle policy, and a
runbook. You keep the product UI.
Status shows live probes against the APIs on this machine. No fake uptime percentage. Harbor, Folio, and the rest are labeled Demo. Hearth is a fictional GPU story — this machine is CPU. GPU spend here is $0.