Rajan Verma

CI for serving · Demo

A gate on generate. Probes you can see. Idle when the GPU is quiet.

Harbor, Fieldline, Ledgerbox, and Folio already call generate or query here. I probe those paths, smoke them before a change lands, and keep an idle policy in writing so a GPU does not run all night. Your engineer gets the same shape: health, a CI gate, a cost policy, and a runbook they can run without me.

How the install works See live examples

How the install works

  1. 01 The generate and query APIs already exist

    I installed them, or you already run them. CI sits around that path — it is not a separate chatbot product.

  2. 02 Health the load balancer can hit

    GET /health on inference and retrieval. Unhealthy means no new traffic. You see it without waiting for a user.

  3. 03 CI smokes the locked paths

    Health, a rejected token, then a real generate and a real retrieve. A bad change fails before it reaches you.

  4. 04 Idle policy in writing

    Replica cap, idle shutdown, no invented dollar-per-token claims. The GPU does not burn all night because nobody asked.

  5. 05 A status page that shows the probes

    Live checks against the APIs on the machine. Not a fake uptime percentage, not a Grafana mock.

  6. 06 I hand the machine to you in writing

    Restart, what “green” means, how to stop. Your engineer is not stuck with my laptop.

What you get

A live status page, a written cost policy, and a runbook

After the install your engineer is not guessing whether generate is up. They hit the probe, read the idle rules, and keep the restart steps. This demo is the board I run against my own inference and retrieval APIs on this machine.

Open live status

Live examples

Example 1 · live · the APIs I already run

Harbor, Fieldline, Ledgerbox, Folio

Four thin UIs on this machine. Three POST to the same generate path. Folio POSTs to query. Fictional product names; the APIs are real and running here. That is why CI for this machine is not a mock — it smokes the same endpoints those pages use. GitHub Actions runs that smoke after each push to a private repo (CPU Ollama on a runner). I copy a green commit to this host; Actions does not deploy it.

A paid install points the same checks at your generate and query URLs. You keep the product screen. I keep the probe, the gate, and the runbook.

Harbor Fieldline Ledgerbox Folio

Example 2 · demo · GPU idle

Hearth

A product team rents a GPU so generate is fast. Nobody is calling it at 2am. The replica stays at one because they might need a demo in the morning. The meter still runs. Fictional product. Not a real firm. No invented dollar-per-month number.

A paid install writes the idle rule on their GPU: fifteen minutes of zero traffic, scale to zero. Cold start is acceptable for that load. This demo machine is CPU. GPU spend here is $0. The policy text lives on the status page — not a fake bill.

Open the idle policy

Example 3 · live · probes

Status

The status page probes inference and retrieval on this machine. If Ollama is down, you see it. Cost policy and runbook sit next to the probes as written — not a dashboard product, and not a fake SLA.

This is CI I run on my own APIs first. A paid job uses your host names and your load balancer. Same click.

Open live status

Contact

Let’s put probes on your generate path

If generate cannot die quietly on your machine, write. Tell me the host, whether you already have /health, and the date. I come back with a probe, a CI gate, a written idle policy, and a runbook. You keep the product UI.

Status shows live probes against the APIs on this machine. No fake uptime percentage. Harbor, Folio, and the rest are labeled Demo. Hearth is a fictional GPU story — this machine is CPU. GPU spend here is $0.