ENGINEERING CASE STUDY / DILIP NALLAMASA
OpsVerse connects Kubernetes diagnostic text to official runbook evidence. The engineering work is in the complete path: ingestion, retrieval, failure handling, provenance, evaluation, and a usable public product.
THE IMPLEMENTED PATH
Revision-pinned Markdown passes through parsing, section-aware chunking, quality filters and security checks.
Known status signatures shape the query. The browser ranks passages from 6 documents and 180 chunks.
Users see detected signals, exact passages, pinned revisions, and links to the full official source.
The exported incident brief records redacted input, source evidence, and the analysis mode for later review.
The public evidence path runs in the browser. A separately configured local model can produce structured drafts through the same request-preparation and evidence-validation functions used by the evaluation runner.
THREE DECISIONS THAT CHANGED THE PROJECT
On 16 synthetic held-out questions, term overlap returned the expected document at rank one in 15 cases; BM25 did so in 14. Both reached 16/16 in the top three. The measurements did not establish an advantage for BM25, so the simpler method ships. These coarse cases are now regression data.
Inspect the methods and outcomes →The original RAG service could fall back to an ungrounded model answer when retrieval failed. It now abstains without invoking the model. The new structured path rejects incomplete output, missing sources, mismatched quotes, and unsupported command forms. Tests exercise those failures through the actual application handler.
Read the failure-path tests ↗The historical capacity selector admitted a constructed run with one success and 99 errors because it checked success-conditioned latency alone. Admission now requires consistent counts, acceptable errors, enough successes, and finite measurements. Repeatability requires a complete matching configuration and distinct run IDs.
Follow the claim-to-evidence ledger ↗REPRODUCIBILITY / ZERO PAID INFRASTRUCTURE
CI regenerates the browser corpus from raw documents, detects stale evidence displays, reruns retrieval checks, tests failure boundaries, and builds the website and API image.
git clone https://github.com/dilipna/OPsVerse.git cd OPsVerse uv sync --all-packages --frozen uv run python -m opsverse_evals.runbook_fixture --check uv run pytest -q cd opslm-demo && npm ci && npm test && npm run build
Inspect CI runs ↗ · Architecture and local generation setup ↗
Lexical matching can miss paraphrases and retrieve tangential material. Matching quotes establish provenance, not entailment. Credential detection is best-effort. The public workflow does not execute cluster commands. A complete served base-versus-SFT comparison is deferred, so no model-quality gain is claimed. Historical GPU measurements and the published SFT artifact remain separately documented research; no external-user or production-reliability impact is claimed.