EVALUATION / METHODS & OUTCOMES
The evidence includes
what didn’t win.
Inspect the baseline, the failure boundaries, and the limits of the model comparison. Every displayed metric comes from a recorded artifact.
The retrieval baseline we shipped
Term overlap · expected document at rank one
BM25 · same corpus and questions
Both methods · expected document in top three
Six pinned documents, 180 passages. These small, synthetic document-level checks did not establish that BM25 earns extra complexity. Term overlap is the default. Selection used these cases, so they are now regression data—not a fresh holdout for future changes.
Read every retrieval outcome ↗NEW / TROUBLESHOOTING PROTOCOL
A separate test for generated explanations
6 development cases and 24 held-out synthetic incidents cover probes, image pulls, resource limits, ambiguous symptoms, unsupported questions and instruction-like diagnostic text.
The protocol was committed before model inference. It compares excerpt-only, base Qwen and OpsLM SFT through the same application handler. Completion, valid JSON, matching source quotes, reference-phrase coverage, abstention and latency are separate measurements. Phrase coverage is not factual accuracy.
| Artifact | Quantization | Revision |
|---|---|---|
| Qwen base | Q4_K_M | ce314c106205 ↗ |
| OpsLM SFT | Q4_K_M | 7951bd9f9b27 ↗ |
The complete held-out base/SFT comparison has not been recorded. No SFT quality gain is claimed. Published artifact identities establish what can be tested, not that the experiment has completed.
The overlap check flagged 0 of the new questions against 593 committed SFT inputs using normalized exact matching and 5-gram Jaccard ≥0.6. Semantic and pretraining overlap remain unknown. The two GGUFs were independently converted, which also limits causal claims about SFT.
Checks before an answer is displayed
Evidence must exist
When retrieval returns no passages, no model is called. Retrieval failure in the Python RAG service now produces explicit abstention.
Responses can be rejected
Malformed JSON, missing sources, mismatched quotes, truncated output and unsupported command forms fail validation. No sample answer replaces a failed request.
Support still needs review
A model can quote a real sentence while explaining it incorrectly. Structural checks are useful, but do not establish factual support or comprehensive injection resistance.
uv run python -m opsverse_evals.runbook_fixture --check uv run python -m opsverse_evals.troubleshooting_contamination node opslm-demo/scripts/model-study.mjs --label excerpts --split heldout cd opslm-demo && npm test && npm run build
Synthetic, agent-authored scenarios and coarse reference checks. No independent human answer-quality scoring. Matching source quotes do not prove entailment. Independent GGUF conversions confound causal SFT interpretation. No external-user impact established.