OpsLM · Inference Benchmark

vLLM vs Ollama, serving one fine-tuned model

Same model (OpsLM-v1, Qwen3-4B), same Tesla T4, same engine-agnostic harness — so the difference is the serving engine, not the measurement.

Tesla T4 · 15 GB 32 req/level · 1→16 concurrency measured 2026-07-25

Under load, vLLM's continuous batching scales throughput 13.4× while lowering tail latency. Ollama, without it, serializes concurrent requests — throughput stays flat and latency inflates 16×.

vLLM · FP16

13.4×
throughput scaling, concurrency 1→16
p95 latency inflation0.81× ↓

Ollama · GGUF Q4

0.89×
throughput scaling, concurrency 1→16
p95 latency inflation16.0× ↑

Throughput scales — or it doesn't

System throughput (tokens/sec) across the concurrency sweep

vLLM fp16 Ollama q4

Latency holds — or it explodes

Per-request p50 latency (seconds) across the concurrency sweep

vLLM fp16 Ollama q4

Prefix caching

TTFT reduction on requests sharing a long prefix. The vLLM control (cache off) isolates the effect from warm-up.

Same server, same probe — only the --enable-prefix-caching flag differs between the two vLLM rows. That gap is the cache.

Guided decoding

JSON-schema parse rate, constraint off → on (n=4)

vLLM enforces the schema via its guided-decoding backend (0 → 1.00). Ollama's OpenAI endpoint doesn't honor response_format, so it can't be constrained (0 → 0).

Throughput at SLO-admissible configurations

A concurrency level counts only if it meets p95 TTFT ≤ 1s and p95 end-to-end ≤ 30s. Tokens generated past that point are served to users who have already left, and raw throughput counts them anyway.

vLLM holds all three levels. Ollama is serviceable only at c=1 — it violates the contract at 4 and at 16, where its p95 TTFT reaches 64.6 s. That turns a 13.4× throughput headline into a 6.3× admitted throughput ratio: smaller, and the one that survives a latency contract. Cost assumes $0.35/hr T4 — an assumption, not a measurement.

Model registry — OpsLM-v1

QuantArtifactEngineStatus Latency p50 (c1)Throughput (c16)