Benchmarks
Same public held-out suite used around OpenJev: s1lv3rj1nx/openjev-heldout. Model: qwen3.5:9b via Ollama · 100% GPU · n=600 per task.
Why NotJev : Kev looks strong here
- Banking77 at 83% — at or above published Jev held-out (~82%) and JevBench full-test 80.3%, on your own open stack.
- High-cardinality choice works — CLINC OOS 86% across 151 intents with cascade / constraints.
- All three primitives — choice, ordinal score (SST-5 within-1), and noul toxicity gates.
- Reproducible & free to re-run — no proprietary weights required; point Kev at any model you control.
| Task | Primitive | Kev | Chance | Reference |
|---|---|---|---|---|
| Banking77 | choice (77) | 83% | 1.3% | Jev ~82% held-out · JevBench 80.3% |
| CLINC OOS | choice (151) | 86% | 0.7% | — |
| AG News | choice (4) | 86.5% | 25% | — |
| SST-5 | score (5) | 89.5% within-1 | 20% | ordinal ±1 |
| Civil Comments | noul | 81% | 50% | — |
OpenJev private 10k mix (cite only): Jev 85.4% · OpenJev 84.0% — questions not fully public, so we do not invent a score against it. SST-5 uses ordinal within-1; other tasks are exact-match.
Reproduce
KEV_BACKEND=ollama
KEV_OLLAMA_MODEL=qwen3.5:9b
pnpm --filter @kev-ai/server start
pnpm bench:heldout -- --mode api --base-url http://127.0.0.1:3000 \
--tasks banking77,clinc_oos,ag_news,sst5,civil_comments_toxicity