An open Jev-style LoRA on Qwen3.5-9B from Bespoke Labs, trained on 2,676 contrastively curated examples to score the allowed answer tokens directly for enums, booleans and rubric levels. Recipe, data and a public benchmark suite are released with it.
A rank-16 LoRA on Qwen3.5-4B built for the JevBench setting. A document, a policy and a question go in; a calibrated distribution over the option letters comes out of one forward pass. Research and demo use only.
Decides
choice, noul, score, classify, route
choice, score, noul, classify
Architecture
nimble
hopper
Fine-tuned from
qwen/qwen3.5-9b
qwen/qwen3.5-4b
License
apache-2.0
Research and demo use only (training data includes RACE, non-commercial); serving code Apache-2.0
Availability
Open weights + hosted API
Open weights
Hosted by
Bespoke Labs
—
Input price
—
—
Decision accuracy
90.1%
68.5%
Calibration error
0.054
0.102
Valid action rate
—
—
Median latency
106 ms
—
p95 latency
—
—
Figures are from each model’s manifest; accuracy and latency are what the publishers report, on their own suites and hardware. Add a third model.
Questions
What is the difference between bespoke-nimble-9b and hopper?
bespoke-nimble-9b is from Bespoke Labs and hopper from HopitAI. bespoke-nimble-9b has open weights and a hosted API; hopper has open weights you can download and run. Both answer choice, noul, score and classify questions. Only bespoke-nimble-9b answers route. hopper is the smaller model, at 4.0B parameters to 9.0B. bespoke-nimble-9b is licensed apache-2.0; hopper, other.
Which is more accurate, bespoke-nimble-9b or hopper?
They report on different suites — bespoke-nimble-9b 90.1% on Bespoke held-out set (324 examples), hopper 68.5% on JevBench public hard tier (111 items, measured by the authors) — so the numbers do not rank them. Test both on your own labelled examples.
Which is cheaper, bespoke-nimble-9b or hopper?
bespoke-nimble-9b: Hosted, price not published, or free to self-host. hopper: Free (open weights). Open weights cost nothing per call beyond your own hardware.
Can I run bespoke-nimble-9b or hopper locally?
Yes, both: systemone pull bespoke-labs/bespoke-nimble-9b and systemone pull hopit-ai/hopper download the weights.
Evaluation suite
Bespoke held-out set (324 examples)
JevBench public hard tier (111 items, measured by the authors)
