A model that says how sure it is
Jev, as in Jevons Paradox, TypeSafe’s first model, answers set questions & reports its own confidence. What follows is an attempt to describe what that really changes.
Ask a good ole language model whether a message is a refund request and it generally writes a paragraph explaining its reasoning. The paragraph prolly reads well. Somewhere downstream, code has to turn it into a boolean, and that code has no real way to tell whether the model was certain or a bit broken.
Ask Jev the same thing and it returns yes, plus 0.53 - a confidence score.
Both models say yes. Only Jev sez just how close it was.
What it actually is
Jev is a classifier. Zero-shot, many criteria evaluated per call, calibrated. TypeSafe calls the class System One, after Daniel Kahneman’s fast thinking, and the name fits the behavior: no text generation, no chain of thought, no explanation. A question fixed in advance, a typed answer, a number attached.
Three forms:
-
Choice picks one option from a defined set.
-
Score rates against ordered levels.
-
Noul answers yes or no as a probability btw 0 & 1.
Each returns the full distribution over its options, and a confidence value derived from the shape of that distribution.
A sharp spike means the model has a clear read. A flat spread across 6 options means the options are close, or the state does not contain the answer at all. Confidence collapses that shape into one number that code can compare against a threshold.
console.typesafe.ai/playground
Where the line doth sit
Here lies the heart of it, and the model has little hand in it. The consequence sits here, and not in the model at all.
Automation has mostly been all or nothing. Either a system acts on model output or a person reviews everything, because nothing in the output distinguished the easy cases from the hard ones. With a confidence value, the same call can act alone above one number, ask for confirmation between two, and escalate below a third.
The thresholds live in application code. They can differ per action off the same call: 0.6 for a read, 0.95 for a write. When priorities change, a coefficient changes. The diff is readable. Git blame means something.
An alternative is a sentence in a prompt. It works, until the question becomes what the rule actually might have been.
What changes for people who don’t write code
Prompt-heavy apps tend to put a person inside every loop. Someone reads the reply and decides what happens next. That person is the reason the workflow cannot run unattended, and no amount of prompt refinement removes them, because the output is prose & that prose needs a reader.
Fixing the question changes the shape of the work. The builder writes it once, ahead of time, so it stops being reworded on every run. The answer comes back as a value plus a number. A rule then sits in the software rather than in a prompt: below 0.7, hand it to a person; above that, let it run.
The loop mostly runs itself and pulls someone in when it should. What is left to do is harder and more interesting than prompt wording. Which question to ask. Where the line goes. What happens on each side of it.
Thirteen questions, one call
Batching 13 checks into a single prompt looks efficient until the tokens of answer 3 begin conditioning answers 4 thru 13. That is arithmetic rather than a context window limit, and it may be difficult to catch from the calling code, since contaminated answers often look exactly like clean ones.
Jev evaluates each question in parallel against the same state. No question sees another’s output. Ordering effects go way away.
TypeSafe published a run over one document with thirteen questions: 11.5 times cheaper and 9.6 times faster than separate calls, answers unchanged. Their own cookbook, worth treating as a vendor figure until someone independent repeats it.
The practical effect is that a 14th question costs close to nothing, which is what makes decomposition worth doing at all. A judgment that weighs several factors stops being one question & becomes a formula with visible weights.
Jev runs mainly inside software, not in a chat window
Most of the world will never prompt Jev, nor care. It’s not a chat window & was not built for one. It lives inside software, inside agents, at the moments a program has to decide something & had been reading reams of data to do it.
Where a thing gets used tells you what it is. Chat interfaces are where humans meet models. This one was built for the other side, where machines consult one another unseen.
LLMs were taught eloquence and never taught doubt. Fluency became the proxy for correctness, which worked well enough while a person was reading every answer and stopped working the moment code started acting on them. A model that can tell when it does not know is a different kind of instrument, and Jev is the first TypeSafe has shipped
bsky.app/profile/schwentker.sandboxlabs.ai/post/3mvo45utybs2e
twitter.com/schwentker/status/2100339976596783586
