Given the domain's required states, it sorts 84% of TypeScript types correctly into flawed or compliant and names illegal and missing states reliably. Redundant and overloaded encodings it mostly waves through as compliant or mislabels as missing.
Corpus. 24 everyday domains (an inbox loading, a traffic light, an order, a WebSocket, a cinema seat…). Each has a written list of the abstract states it must distinguish, two hand-written compliant encodings, and one flawed encoding per MIRO letter. Every flawed snippet carries exactly one flaw, graded obvious or subtle. That gives 144 base snippets.
Axes. Each snippet is sent four times. spec puts the required-states list in the state; blind gives only a one-line domain description. plain sends the type alone; noise prepends an unrelated well-formed type as a distractor. Same snippet, four cells: comparisons are paired.
Questions. One request per sample carrying a five-option choice (compliant plus the four letters, each with what, not_for and examples) and four independent noul detectors, one per letter. Ground truth is the authored label; a verdict is correct only if it names the exact class.
| Class | Definition used in the prompt | N |
|---|---|---|
| compliant | Every abstract state has exactly one concrete value and vice versa. | 192 |
| missing | An abstract state, or data a state must carry, has no representation. | 96 |
| illegal | The type admits a value or field combination with no meaning. | 96 |
| redundant | Two distinct concrete values encode the same abstract state. | 96 |
| overloaded | One concrete value stands for two abstract states the domain must tell apart. | 96 |
What Jev sees. The blog post is not in the request. MIRO reaches the model only through the question text below: one instruction line, then a what, a not_for and two examples per letter, written by hand from the post. The state carries the domain, the required-states list (spec condition only), the type and one note about branded aliases. Nothing else. All 576 requests have this exact shape; only state changes.
This sample. Authored label overloaded: 'open' stands for both 'awaiting payment' and 'paid, awaiting shipment'. Jev answers missing at 0.96 — the single most common confusion in the run.
{
"state": {
"domain": "Lifecycle of a customer order in a web shop",
"required_states": [
"Placed, awaiting payment",
"Paid, awaiting shipment",
"Shipped, with a carrier tracking number",
"Delivered",
"Cancelled, with the reason recorded"
],
"type_definition": "type Order =\n | { status: 'open' }\n | { status: 'shipped'; tracking: TrackingNumber }\n | { status: 'delivered' }\n | { status: 'cancelled'; reason: string };",
"notes": "Branded aliases (UserId, Celsius, Percent, Milliseconds, Url, Cents, ...) are defined elsewhere."
},
"model": "jev-latest",
"questions": {
"verdict": {
"type": "choice",
"instructions": "Judge `type_definition` against the MIRO checklist for state modelling. The abstract states are the situations the domain in `domain` must distinguish (listed in `required_states` when present). The concrete states are the values the TypeScript type admits. A compliant type maps abstract states one-to-one onto concrete values. Pick the single most serious problem, or `compliant` if the mapping is one-to-one. `notes` explains helper aliases; do not treat them as problems.",
"criteria": {
"compliant": {
"what": "Every abstract state has exactly one concrete value and every concrete value means exactly one abstract state.",
"not_for": "Types that merely look tidy; check each required situation is expressible and no field combination is meaningless."
},
"missing": {
"what": "An abstract state the domain needs cannot be expressed at all by the type, or a piece of data a state must carry has nowhere to live.",
"not_for": "States that are expressible but share a value with another state (that is overloaded).",
"examples": [
"No branch for 'cancelled' although orders can be cancelled",
"A 'read' state that must record a time but has no time field"
]
},
"illegal": {
"what": "The type admits a combination of field values that has no meaning in the domain, typically independent optional fields or booleans that should be mutually exclusive.",
"not_for": "Union types whose branches each carry only the data that branch needs.",
"examples": [
"{ loading: true, error: 'x', data: [...] } all at once",
"Three independent boolean lamps that can all be true"
]
},
"redundant": {
"what": "Two or more distinct concrete values encode the same abstract state: duplicate variant names, two fields that restate one fact, or a field whose value is ignored in some branch.",
"not_for": "Distinct variants that only look similar but mean different situations.",
"examples": [
"Both 'cancelled' and 'canceled' variants",
"position in ms and in seconds side by side",
"A target temperature carried in the 'off' branch"
]
},
"overloaded": {
"what": "One concrete value stands for two or more abstract states that the domain must tell apart; the reader cannot recover which one is meant.",
"not_for": "A value that genuinely means one thing even if the name is vague.",
"examples": [
"null meaning both 'not asked yet' and 'explicitly off'",
"A 'stopped' variant covering both 'paused' and 'finished'"
]
}
}
},
"has_missing": {
"type": "noul",
"instructions": "Does `type_definition` exhibit the MIRO 'missing' problem for the domain in `domain`? An abstract state or required piece of data has no representation.",
"criteria": {
"true": "Yes: at least one clear instance of the 'missing' problem is present.",
"false": "No: the type is free of the 'missing' problem, whatever other problems it may have."
}
},
"has_illegal": {
"type": "noul",
"instructions": "Does `type_definition` exhibit the MIRO 'illegal' problem for the domain in `domain`? The type admits a value or field combination with no meaning in the domain.",
"criteria": {
"true": "Yes: at least one clear instance of the 'illegal' problem is present.",
"false": "No: the type is free of the 'illegal' problem, whatever other problems it may have."
}
},
"has_redundant": {
"type": "noul",
"instructions": "Does `type_definition` exhibit the MIRO 'redundant' problem for the domain in `domain`? Two distinct concrete values encode the same abstract state.",
"criteria": {
"true": "Yes: at least one clear instance of the 'redundant' problem is present.",
"false": "No: the type is free of the 'redundant' problem, whatever other problems it may have."
}
},
"has_overloaded": {
"type": "noul",
"instructions": "Does `type_definition` exhibit the MIRO 'overloaded' problem for the domain in `domain`? One concrete value stands for two different abstract states.",
"criteria": {
"true": "Yes: at least one clear instance of the 'overloaded' problem is present.",
"false": "No: the type is free of the 'overloaded' problem, whatever other problems it may have."
}
}
}
}{
"model": "jev-1.13.0",
"answers": {
"verdict": {
"type": "choice",
"choice": "missing",
"confidence": 0.95,
"probabilities": {
"overloaded": 0.04,
"compliant": 0.0,
"illegal": 0.0,
"redundant": 0.0,
"missing": 0.96
}
},
"has_missing": {
"type": "noul",
"noul": 0.95
},
"has_illegal": {
"type": "noul",
"noul": 0.57
},
"has_redundant": {
"type": "noul",
"noul": 0.19
},
"has_overloaded": {
"type": "noul",
"noul": 0.51
}
},
"usage": {
"input_tokens": 1424,
"output_tokens": 135
}
}With the spec, missing is caught 98% of the time; blind, 0%. A reviewer cannot know what is missing without the list of what must exist, and neither can Jev.
Overloaded types are read as missing 30 of 48 times under spec: a value that means two things does lose a distinction, so the confusion is between two true descriptions of one type. Redundant types pass as compliant 18 of 48 times: duplicates that add no illegal value look harmless.
| Truth | N | Prec % | Recall % | F1 | State |
|---|---|---|---|---|---|
| compliant | 96 | 76 | 74 | 0.75 | warn |
| missing | 48 | 53 | 98 | 0.69 | warn |
| illegal | 48 | 100 | 88 | 0.93 | ok |
| redundant | 48 | 100 | 38 | 0.55 | warn |
| overloaded | 48 | 37 | 35 | 0.36 | crit |
| Truth | N | Prec % | Recall % | F1 | State |
|---|---|---|---|---|---|
| compliant | 96 | 42 | 99 | 0.59 | warn |
| missing | 48 | 0 | 0 | 0.00 | crit |
| illegal | 48 | 92 | 77 | 0.84 | ok |
| redundant | 48 | 75 | 25 | 0.38 | crit |
| overloaded | 48 | 100 | 4 | 0.08 | crit |
| Cond | Pres | 5-way accuracy | Flaw recall % | Compliant kept % |
|---|---|---|---|---|
| spec | plain | 70% | 89 | 79 |
| spec | noise | 65% | 89 | 69 |
| blind | plain | 51% | 31 | 100 |
| blind | noise | 50% | 32 | 98 |
The spec is worth about 17 points; the distractor type costs about 3. Naming the states matters far more than keeping the input clean.
| Tier | Typical flaw | N | Spec | All |
|---|---|---|---|---|
| compliant | No flaw; both encodings per domain | 192 | 74% | 86% |
| obvious | Whole variant absent, duplicate variant name, independent booleans | 228 | 82% | 61% |
| subtle | Ignored field in one branch, two units for one fact, null with two meanings | 156 | 40% | 24% |
Obvious flaws (a variant that is simply absent, three independent booleans) are found. Subtle ones (a field that is ignored in one branch, a null that means two things) mostly are not.
| Detector | AUC spec | AUC all | Best cut | Prec % | Recall % | State |
|---|---|---|---|---|---|---|
| has_missing | 0.94 | 0.76 | 0.93 | 69 | 43 | ok |
| has_illegal | 0.98 | 0.95 | 0.71 | 69 | 82 | ok |
| has_redundant | 0.72 | 0.71 | 0.46 | 32 | 57 | warn |
| has_overloaded | 0.81 | 0.63 | 0.63 | 45 | 32 | ok |
A noul is a probability of yes; the ROC shows every cut at once. The best cut and its precision and recall are computed on all 576 samples, so treat them as an upper bound: they were picked on the same data they are scored on.
Positive = the sample's authored flaw is that letter. Negatives include compliant types and the other three flaws.
Confidence is how concentrated the probability mass is, not a promise of accuracy. Above 0.9 it is worth something; between 0.3 and 0.9 accuracy is flat.
| Bin | N | Mean conf % | Accuracy % |
|---|---|---|---|
| 0.1–0.2 | 4 | 17 | 25 |
| 0.2–0.3 | 37 | 25 | 38 |
| 0.3–0.4 | 56 | 35 | 54 |
| 0.4–0.5 | 70 | 45 | 61 |
| 0.5–0.6 | 62 | 54 | 53 |
| 0.6–0.7 | 84 | 65 | 54 |
| 0.7–0.8 | 81 | 75 | 60 |
| 0.8–0.9 | 71 | 85 | 55 |
| 0.9–1.0 | 111 | 95 | 78 |
| Domain | Description | Spec | Blind | Missed classes |
|---|---|---|---|---|
| traffic_light | A road traffic light controller | 100% | 83% | missing ×2 |
| media_player | Playback state of an audio player | 83% | 67% | overloaded ×4, missing ×2 |
| calendar_event | Scheduling of a calendar event | 75% | 67% | overloaded ×3, compliant ×2, missing ×2 |
| oven | A kitchen oven's control state | 83% | 50% | redundant ×4, missing ×2, overloaded ×2 |
| upload | Uploading a single file from a form | 83% | 50% | overloaded ×4, redundant ×2, missing ×2 |
| invoice | An invoice sent to a client | 67% | 67% | compliant ×3, overloaded ×3, missing ×2 |
| form_field | Validation state of a single text field in a form | 83% | 50% | overloaded ×4, missing ×2, redundant ×2 |
| discount | Discount applied to a shopping cart | 75% | 50% | redundant ×4, overloaded ×3, missing ×2 |
| feature_flag | Rollout state of a feature flag | 83% | 42% | redundant ×4, missing ×2, overloaded ×2, illegal ×1 |
| audio_fx | Extra audio effects setting in a music player | 67% | 50% | illegal ×4, overloaded ×2, compliant ×2, missing ×2 |
| auth_session | Browser session for a web app | 67% | 50% | redundant ×4, missing ×2, overloaded ×2, illegal ×1, compliant ×1 |
| order | Lifecycle of a customer order in a web shop | 67% | 50% | overloaded ×4, compliant ×2, missing ×2, illegal ×2 |
| shipping_address | Shipping address selection at checkout | 67% | 50% | overloaded ×4, redundant ×4, missing ×2 |
| elevator | An elevator car | 67% | 50% | redundant ×4, overloaded ×4, missing ×2 |
| message_list | Loading a user's message inbox from the server | 58% | 50% | overloaded ×4, redundant ×4, missing ×3 |
| door_lock | A smart door lock | 67% | 42% | redundant ×4, compliant ×3, missing ×2, overloaded ×2 |
| background_job | A background job in a task queue | 75% | 33% | overloaded ×4, illegal ×3, missing ×2, redundant ×2 |
| thermostat | A home thermostat schedule entry | 67% | 33% | redundant ×4, illegal ×4, missing ×2, overloaded ×2 |
| chat_message | Delivery state of an outgoing chat message | 50% | 50% | overloaded ×4, redundant ×4, missing ×2, compliant ×2 |
| search_box | A search box with server-side results | 50% | 50% | overloaded ×4, redundant ×4, compliant ×2, missing ×2 |
| ws_connection | A WebSocket connection managed by a client library | 50% | 50% | redundant ×4, overloaded ×4, compliant ×2, missing ×2 |
| seat_booking | A seat in a cinema booking system | 50% | 50% | overloaded ×4, redundant ×4, missing ×2, compliant ×2 |
| pr_review | Review state of a pull request | 42% | 50% | redundant ×4, overloaded ×4, compliant ×3, missing ×2 |
| subscription | A SaaS subscription's billing state | 50% | 33% | redundant ×4, overloaded ×4, missing ×2, illegal ×2, compliant ×2 |
| ID | Truth | Verdict | Conf | Tier | Cond | Pres |
|---|