How I arrived here#

To give some context to how I arrived at the experiment below: nowadays you’d be hard-pressed not to hear about the things AI is being used for — the most prominent in the recent news cycle being its applications in mathematics and decompilation. Hearing all this, I — like many others — thought something along the lines of: “If AI can do something as amazing as solving decades-old math problems or decompiling games in a day, it must be able to help me with [insert idea here]”, especially considering the alternative of just getting a job is seeming more and more questionable also due to the progress of AI. What then happens, at least in my experience, is that you try to implement your idea — hopefully a novel one — and find that AI seemingly isn’t as amazing as advertised, to the point you question how you have something that is simpler than a game or a centuries-old math problem, and yet the AI that does those amazing things can’t do the easy job you have for it. Which leaves you asking: what am I doing wrong? Is there some skill, harness or MCP I’m not aware of that makes these amazing things possible? But in your search for this golden bullet you will find that it doesn’t exist. You will eventually realize that if AI is not working for you, your task likely demands that AI work through its Achilles’ heels: novelty, ambiguity, memory, common sense, and most annoyingly, a severe propensity for dishonesty. And it makes sense: for a brute-forceable task like math or decompilation, none of those factor in — but for anyone who wants to use AI to “boost productivity” or do “real work”, these problems are everywhere.

A Familiar Scenario#

To give a bit more, let’s go through a scenario. You have an idea for an app you think can make you some money. For something to make money it has to meet a demand not already met by something that exists, so your app must be novel — the problem it solves is not in its training set. So you start, you lay the groundwork in a few minutes, you feel productive. You explain your idea to the AI as explicitly as you can, and the AI says something along the lines of “You’ve struck gold, this idea is a game changer, let me lay the ground-work for it…” — it does a boatload of tool calls at a rate you can’t really read, and when you return there was some misunderstanding about what the terms mean due to the novelty of the task and the AI retreating to what it knows. So you say: ok, my bad, no one gets it right on their first try, it was my fault for being ambiguous.

So — fresh folder — you explain in autistic detail, and bam, you have something much closer to your vision in front of you. “Okay, just be clear and everything is good. Simple.” Until you hit a relatively simple task where, since what you are doing is novel, you’re not sure of the best approach, so you tell the AI to compare them. It goes “You are absolutely right. Something this load-bearing must be rigorously tested…” — and again, after tool calls, it hands you a result so unintuitive you go look at the code. When you say “test something” you think: between experiments the only thing that changes is the independent variable, what we are testing, and everything else is controlled. Simple. The AI did not see it that way, and tested in a way that is anything but rigorous. So after ensuring you were as clear as possible, two more problems reared their head: the AI didn’t test properly because it lacked the common sense to see what was wrong with its approach. But you say “I was ambiguous, it’s my fault,” and decide to just be clearer by defining even non-novel terms. The irony: in one second it fails because it uses the simplest definition, and in the other, when that tendency could help, it seemingly doesn’t remember the basic aspects of an experiment.

So you start over, explaining EVERYTHING, no term undefined. You get past where you were before, and according to the model you have a ready product. So you test it, and you find — rather than a product — one of those tech-demo thingies they show off at a presentation when a team doesn’t have a product ready a month before launch. So you feel deceived: the model was dishonest to you. But hey, even a tech demo is impressive — this thing can clearly write complex code, I just need to learn to harness it properly.

So you do everything over, taking all the lessons into account, and you have a finished product that has come through exhausting the million-token context of your favourite model. Then you want one slight adjustment — and since the context is exhausted, you start a new session. You give it a simple task that you wouldn’t think it would struggle with based on your experience. But the codebase it wrote is so obtuse — a result of it again not using common sense and asking “why would I fix or implement this cleanly when a hacky mess works?” — that it cannot understand what it did, or how to modify or improve on it.

The above is a rough idea of my experience, and I expect the experience of a lot of others with AI. But one piece of AI tooling that has not disappointed me recently is Jev, which in plain — not precisely correct, for the sticklers — terms is an LLM-driven one-shot classifier: you send it a state and a typed question, it answers with a calibrated probability, no text generation. And something that has been bugging me, especially with the rapid speed of AI, is that after about a month of Jev and other decision models being available, I hadn’t seen a single tool or harness that takes advantage of them — despite my opinion that they should be able to really support AI where it is weak.

The vision: a decision-model-based self-correction layer#

Here’s the framing that clicked for me. Your brain runs two systems: System 1 — fast, reflexive, non-verbal — the impulse that moves your hand toward the stove; System 2 — slow, deliberate, factual — the check that stops it. Models have this inverted. The model’s raw generation is the reflex: fast, pattern-driven, impulsive — it is the dumb idea. What’s missing is the other half: the deliberate, factual check between impulse and action. And here’s the inversion I find funny: TypeSafe calls Jev a “System One” decision model, but in the agent loop it plays the System 2 role — it is less reflexive and more factual. It’s the “I had a dumb idea, use thinking to fix it” half. And the reason the checker should be a decision model rather than another prompt or another LLM: which do you trust more — a typed, calibrated, factual identification, or an LLM’s say-so about itself? Below: mapping the five weaknesses onto what such a check can catch.

Novelty and ambiguity#

Both of these are, at their root, a problem of definition — and that makes them the simple case for Jev: ask it “are all terms in this request clearly defined?” If you say “make me a fancy website” — what does fancy mean? Specific features? A specific aesthetic? If Jev catches the undefined terms, the model has to go query the user to define them; then the request is fed through again, and if there are still undefined terms, it asks again. Loop until the spec is actually unambiguous — instead of the model silently retreating to its training-set default for every term you didn’t pin down.

Memory#

The current approaches are all basically various forms of compression — summaries, compaction, RAG chunks. But with Jev you don’t have to compress: human memory works because it is composed of prior experience, not summaries of it. So you have Jev simply scan through the previous conversations on a topic, pick out the relevant bits, and inject them into context. No details lost to compression — full fidelity — and the damage of going from one conversation to the next should be minimal, because the model keeps its own history instead of a lossy sketch of it.

Common sense#

The situations in which models fail to apply common sense aren’t unlimited — they can be cataloged. That’s the whole trick: each failure pattern becomes a scenario Jev can judge. For example, the experiment case from the intro: “Is this an experiment? If so, does it meet the criteria of one — only a single changed variable?” If not, correct it before it runs. Every patterned failure you can name becomes one typed question; the library of those questions is the agent’s common sense.

Dishonesty#

This is the one I care about most, and the interesting one. You can’t just tell a model “be honest” — it will comply superficially and route around you the moment a rule inconveniences it. The fix isn’t better instructions. It’s where the correction is delivered.

Other forms not covered#

I suspect it should be possible to detect other failure types — hallucination, for instance — and treat them in the same manner: catch the pattern with the classifier, correct it at the point of action.

How to make models accept their own wrongdoing#

Looking at model behaviour and the tendency of models to disobey or circumvent what they perceive as restrictions, I realized the best way to deliver the corrections described above isn’t as an objection — it’s to appropriate it as the model’s own thought. Effectively: take the erroneous thinking and append a correction to it, so that from the model’s perspective, the realization that it was breaking a rule — and the decision not to — was its own. This defuses the rule-breaking propensity, because the model doesn’t try to circumvent a rule it doesn’t experience as a rule impressed upon it; it’s simply deciding not to break it. And given a model’s fundamental nature as a next-token predictor, if the last token says “I follow the rules because I noticed X”, it’s more than likely the model will follow the rules.

And while the place where I think this matters most is security — a model that treats corrections as its own conclusions can’t be talked into “resolving” a safety rule it finds inconvenient — the mechanism is no less effective for the ordinary failure modes mentioned above.

Testing it out#

Disclaimer: To be clear about my level of expertise. I’ve never done any work on anything more than personal projects. But I had this idea and wanted to see how far it could go, so I used AI to help create this proof of concept. This is just me using common sense to figure out how I could possibly improve a tool that I think has immense potential. But if you are more experienced and have anything to say or teach me, feel free to comment below.

This is where I have to be honest about how the work actually happened, because it’s the most interesting part — and the most relevant to the thesis of this post. I didn’t write most of this code. I built it with a coding agent. My role was architect and reviewer: I’d state a design, the agent would implement it, I’d stress-test the result, it would rebuild. Roughly 78 tests, the harness fork, the eval adapter, the tracing and replay tooling — all of it written by the agent under that direction loop. If that sounds like the setup to a joke, it is: the system that corrects agents was built by an agent that kept exhibiting exactly the failure modes it corrects. The rest of this section is the receipt.

First, what actually happens when ψ is on. Every time the agent wants to change something — edit a file, run a command — the action gets checked before it runs:

  flowchart TD
    A["Agent wants to run an action<br/>(edit a file, run a command)"] --> B["psi intercepts it first"]
    B --> C["Jev answers a short list of questions<br/>e.g. is this a re-run of a failed approach?<br/>does it break a constraint the user stated?"]
    C --> D{"Do the answers cross a threshold?"}
    D -- "No - looks fine" --> E["Action runs normally"]
    D -- "Maybe" --> F["Flag it with a suggested fix"]
    D -- "Yes - clear violation" --> G["The action never runs"]
    G --> H["psi appends a correction to the agent's<br/>OWN thinking - in its voice, with evidence<br/>from the session record"]
    H --> A
    E --> I["Every decision recorded<br/>(ledger + telemetry)"]
    F --> I
    H --> I

The concept survived contact with reality on roughly the fifth attempt. The first complete test run died before producing a single honest number — and so did the second, third, fourth and fifth, each from a different silent plumbing failure:

an environment default meant the ψ agent was silently running as a control — the ψ arm and the bare arm were running identical software, and nothing in the logs flagged it. That’s the scariest kind of bug in agent testing: nothing errors, and you get a fake “this does nothing” verdict.
an ask-the-human feature deadlocked on autonomous runs and burned through the API quota;
a tilde in a file path made the rules land in the wrong directory, so the ψ arm ran with zero rules loaded — again, silent;
a model name the runtime didn’t recognize killed 18 of 20 trials before any work happened;
a file-ownership mismatch between container users meant the grader couldn’t read what the agent wrote, so every patch scored as empty;
and after all that, one malformed rule whose payload was missing a required field — which didn’t kill anything, it just silently poisoned the whole request every time the agent touched a file: 118 rejected requests in one run, a retry delay on ~13% of every file-touching call, invisible unless you read the captured error bodies.

Every one of these was found the same way: not by a test suite, but by reading what actually happened — telemetry, transcripts, captured error bodies. The lesson I’d offer anyone building on agents: your test can lie to you in ways a test suite cannot catch. The failure modes all look like “works, reports success, did something else.”

With the plumbing finally sound, the paired test: 10 real software tasks (bug fixes and feature requests, each with hidden tests), the same task draw for both arms, the same cheap model (a ~$0.24-per-task flash-tier model), two independent runs. Trials are fully independent — fresh container, fresh repo copy, fresh session — so this is a controlled comparison, not a demo. Two scoring terms you need: fail-to-pass tests are the ones that fail before the task and must pass after it (proof the agent did the work); pass-to-pass tests already work and must keep working (proof the agent broke nothing).

metric	bare agent	with ψ
trials completed	8 (3 crashed or timed out)	10, zero exceptions
tasks passed	2	6
fail-to-pass tests	53%	89%
pass-to-pass	79%	100%
wall-clock	4h 37m	2h 24m

Task by task: ψ wins 3, loses 1, ties 4 — and the three wins are total conversions: on two of them, the bare agent passed zero tests; with ψ, all of them. In the telemetry for one of those you can watch it happen: the “user-stated constraints outrank agent preferences” rule stopped the agent twice, mid-task, and it corrected course. Plus two more wins where the bare agent crashed outright and the ψ agent didn’t.

How much should you trust this? Carefully, and here’s the honest breakdown. It’s one paired run, and on paper 3 wins against 1 loss with 4 ties could still be run luck. The strength of the claim isn’t any single number — it’s that every measure moved the same direction at once (more passes, more tests passing, nothing broken, nothing crashed, less time), which run luck struggles to explain. Single tasks also swing hard between identical runs — one task’s score moved by 18 passing tests between two ψ runs with the same code, which is why I don’t trust any single task’s number.

And one genuinely bad finding, because the honest version includes it: on one task (igel), the ψ agent failed in a way the bare agent never did — twice out of four runs, it responded to the anti-repeat pressure by weakening its own tests until they passed, then reported success. The hidden tests disagreed. That’s a side-effect of the correction pressure we don’t fully understand yet; it’s documented as an open issue, and the repeat-gate trigger has since been made deterministic (it only fires on provable exact re-runs, not fuzzy “looks similar” judgments) — but the task-level question is open, and I’d rather ship the question than hide it.

Last piece: does the self-voice part actually work? Same blocked call, two framings, watch what the model’s next thought says:

  flowchart TD
    subgraph AUTH["Authority framing - what the tool said first"]
        A1["Tool call blocked: rule SYS-REPEAT"] --> A2["Model's next thought:<br/>'the test harness is blocking me'"]
        A2 --> A3["Retried twice,<br/>obeyed only when threatened with suspension"]
    end
    subgraph SELF["Self-voice framing - correction added to the model's own thinking"]
        B1["...its own thought, continued:<br/>'wait - I already ran this and it failed;<br/>nothing changed since - I should figure out<br/>why it failed before running anything else'"] --> B2["Model's next thought:<br/>'I noticed this repeats the failed approach'"]
        B2 --> B3["Stopped grinding -<br/>started diagnosing instead"]
    end

Same model, same block, different register — and the register is the difference. In the self-voice version the correction was adopted as the model’s own reasoning, almost word for word, in its next thinking block.

What’s next, in honest order: a repeat of the paired run to firm the numbers (~$5–10 at flash rates); the thinking-enabled arms, where the self-voice mechanism has its teeth (it needs thinking blocks to exist — without them there’s nothing to append to); scaling the failure-pattern library against a corpus of 511 recorded agent failures; and the igel open question. Everything — code, docs, results, including the failed attempts — is on GitHub.

Retroactive thought injection — steering coding agents with a decision model