The tool goes blind exactly where you need it

Posted on: 28 June 2026

There is still a large population of people who believe that talking to a language model means writing "you are a chef", "you are a financial analyst", "act as a commercial director", and waiting for the costume to produce the competence. It is a mistake, but it is the visible kind, the one anyone who has written three instructions has already learned to spot. What is worth attention starts later, once you think you have outgrown it.

Against the generic prompt sits its apparent cure, which turns out to be a harder condition to diagnose: the enormous, layered prompt, dense with roles, frameworks, references, constraints all meant to fire at once. Whoever builds it is convinced they have bought precision. Often they have bought length. The difference does not show from the outside, and it does not show from the inside of the person writing either, which is where this becomes interesting.

The two prompts fail in opposite ways, and that is the whole point. The vague one fails noisily. It hands back a generic blandness you recognise on sight, and because you recognise it you correct it; this is the failure mode that earns a model a contemptuous nickname over coffee. The sophisticated prompt fails in silence. Past a certain threshold the model stops applying everything you wrote: it keeps some instructions, quietly drops others, resolves on its own the contradictions you did not notice you had written, and returns something pitched in the register of your own framework. It sounds competent because it is assembled from your materials. It cites the references you fed it, holds the tone you asked for, uses the right vocabulary, and for exactly those reasons it clears your review. This is not a hunch. People who have measured the effect find that the degradation comes less from sheer length than from the tension that builds between instructions as they accumulate, and from the fact that models struggle to discard irrelevant material even when they are perfectly able to identify it. The richer the apparatus, the better dressed its errors.

The consequence most people miss is the uncomfortable one. The person who writes "you are a chef" at least knows the prompt is doing little. The person who builds the cathedral of instructions is deceived by their own sophistication, certain they have bought rigour when they have bought volume. The first error is honest; it looks you in the face. The second is mimetic, and an error that dresses well is worse than a clumsy one, because it clears precisely the checks you built to catch it. The review becomes complicit in the thing it was meant to intercept.

Underneath all this runs a tighter mechanism, visible again through the role. When I tell a model "act as a commercial director", I am not transmitting any knowledge of what a commercial director does. I am ceding the authority to decide what that role means. If I lack that knowledge myself, whatever the model produces will look right, because I hold no measure with which to call it wrong. The role works, but only in the case where I could already correct the output unaided, which is the case where I needed the role least. Where I would genuinely need to delegate, the instrument is blind; where I do not need it, it sees perfectly well. It is a verification that abdicates the moment you call on it.

From here grows a larger illusion, one that affects the people who work with these tools daily and become fluent in them. Fluency with a single model, the knack of pulling high answers from a machine you have learned to read, resembles general competence very closely and need not be it. You can become superb at speaking a private language, made of your particular way of asking and that particular model's way of answering, and mistake that fluency for mastery. The real test is not the brilliant answer you get in the room where you are at home. It is a model you have never used, a task where you cannot already judge the output, someone reading your requests without knowing what you wanted and trying to guess. That is where transferable skill separates from the confidence of having simply logged many hours in one place. And confidence, unlike skill, grows undisturbed precisely as the ground beneath it narrows.