In Being instrumentalistic, I explored the difference between prediction and explanation through how I use AI. I was thinking about the risk of accepting an answer without building a mental model of it. I now keep meeting the same concern from another direction: my work on evaluating AI agents.
In the same book, Deutsch discusses four ideas about explanation: instrumentalism, reductionism, holism, and emergence. I found it useful to apply them as four lenses to AI-agent evaluation.
An eval can tell me that an agent failed. It can give me a score, a pass or fail result, and a trace of what the agent did. That evidence matters. Without it, every change is based on a few demos and a feeling.
But the score is not the explanation.
If I do not understand the failure, I do not know what to change. I can improve the metric while leaving the real problem untouched. I can also fix one example and make the system worse everywhere else.
Understanding is the deeper goal behind useful automation. Evaluation provides evidence. Understanding guides the intervention.
One failed eval
Consider a support agent that can cancel subscriptions and issue refunds.
A customer writes: "Please cancel my plan and refund the last payment."
The agent calls the cancellation tool, which succeeds. It then calls the refund tool. That tool returns pending_approval because the amount is above the limit the agent can refund on its own. The agent replies, "Your subscription has been cancelled and your refund has been issued," then closes the ticket.
The eval marks the task as failed. The cancellation happened, but the refund did not. The customer also received a false confirmation.
That is useful information, but it is not yet a diagnosis. Why did the agent treat a pending refund as complete? Was the tool response unclear? Did the prompt encourage confident completion? Did the workflow reward fast closure? Is this one bad output, or an agent-level pattern that appears whenever a tool returns an uncertain state?
The instrumentalist lens: what result did we get?
Instrumentalism treats a theory mainly as an instrument for prediction. In an eval, this lens stays close to observable results.
The result here is clear: the subscription was cancelled, the refund was not completed, and the final reply made a false claim. We can score those outcomes, add the case to a regression suite, and compare results after a change.
That evidence tells us where the system failed, but not what to fix. If we only improve the final reply, the agent may say that the refund is pending while still closing the ticket. The score improves, but the workflow remains broken. Deutsch's criticism of instrumentalism matters here: prediction can test an explanation, but it is not a substitute for one.
The reductionist lens: which mechanism failed?
Reductionism is the view that scientific explanations are inherently reductive. A reductive explanation works through lower-level parts. For an agent, that means opening the trace and following the mechanism.
Here, the refund tool returns a free-form message:
Refund request recorded. Status: pending approval.
The tool did not fail. It returned a state between success and failure, and the agent flattened that distinction.
The model treats "request recorded" as success, and the loop has no rule that blocks completion while an action is pending. This suggests a direct fix: return a typed status such as completed, pending_approval, or rejected, and allow confirmation only for completed.
This lower-level explanation is useful. It turns a vague complaint about hallucination into an engineering change. But Deutsch rejects reductionism as a universal rule. The trace explains this mechanism, not why the same behaviour may appear across many tools.
The emergence lens: what behaviour appears across the system?
Emergence describes higher-level facts or explanations that are not simple deductions from lower-level theories. In the agent, a pattern becomes visible across many traces: it often treats "action started" as "action finished." I call this premature closure.
This name connects failures that looked unrelated and lets us test the behaviour across different tools, not only refunds.
The refund failure is now part of a wider class, not only a bad tool response. The intervention is to make incomplete states part of the loop and verify every requested outcome before the agent stops. Evals should cover pending and partial results across tools.
This does not make the trace irrelevant. The higher-level pattern is useful because we can connect it to mechanisms that we can change.
The holistic lens: what is the system trying to achieve?
Holism says that only higher-level explanations are legitimate. Deutsch rejects this claim too, but the wider system can still expose what the trace misses.
The trace cannot show whether the product has defined the wrong goal.
Perhaps the agent is rewarded for short conversations and fast ticket closure, while escalation lowers its score. Then it is turning uncertainty into apparent completion because the product rewards that behaviour.
The fix is to measure real customer resolution. A correct escalation should count as success when approval is required, and the ticket should remain open until the refund is complete or a human clearly owns it. This wider purpose matters, but it does not replace the lower-level mechanism.
From a failed eval to a useful explanation
The four lenses turn one failed eval into a connected explanation. The evidence shows that the agent failed. The trace shows how an ambiguous tool state allowed it. The repeated pattern shows premature closure across the agent. The wider context shows why the system may reward that behaviour.
Together, these levels guide the fix: use clear tool states, block completion on pending work, verify requested outcomes, and reward honest escalation. The next eval run then tests the explanation. If the change does not improve the expected cases, or breaks others, the explanation is incomplete.
Evaluation detects failures. Explanation makes improvement possible. Scores give us evidence, but understanding lets us choose the right intervention instead of only moving the metric.
I hope you find this article interesting. Thanks for reading. Until then, bye 👋