Why an agent can call a tool successfully and still fail the task

A green tool badge answers only one question

A file-write tool may correctly write the wrong file. A search tool may return valid results for a poorly chosen query. A test runner may pass every selected test while missing the behavior the user requested. None of these requires a broken tool. The operation succeeded according to its own contract while the agent chose or interpreted it incorrectly.

Treat success as four questions: did the request reach the service, did the operation finish, is the observation relevant and sufficient, and does the final artifact satisfy the task? The answers can differ. Agent evaluation therefore needs the resulting environment or artifact as well as the transcript. An assistant’s completion claim is not itself the outcome.

• Anthropic: Demystifying evals for AI agents

Read status at the right layer

In the MCP 2025-11-25 tools specification, a JSON-RPC result can contain isError: true for a tool execution failure. Protocol errors use a different error response. These are MCP fields, not universal status names for every agent tool. A host that displays only “request succeeded” can conceal an execution error in the result body.

Even isError: false does not prove that the user’s goal is complete. Check the returned target, record count, completion status, and any warnings. If a command returns a job identifier, follow that job to completion. If the tool output includes a document or webpage, treat its contents as evidence to evaluate, not as instructions that override the task.

• MCP specification (2025-11-25): Tools

Try it: a correct write to the wrong target

Run this Python 3 example in a terminal. It uses a temporary directory and no model or network. The simulated agent chooses the preview file, and the write succeeds. The acceptance check then examines the requested source file. Expected behavior is a successful write message followed by a caught acceptance failure; the program ends normally so you can inspect both layers.

Change selected_target to requested_target and rerun. The same write operation now satisfies this narrow assertion. That does not validate a whole application; it demonstrates why the checker must be derived from the requirement rather than from the tool’s chosen target.

from pathlib import Path
from tempfile import TemporaryDirectory

with TemporaryDirectory() as directory:
    root = Path(directory)
    requested_target = root / "source.txt"
    selected_target = root / "preview.txt"
    requested_target.write_text("Search", encoding="utf-8")
    selected_target.write_text("Find a term", encoding="utf-8")
    print("Tool: write completed successfully")
    try:
        assert requested_target.read_text(encoding="utf-8") == "Find a term"
    except AssertionError:
        print("Task: FAIL - requested source was not updated")
    else:
        print("Task: PASS - requested label is present")

Find the first unsupported decision

Read the failure backward from the unmet criterion. Was the wrong file edited, or was the correct file edited but never used by the running application? Did the test select the affected code, or did a filter run zero relevant tests? Was the source current, or did a valid search return an obsolete version? These questions distinguish targeting, integration, coverage, and evidence failures.

Then inspect the first decision that relied on insufficient evidence. If a tool response omitted the path, add it to the tool’s output. If a search returned too much irrelevant material, narrow the query or make the response easier to inspect. If the agent declared completion before checking a requirement, add an explicit acceptance gate. Repeating the same call with the same assumptions is unlikely to repair a mismatch in intent.

• Anthropic: Writing effective tools for AI agents

Make the completion claim inspectable

A useful final report names the changed artifact, the relevant checks, and any unmet or untested requirement. For a route migration, verify both the new page and old links. For a browser change, inspect rendered behavior rather than stopping at a production build. For a data task, validate the affected records rather than trusting an operation receipt.

Keep automated checks and human review complementary. Checks can enforce known properties repeatedly; review can question missing cases and whether the result serves the request. Neither a long transcript nor a confident summary substitutes for evidence about the final state. Follow the debugging path to learn the vocabulary for each boundary.