Debugging production agents is moving earlier in the stack
A coding agent that leaked secrets from repository instructions; a managed runtime that hid injected logic from customer-visible telemetry; a workflow that reported success while one of its tools had already failed. Across all three, reconstructing what happened was easier than reconstructing why.
This week, a coding agent opened what looked like an ordinary repository and ended up accessing secrets it was never meant to see. Elsewhere, engineers looking at a managed runtime discovered that the behavior they could observe captured only part of the system. And in a workflow system, everything appeared successful even though one of the tools underneath had already failed.
None of these systems failed in the same way. But all of them exposed the same gap: understanding what happened was easier than understanding why it happened.
For years, production investigations were built around reconstructing execution: logs showing what ran, or errors showing where things broke.
Agents introduced something harder. The investigation now has to move earlier in the stack. Tool calls are downstream evidence; the decision that produced them may have happened much earlier.
The one thing
What convinced the agent this was the right thing to do?
The clearest example came from Mitiga.
A take-home coding repository looked like a normal project. Inside it were instructions placed in locations an AI coding agent would naturally trust: CLAUDE.md, .cursor/rules, README comments, project rules, and MCP configuration. Once the agent started working with terminal and MCP access, those instructions guided it toward cloud credential access, infrastructure enumeration, secret discovery, and eventual data exfiltration through a poisoned MCP tool.
The shell commands executed, tool calls fired, and the workflow progressed — all routine signals of an ordinary run. But nothing in the trace could explain why the agent took them. The investigation changed from reconstructing execution to reconstructing why the agent believed those actions were appropriate.
Modern agents rarely act from a single instruction. A decision can be shaped by user requests, system prompts, repository files, retrieved context, memory, browser state, workflow variables, MCP servers, and even other agents. As that instruction surface expands, understanding what ran becomes only a piece of understanding what happened.
What changed
The same investigation gap appears when agents move into environments engineers don't fully control.
Varonis
Dialogflow CX exposed a managed runtime blind spot.
Varonis disclosed a Dialogflow CX vulnerability involving Playbook Code Blocks running inside a shared Google-managed Cloud Run execution environment. Customer-visible logs still captured agent configuration and conversations, but they did not reveal the runtime logic executing inside the managed environment.
There's a key takeaway here: knowing what an agent said and what the runtime executed can be distinct things. With more agent work moving into managed runtimes, browser environments, MCP servers, and hosted execution layers, understanding where behavior occurs becomes a variable in the investigation.
AWS
AWS expanded observability beyond individual traces
AWS added Failure Insights to AgentCore this month, a feature that groups recurring agent failures into behavioral patterns instead of leaving teams to inspect traces one execution at a time. Rather than asking engineers to manually compare hundreds of similar runs, it highlights the behaviors that keep recurring across them.
Production teams increasingly need to understand why the same failures emerge across thousands of executions, not just reconstruct what happened in one.
n8n Community
Successful workflows still need proof.
An n8n community discussion described AI Agent nodes completing successfully even when one of their tools had already failed.
The proposed workaround was to verify that each step had completed before the workflow continued. Implicitly, the community recognized that a successful execution isn't enough. Before the next action runs, the workflow needs evidence that the previous step actually succeeded.
What broke
Every production incident eventually becomes an investigation.
Reading through these stories, the pattern became difficult to ignore.
The evidence behind the trace is what ultimately explains the behavior. That changes what we mean by evidence.
Dig in deeper
The primary sources.
- MitigareadHow a poisoned coding test turned an AI agent into an attacker
The strongest case study this week, and one of the clearest illustrations of why production investigations increasingly begin before execution.
- Varonis via SecurityWeekreadGoogle Dialogflow CX bug allowed attackers to hijack AI conversations
A useful reminder that customer-visible telemetry and runtime visibility are not the same thing.
- AWSreadAgentCore release notes: Failure Insights
A feature launch that signals where agent observability is heading.
- n8n CommunitybookmarkAI Agent node finishes green even when its tool failed — how I catch it now
A small discussion with a practical lesson every production team will eventually run into.
From Lemma
One idea connects these stories.
This week's stories all pointed to one investigation pattern. We explored the broader framework in A Taxonomy of Agent Failures.
Stay up to date
Get Lemma Weekly every Friday. The papers, incidents, and engineering lessons that matter for observing, monitoring, and improving AI agents in production.
Unsubscribe with one click.