Enterprise AI

AI Production Debugging: Tools, Techniques & Best Practices

How AI tools help find the cause of production incidents, where they fall short, and what to measure instead of one MTTR number.

Cici Feng13 min read

On November 12, 2026, a Microsoft 365 outage left many customers without Outlook and parts of Teams for nearly a day. Microsoft restored service in stages. It then said the incident "was introduced by an issue during an earlier network capacity upgrade that surfaced following a recent network change," and promised a full report after its review. Service came back well before the explanation did.

AI production debugging is the use of AI to find, explain and fix the causes of failures in live software. Some tools summarize an alert. Others are agents that query logs, metrics and code to test theories about a cause. This article covers incidents in production software, not the debugging of AI agents that run in production.

It pays off when a team ships often and has more alerts than people to read them. It also needs rich telemetry, the metrics, logs and traces a system emits. It helps less during an active outage, when rolling back the last change usually beats any investigation. It cannot help where the system recorded no evidence at all.

Figure 1. In the illustrative cents example, restoring service took one rollback on day two. Explaining the cause meant reading the diff, the report's code and the bad rows, and took days.
Figure 1. In the illustrative cents example, restoring service took one rollback on day two. Explaining the cause meant reading the diff, the report's code and the bad rows, and took days.
Get Started with Theresa for FREE

What Is AI Production Debugging?

AI production debugging means using models to help responders work out why a live system misbehaves and what to change. Unlike debugging in an editor, the evidence is rarely a failing test on a laptop. It is spread across alerts, logs, traces, deploy history, code, and the data the system writes.

The work splits into two jobs that vendors often sell as one. Restoring service gets users back to a working system, often by rolling back a change. Explaining the cause finds which code or data went wrong, how far the damage spread, and how to stop it from happening again.

Restoring service takes minutes to hours and belongs to the on-call engineer. Explaining the cause can take days and usually needs the team that owns the code and the data. For infrastructure faults, telemetry often explains the cause too. For logic and data faults, the explanation usually sits in code and tables.

We use one illustrative scenario throughout this guide. A deploy changes the order-total field in the order service's JSON events from integer cents to decimal dollars. A downstream reporting job still assumes cents and divides by 100. Nothing errors, latency stays flat, and no alert fires.

Two days later, an analyst in finance notices that daily revenue since the deploy is 100 times too low.

Get Started with Theresa for FREE

Why Is Production Debugging Important?

Most production failures follow a change. Google's SRE book reports that roughly 70% of outages are due to changes in a live system. A team that ships every day therefore debugs in production often, whatever its test coverage.

Production also exposes bugs that tests rarely reach. Researchers from the University of Chicago and Microsoft studied the 112 high-severity Azure incidents caused by software bugs over six months in 2018. In 21% of them, different components or versions held inconsistent assumptions about a data format. Each component can pass its own tests while the two disagree.

The same study also describes silent corruption, a different fault class, where stored data became wrong without any error code. These failures cost more than downtime. A wrong revenue figure can feed a forecast, a price change, or a board report before anyone questions it.

Get Started with Theresa for FREE

How Does AI Production Debugging Work?

Good AI tools automate a method careful engineers already use. Google's troubleshooting chapter describes the work as an application of the hypothetico-deductive method: observe the system, propose a cause, then test it. An AI agent does the same with tool calls. It gathers context, lists competing hypotheses, runs a targeted query for each, and returns a verdict or says the evidence is inconclusive.

Datadog's engineers show why targeting matters. An early version of Bits Investigation made 12 tool calls on a Kafka lag incident, where a message queue fell behind.

One call found the real cause, a spike in commit latency. The summary still named the wrong one, distracted by alarming upstream errors. The newer version tests each hypothesis against the data that bears on it.

Ask for three things in every AI verdict: the evidence behind it, the alternatives it ruled out, and a query you can run yourself. An inconclusive verdict is still useful if it says which evidence is missing.

The table, drawn from vendor documentation rather than accuracy tests, shows which evidence each kind of tool reaches.

ApproachExampleEvidence it readsBlind spot to check
Assistant inside one toolSentry SeerErrors, traces, logs (beta), code in linked repositories, some external toolsBusiness data and tables
Matching past incidentsPagerDuty Probable OriginIncident patterns from the prior six monthsFailures with no precedent
Agent across toolsDatadog Bits Investigation, incident.io InvestigationsTelemetry, deploys, code changes, past incidentsCauses that leave no telemetry
Cloud provider agentAWS DevOps Agent, Azure SRE AgentTelemetry, code and deployment dataSources its integrations do not reach
Causal topologyDynatrace IntelligenceA dependency map of services and hostsFaults that raise no anomaly

In the cents example, none of these starts on its own, because nothing alerted. The evidence sits in a code diff, an event payload, and the rows themselves.

Waggle, AI Theresa's open-source root-cause auditor, works on that evidence, read-only. It reads code, runs SELECT-only queries on BigQuery, and reaches other sources through MCP, a standard way to link AI tools to data, using servers you set up as read-only.

In the cents example, a team could start with a Quick Ask on the order and reporting code: "Which readers of order_total assume cents, and what did the deploy change?" Through the Claude Code plugin, the answer comes with file and line evidence. To size the damage, a Full Scan with BigQuery connected tests claims about the affected rows with queries. A claim that does not hold up becomes an open question, and a hypothesis the evidence contradicts is reported as ruled out.

Get Started with Theresa for FREE

How Does AI Detect Production Issues?

Most AI detection works on telemetry. Baselining learns the normal range of a metric, such as error rate by hour of day, and flags departures from it. Alert correlation groups related alerts into one problem, so a responder sees one incident instead of forty pages. Log clustering reduces millions of log lines to a few templates, so a new error pattern stands out.

Change correlation lines up an incident with recent deploys, configuration pushes, and feature-flag changes. Since most outages follow a change, this is often the fastest route to a rollback.

All four methods need a signal that moves. In the cents example, request rates, latency, and error counts stay flat. The order service returns 200 OK, the status code for success, with a well-formed number in the wrong unit. Data-quality monitors, which watch table freshness, row counts and value ranges, could catch the drop, but few teams point them at business metrics.

That leaves a person in finance as the only detector. The Azure study shows this class of bug is common. In one public e-commerce case study, Theresa's team "traced untrustworthy dashboards to upstream schema drift and rebuilt them on a storage layer that holds." Schema drift means the shape or meaning of data changing upstream without the readers being updated.

Monitor the data against an independent truth. A daily reconciliation of reported revenue against the payment processor's settlement totals would have caught the cents bug on day one. So would an alert on a large day-over-day drop.

Figure 2. In the illustrative cents example, every service metric looked healthy. The evidence was in the event payload, the report's code, and a comparison with payment totals.
Figure 2. In the illustrative cents example, every service metric looked healthy. The evidence was in the event payload, the report's code, and a comparison with payment totals.
Get Started with Theresa for FREE

What Are the Causes of Production Bugs?

A useful habit is to separate the trigger from the root cause. Google's postmortem analysis records both for each outage.

Binary pushes and configuration pushes were its most common triggers from 2010 to 2017, while its top root-cause category was software. Microsoft's November outage had the same shape. A recent network change was the trigger, but the issue itself dated from an earlier upgrade. Rolling back a trigger restores service and leaves the root cause in place.

CauseHow it shows upWhere the evidence lives
Code changeNew errors or wrong behavior right after a deployDeploy history, the diff, the error tracker
Configuration or flag changeA sudden shift with no code deployConfig history, the flag audit log
Data-format or contract mismatchWrong values, often with no error at allProducer and consumer code, schemas, the data
Faulty error handlingA small fault grows into an outageException handlers, logs at the time of the fault
Load or user-behavior shiftSlowness or timeouts under new trafficMetrics, traces, capacity data
Dependency or providerFailures that match a third party's outageProvider status pages, network metrics

In the cents example, the trigger is a binary push, a deploy of new code and Google's most common trigger. The root cause is a contract mismatch: two services never agreed on a unit for the field. Most incidents have several contributing factors. Here they include a field name with no unit, no contract test, and no check on the data.

The distinction changes the fix. Rolling back the order service stops new dollar values, but events written before the rollback stay in dollars. The deploy and rollback times bound the rows to correct.

Get Started with Theresa for FREE

Benefits of AI-Powered Production Debugging

The main benefit is speed to a first hypothesis that comes with the queries behind it. Google's SRE book reports that recorded playbooks make repair roughly three times faster than "winging it." Repair there means restoring service, not finding the cause, and an agent that runs playbook steps helps that same stage.

Vendors publish large reductions in time to resolution, usually without a method a reader can check. The table lists benefits that show up in practice, and what to check before trusting each one.

BenefitWhen it shows upWhat to check
Faster first hypothesisAlerted incidents with rich telemetryIt cites a query you can run again
Runbook steps that run themselvesRepeated incident typesSteps are read-only, or a person approves each change
Postmortem draftsAfter every significant incidentThe draft names contributing factors as well as the trigger
Memory of past incidentsRecurring failuresA match to an old incident is a lead, not a verdict

A postmortem is the written review after an incident. AI drafts save time, but a draft that stops at "a deploy caused it" repeats the trigger. In the cents example, the useful postmortem names the missing unit contract and the missing data check.

Get Started with Theresa for FREE

Challenges of Production Debugging

Independent accuracy is improving but still low. ORCA-bench is a 2026 benchmark co-authored by Cornell Tech, Columbia and the AI SRE vendor Traversal. It gives agents 1,079 tasks on a demo microservice system, with six days of telemetry and full source code. On its realistic medium-difficulty tasks, the best of five frontier agents reached 25.3% accuracy.

The same paper supports the case for code. Removing source-code access lowered accuracy and raised the rate of invented root causes for every model tested. A tool that cannot read code or query business data will not explain the cents bug, however good its model.

Most AI incident tools start from telemetry. Fewer read code, and very few query business data. New Relic says Autopilot "is not currently intended or designed to perform source code review or source code analysis." Datadog's Kafka example shows the other risk: a fluent summary that names the wrong cause.

Access needs the same scrutiny. Sentry's documentation notes that Seer Agent makes organization-wide queries and does not yet have finer-grained access controls. Check what each tool can read, and where that data goes, before you connect it.

Waggle has limits too. It does not detect incidents, ingest alerts, logs or traces, or run code. It works next to Sentry, Datadog or PagerDuty and does not replace them.

It publishes no accuracy benchmark. In a Full Scan, a separate claim audit re-checks each claim, but a model checking a model is a filter, not a test.

Get Started with Theresa for FREE

How Can AI Reduce Mean Time to Resolution (MTTR)?

Here MTTR means mean time to resolution, from the start of an incident until its cause is fixed and verified. Some teams use the same letters for repair, recovery or response, so check which one a vendor means. The average hides four stages that AI affects differently.

Detect is the time until someone knows. Acknowledge is the time until a responder owns it. Mitigate is the time until users stop being hurt. Resolve is the time until the cause is fixed and verified.

Baselines shorten detection, and alert grouping shortens acknowledgement. Pointing at the bad change shortens mitigation, because a rollback decision needs a suspect. Root-cause analysis on code and data shortens resolution, and its follow-up actions lower the chance that the bug returns.

In the cents example, detection took two days and came from a person, not from any AI tool in this article. Mitigation meant rolling back the order service, or pinning the field's unit back to cents, and putting a warning on the dashboard. Resolution meant correcting the rows written between deploy and rollback, finding every other reader, and adding a contract test that checks the field's unit.

Verification there is a query, not a log search. Reported daily revenue should match the payment processor's totals for every day, before, during and after the bad window. Waggle's incremental re-scans call a finding fixed only when its code changed. That checks the code side; the reconciliation query checks the data.

MTTR itself is a weak metric, because a few long outages skew it. DORA replaced MTTR in 2023 with failed deployment recovery time, which counts only failures caused by a change.

Measure the stages instead, as distributions per severity over a long window, since a quarter's incidents rarely show a trend. Track the time from page to the first evidence-backed hypothesis and the share of incidents with a confirmed root cause. Add the repeat-incident rate and the share of AI verdicts later overturned.

Theresa's Enterprise plan helps with the explaining half. It adds a dedicated team of a product lead and two engineers, runs in your own cloud, and starts with a 14-day free trial.

Get Started with Theresa for FREE

Conclusion

AI production debugging works best as two jobs with different evidence. Telemetry AI helps restore service and often explains infrastructure faults. Code and data analysis explains the failures that telemetry cannot see.

Mitigate first, with people and rollbacks. Use AI to generate and test hypotheses, accept only verdicts with evidence you can run again, and measure each stage rather than one average.

More from the blog

All posts →
Enterprise AI

Best Solutions for AI-Driven Debugging

Seven AI debugging tools compared by the bugs they fit, the evidence they show for a root cause, and what they cost to start.

Joena Zhang13 min read
Enterprise AI

AI Agent Debugging: The Ultimate Guide

How to find out why an AI agent failed: traces show what happened in a run, and the agent's code shows why the system allowed it.

Edward Yang16 min read

See what AI can bring to your business.

Contact us