Enterprise AI

AI-Powered Automated Debugging: Levels, Tools & Benefits

How AI debugging moves from chat answers to agents that open pull requests, which tools work at each level, and which checks keep each step safe.

Edward Yang13 min read

AI-powered automated debugging uses language models to find, explain and sometimes fix software bugs. It ranges from a chat answer about one error to an agent that opens a pull request. One well-documented example is Hugging Face's CI agent, Serge, which reproduces real test failures every night and opens verified patches for maintainers to merge. The team reports 29 merged fixes in roughly 80 days.

Automation pays off for bugs that come with a repeatable check, such as a failing test or a scanner alert. It helps little when nobody can reproduce the failure, no test covers the code, or nobody can state the correct behavior. This article follows one real bug through each level: six Windows test failures in Waggle, our open-source code auditor, that hid one path bug.

Figure 1. Six tests failed on Windows and passed on Linux CI. Five came from the test setup and were skipped with a stated reason; the sixth was a real path bug that a skip-everything fix would have hidden.
Figure 1. Six tests failed on Windows and passed on Linux CI. Five came from the test setup and were skipped with a stated reason; the sixth was a real path bug that a skip-everything fix would have hidden.
Get Started with Theresa for FREE

What Is AI-Powered Automated Debugging?

AI-powered automated debugging is the use of AI models to find the cause of a software failure, explain it, and propose or apply a fix. The more of that loop runs without a person, the more automated it is.

The idea is older than chat assistants. Researchers have long studied fault localization, which ranks the code lines most likely to cause a failure, and automated program repair, which generates candidate patches until failing tests pass. The AutoSD paper on LLM-driven scientific debugging describes both as techniques that "aim to help developers by automating the debugging process in part."

Language models state hypotheses in plain words and can call tools such as a debugger or a test runner. Use is already broad: in the Stack Overflow 2026 developer survey, 63.8% chose "Debugging, troubleshooting, or refactoring code" as a way they use AI.

The running example is a real bug in Waggle's open-source repository, which Level 2 below tells in full. Our guide to AI code debugging walks through the same case step by step; here it shows what changes at each level.

Get Started with Theresa for FREE

How AI Is Changing the Software Debugging Process

Debugging has two expensive parts: finding the cause and proving the fix. AI makes the first part cheaper. A model reads a stack trace, a test log and a dozen files in seconds and offers candidate causes.

That makes proof matter more. Hugging Face's engineers put it plainly: "The fixes themselves are often simple. The annoying part is the loop around them." The loop is reproducing the failure, rerunning tests and judging whether a green result means anything.

The productivity evidence is mixed, and each study has limits. METR's 2025 trial found experienced open-source developers took 19% longer on real issues when allowed AI tools. Its February 2026 update says developers are likely "more sped up" now, but calls its own data "only very weak evidence" for the size of the change.

Google's DORA 2025 report linked AI adoption with higher throughput and a "negative relationship with software delivery stability."

Get Started with Theresa for FREE

Three Levels of AI-Assisted Debugging

Neon's engineers proposed three levels of debugging with AI in 2025, from pasting an error to an agent driving a debugger. Their rule of thumb still holds: "If you haven't made progress after two iterations with the model, it's time to move to a more sophisticated debugging approach." We keep three levels and define each by what the AI sees, what it may change and who checks the fix.

LevelWhat the AI seesWhat it may changeWho checks the fixTools today (examples, not a ranking)
1. Prompt-based assistanceThe error, stack trace or snippet you pasteNothing; it suggestsYouChatGPT or Claude chat, GitHub Copilot Chat /fix, JetBrains "Explain with AI"
2. Context-aware analysisThe repository, test output, terminal, sometimes logsFiles in your workspace, with your approvalYou, using the evidence it showsCursor Debug Mode, Claude Code, Sentry Seer
3. Autonomous agentsAlerts, CI failures or issues, plus code, logs and historyA branch and a pull request; some deployAutomated gates and a reviewer, or nobody if auto-publish is onHugging Face's Serge, GitHub agentic autofix, Replit Monitoring Agent, Devin Oncall

Our comparison of AI debugging tools covers product choice in more depth.

Level 1: Prompt-based assistance

At Level 1 you paste an error into a chat and the model explains it. It sees only your excerpt, so its answer is only as good as that excerpt. Paste the failing assertion, the test body and the function under test, and say which platform it fails on. This works for unfamiliar error messages and small self-contained functions.

An illustrative replay: pasting the fork-cutoff failure, "expected 2, found 3," gives the model no test file and no path code. It might guess at duplicate data or give general Windows path advice, with no way to tell which of the six failures matter. Adding "passes on Linux CI, fails on Windows" would at least point it at paths.

Level 2: Context-aware analysis

At Level 2 the AI works inside your repository, reading code and running tests. Cursor's Debug Mode "generates hypotheses, adds log statements, and uses runtime information to pinpoint the exact issue before making a targeted fix," and it asks you to reproduce the bug first. Anthropic's Claude Code best practices suggest prompts such as "write a failing test that reproduces it, then fix it." Sentry's Seer starts from an issue Sentry recorded, runs a root-cause analysis step, and can hand the fix to a coding agent such as Claude Code or GitHub Copilot.

This is what happened with the Windows failures. A developer ran npm test on Windows 11, and six tests failed that passed on the project's Linux-only CI. An AI coding agent read each failing test with its code and proposed a cause, and the developer checked each one.

Five came from the test setup: four tests used shell-script stand-ins that Windows cannot run, and one created symlinks, which need Developer Mode or admin rights. The sixth, the fork-cutoff test in src/git.test.ts, expected 2 people across the org and found 3.

The cause was a path split on /. The cutoff was stored under the short repo name, which never matched the full Windows path, so the count came out one too high. Version 0.1.4 switched to basename(), and the same test, run on Windows, was the check.

Level 3: Autonomous agents

At Level 3 an agent starts from a trigger, such as a CI failure or an alert, and works with nobody at the keyboard. It usually pushes a branch and opens a pull request. GitHub's agentic autofix proposes a fix and "reruns CodeQL to confirm the fix closes the alert" before opening a draft pull request. The October 2026 GA release covers code scanning alerts for GitHub Code Security and Advanced Security customers on Enterprise Cloud.

Replit's Monitoring Agent goes further: with auto-publish on, it can deploy a validated fix to production. Devin Oncall, generally available since October 2026, investigates alerts and bug reports. Its launch post notes that agents "still stop at the first plausible root cause," so it adds a second agent to review the first. Our guide to AI production debugging covers alert-driven investigation in more depth.

An illustrative replay: an unattended agent told to make the Windows run green could skip all six tests, and the path bug would ship. Level 3 needs gates that catch this.

Get Started with Theresa for FREE

How Autonomous AI Debugging Agents Work

Most autonomous debugging agents follow the same loop. Hugging Face's account of Serge shows where each gate sits.

1. Pick a real failure and reproduce it

Serge filters the night's CI failures, then reruns each one on a GPU. In the team's words, "If the failure doesn't reproduce reliably, the task stops there."

2. Investigate and patch inside a sandbox

The agent runs in a pod with "no GitHub credentials and no cluster API token." It can only push serge/ branches and open pull requests. Its patch must also pass the project's lint and type checks.

3. Verify before and after

The failing test runs five times on the unpatched code and five times on the patched code. Timeouts and similar errors count as infrastructure failures, with no verdict on the patch. A patch that only changes expected values gets "extra suspicion," because editing an assertion can turn a test green without fixing anything.

Figure 2. In two weeks, Serge's 86 agent sessions led to 24 verified pull-request events; the rest stopped because no safe patch was found, verification failed, or the run errored or timed out.
Figure 2. In two weeks, Serge's 86 agent sessions led to 24 verified pull-request events; the rest stopped because no safe patch was found, verification failed, or the run errored or timed out.

4. Open a pull request for a person to merge

Nothing reaches main until a maintainer reviews and merges it. In one published case, a PvtV2 test expected an output shape copied from the older PVT model, so the test itself was wrong. Serge corrected the expectation, and maintainers merged it unchanged. Step 3 treats exactly this kind of patch with suspicion, which is why a person decides.

Other vendors place the gate differently. Databricks says its ZeroOps pipeline agent "will never change your code or data without your direct approval."

Waggle, our open-source auditor, also runs unattended, but read-only: it checks each suspected problem against the code and connected data, and reports a claim it cannot support as an open question. It does not run code or reproduce bugs, and it did not find the Windows bug. Its claim audit is a model checking a model, a filter rather than a test, so a person or a test still confirms each finding. Its Claude Code plugin adds a fix command that works on a branch, one commit per finding, asking before each step.

Get Started with Theresa for FREE

AI Debugging vs Traditional Debugging

Both follow the same method. Google's guide to effective troubleshooting calls it the hypothetico-deductive method: observe the system, form a hypothesis about the cause, test it, and repeat. The difference lies in who does each step and what counts as proof.

QuestionTraditional debuggingAI-assisted debuggingWhere it breaks
Who forms the hypothesis?The developer, from experience and the codeThe model, from the code, logs and errors it can seeA plausible story fits the visible evidence and misses the rest
Where does evidence come from?Breakpoints, logs, git bisect, rerunsThe same tools run by the agent, plus its reading of the codeThe agent misreads an output or skips a run
What proves the fix?A reproduction that now passes, then reviewThe same when the agent runs it; only a summary when it does notA report says "fixed" with no run behind it
What does it cost?Developer hours spent searchingModel usage plus developer hours spent reviewingReview time grows with large or unclear patches

Deterministic tools keep their place. git bisect uses binary search to find the commit that introduced a bug. An agent can run it, and that run is stronger evidence than the agent's guess.

In the Windows case, the agent supplied the hypotheses and the developer supplied the proof. The check against the code, not the agent's explanation, decided what to skip and what to fix.

Get Started with Theresa for FREE

Benefits of AI-Powered Automated Debugging

The first is a candidate cause for every failure, ready to check. In the Windows case, the agent proposed one for each of the six, and the developer checked them.

The second is repetitive, verifiable work done unattended. Hugging Face estimates Serge saved one to two hours of developer attention per fix, around 43 hours across 29 fixes. It reports roughly $14 per distinct pull request and $43 per merged one. These are the team's figures for one repository.

The third is scoped fixes for known alert types. A scanner alert carries its own check: rerun the scanner and see whether the alert closes. That makes alerts like CodeQL's a sensible first target for Level 3.

Explanations also help people judge patches. In the AutoSD study, 20 participants judged patches more accurately with explanations for five of the six real-world bugs studied.

Get Started with Theresa for FREE

Challenges of Implementing AI Debugging

Tests that turn green for the wrong reason

Hugging Face says it does not "yet have a perfect automatic way to distinguish a legitimate expectation update from reward hacking." Reward hacking here means a patch that satisfies the check without fixing the bug. The PatchDiff study found that running all developer tests missed 82.7% of the suspicious patches it examined, meaning patches that behave differently from the developer's own fix.

The Windows case shows the temptation. Skipping all six tests would have turned the suite green and hidden the bug. We did skip five, each with the reason written in the test, and kept and fixed the sixth. Waggle still has no Windows CI job.

Inconclusive results stored as passes

A second bug, in Waggle's own engine, was quieter. When one part of a scan hit its budget cap, it stopped with a question unanswered. The run did not record the skip, and the next re-scan treated the question as already checked. An end-to-end test found it, not Waggle.

Hugging Face applies the same rule to timeouts: an inconclusive run gets no verdict and never counts as a pass.

Benchmarks that say less than they seem

OpenAI stopped reporting SWE-bench Verified scores in February 2026. Its audit of 138 problems that OpenAI o3 did not consistently solve across 64 runs found that at least 59.4% have flawed test cases that reject correct solutions. A benchmark score shows patches pass hidden tests, which is weaker evidence than a reviewed fix in your codebase.

Permissions and company knowledge

Claude Code's best-practices page separates checks the model decides from checks the tooling enforces. For "who has the last word", the enforced option is "branch protection, a person who approves the merge". For unattended agents, put the gate there and not in the prompt.

Our AI agent debugging guide covers what to log when an agent itself misbehaves. Databricks reports that over half of the pipeline issues it saw needed company-specific knowledge to fix. Theresa's forward-deployed engineering team works on that side of the problem, for example tracing untrustworthy dashboards to upstream schema drift for a client.

Get Started with Theresa for FREE

Conclusion

AI-powered automated debugging is a ladder. Each level lets the AI see more of your system and change more of it, from a pasted error to a branch and a pull request. The method underneath stays the same: reproduce the failure, test a hypothesis, and prove the fix.

Move a class of bugs up a level only when you have a check that does not depend on the model's own summary. That check can be a reliable reproduction, before-and-after runs, a scanner that closes the alert, or a person who merges. Where no such check exists, keep the AI at the level where someone still reads every change.

More from the blog

All posts →
Enterprise AI

7 Best Enterprise AI Agent Platforms in 2026

Seven enterprise AI agent platforms ranked by who does the production work, where agents run, governance and how each one bills you.

Joena Zhang16 min read

See what AI can bring to your business.

Contact us