Category is product. Runs an AI agent against Ethereum’s protocol code.

A note from the Ethereum Foundation’s Protocol Security team running a tuned AI agent against real protocol code, including how the work is structured, what to investigate, and what client teams and security researchers can gain from it. This post stands on its own. Future posts will cover individual customers in more detail.

What we ran into and what surprised us.

The Ethereum Foundation’s Protocol Security team has been running AI agents tuned to the types of systems the network relies on, such as system software, cryptographic code, and correct contracts. The agent found a real bug. One of them has now been revealed. A remotely triggerable panic in libp2p’s gossipsub was implemented and modified in the peer-to-peer layer Ethereum consensus client and has been released as CVE-2026-34219 with credit to the team.

It wasn’t a surprise to the agent who discovered the bug. What was surprising was how little effort went into finding the bug and how much effort went into figuring out the real bug from a bug that looked real.

This post is for client teams and security researchers who want to do the same. We’ll cover how to structure your agents, what criteria a candidate must pass before being considered for results, and habits to keep your results reliable.

Other teams are converging in the same way. Anthropic’s Frontier Red Team has built agents that write property-based tests and discover real-world bugs across the Python ecosystem. Cloudflare has implemented a pioneering model with security research tools on its own systems. Everyone gets caught in the same loop. Specifies competent models in your code base, retrieves them, and classifies what is returned. So the real question is how to do this without falling into confidence noise.

One caveat: Agent-centric audit tools move quickly, and certain settings may not be out of date within a few weeks. So this post is intentionally more about ongoing methods rather than tools. Disclosure is a topic of its own and would probably be its own post.

Agents that point to your codebase are search tools, very similar to fuzzers. The difference is what comes back. Fuzzer delivers crashes and stack traces. Agents provide much more, including writing proof-of-concepts that can be run against real code (call chains, impact assertions, suggested severities) and the artifacts that support them.

All of this makes the results easy to read and trust, and best of all, a proof of concept in action. So don’t count the number of candidates your agent generates. Count how many people are real.

How the work is structured

We run multiple agents in parallel for one target. Versioning coordinates through the repository itself, without a central process to share state and forward work. Agents log claims where others can see them, take action, and commit them.

We got this approach from Anthropic’s article about building a C compiler with a set of agents coordinated in the same way. There is no central coordinator to build or maintain, and there is less that can go wrong.

Roles are created by discovered tasks.

• Recon turns your attack surface into concrete, testable hypotheses. It’s not “audit the decoder”, it’s “this field is trustworthy after this point. Here are the properties that should be kept, how they might be broken, and the evidence to resolve them.”
• Hunting takes a hypothesis, traces its code path, and attempts to build a replicator.
• Gap filling looks at what’s accepted and what’s rejected, creates the next batch of hypotheses, and tracks coverage to ensure agents aren’t covering the same ground over and over again.
• Verification is determined after independently double-checking each candidate and removing duplicates.

We didn’t invent this pipeline. Cloudflare describes the same steps, reconnaissance, parallel search, independent verification, deduplication, and reporting, and our history has helped shape us.

Here’s what candidates look like before they are considered results.

target:      component and entry point an attacker can actually reach
invariant:   the property that must hold
mechanism:   the specific way it might be made to break
success:     observable proof: a panic, a stall, an accepted-invalid input
reproducer:  a self-contained artifact that runs against the real code
dedup:       a key, so two agents don't chase the same thing

Schemas are there for a reason. This forces specific, testable claims and a clear definition of done. An agent who has to record observable evidence cannot decide, “This looks dangerous.”

Reproducible or not occurring

One rule is more important than another. Candidates won’t be discovered until you have a self-contained artifact that reproduces the failure against real code and runs for someone who didn’t write it.

The player doesn’t read the text and doesn’t care how confident the model sounds. Either it runs or it doesn’t run.

Most of its value lies in the false positives it catches. Three of them keep popping up, each an agent getting a pass for the wrong reasons.

• This panic only occurs in debug builds. Compiling and running the software the way it is actually delivered increases its value. Nothing crashes. It looks like a crash, but it’s not a crash.
• It is a reproducer that builds by hand internal values ​​that the real input cannot produce, because all paths controlled by the attacker are rejected earlier. The bug is “reproduced” only for functions that don’t call anything reachable that way.
• A proof that passes a formal verification task but doesn’t mean what you want it to mean. This statement is either roughly true regardless of what your code does, or it’s weaker than the property you’re trying to capture. The verifier is satisfied, but the theorem does not actually constrain the behavior of interest.

This is nothing new. It’s like a test you pass because it doesn’t actually check anything. What’s new is volume. Agents write useless versions as quickly and confidently as real versions. Therefore, verification must occur automatically. You can’t expect agents to catch it themselves.

Signal to noise is what most of the work is about.

Most candidates are incorrect, redundant, or out of scope. It doesn’t matter how. That’s how it works. The goal is to quickly reject what is false and support what is real with evidence that is hard to dispute.

All surviving candidates receive two independent checks. Would a real attacker actually have access to a legitimate configuration? And how much does it cost an attacker to succeed compared to what it would cost the network if it were operational? Bugs that a single peer can cause are very different from bugs that require special access or huge amounts of resources.

Check everything against a running list of what is already known, what has been modified, and what has been rejected. Otherwise, agents will keep rediscovering and reporting the same issues that were closed over and over again.

Passing rates vary greatly from subject to subject, and these variations are useful in themselves. If you run this against mature, thoroughly audited code, you’ll be left with almost nothing, which is still worth knowing about. “We searched hard and found nothing” is the actual result. Run against less explored or formally validated code. Here, the machine-checking proof deals with the model and the deployed bytecode is assumed to match the model only and passes on more.

We’re not the only ones who find classification difficult. Cloudflare’s main takeaway is that narrow scope is better than broad search. Anthropic’s asset-based testing agents generated thousands of candidate reports and then used rankings and expert reviews to work their way down to the top tier, which they did about 86% of the time. Generation was the easy part. I won’t reveal our own phone numbers here. Being tied to a specific goal speaks more about the goal than the method.

Areas that counselors are good at and areas that cause misunderstandings

There’s hype both ways, so here’s a quick list of what agents do well and what they misunderstand.

Splitting is not stable from one task to the next. Stanislav Fort, who tests different models for real-world vulnerabilities, calls this a jagged boundary, where a model that recovers the entire exploit chain in one codebase may fail to trace the underlying data flow in another. You can’t assume that one good result will hold for the next. This is another reason why all candidates undergo self-verification.

The last line is the important line. A single agent session is suitable for one-time inference, but not for bugs that occur over a series of steps where each step is valid and only out of order. In that case, the agent is not a discovery tool. Its job is to suggest which sequences are worth running through a stateful test harness. It works fine when used that way. Using it as a replacement for a harness means you’ll miss the most expensive bugs – bugs that only appear in sequences.

keep it honest

A few habits do most of the work of making agent results reliable, and none of them are complicated.

• The provenance of every artifact: what produced it, in what context, and what revision it is based on. The results should be something you can run back on months later.
• Crucial determinism: One environment, one way to build and run, so “playing” means the same on all machines, not just the one on which it is found.
• Norms, not scripts: Tell agents what’s important: invariants and standards for actual outcomes instead of numbered procedures. An over-scripted agent will break in the same way as an over-specified test, continuing to follow steps even after they don’t make sense. A study of repository context files showed the same results. The additional requirements reduced the success rate of the operation and increased the cost by more than 20%, and the authors recommend keeping the context as a minimum requirement.
• People make the final decision. The agent makes a suggestion. They don’t decide what’s real, what overlaps with known issues, or who and when to disclose.

The bottleneck has moved.

AI has not replaced security researchers. The work has been moved. Time once spent creating and tracking hypotheses is now spent determining them at scale, including building oracles, running classifications, maintaining lists of known issues, and processing them to the public.

The bottleneck has not gone away. We moved from finding bugs to trusting the results. This is where it’s better because human judgment is where it really matters. But bottlenecks still occur, and if you ignore them, you’ll end up getting a bad delivery that says “it’s ok.”

The practices that make this possible are not new. Reproducible failures, real oracles, and careful classification are the same practices that have transformed fuzzing from a research topic to standard practice over the past 15 years. The tool is new. The practice is not like that.

The question of how quickly tools change is open. The cautious and once skeptical Nicholas Carlini argues that the exponential case is worth taking seriously, even if he maintains a wide margin of error for it. If the generation side goes up that fast, the judgment side has to go up along with it, otherwise the gap between what is produced and what is actually tested will only widen.

For the systems that Ethereum relies on, this is an important part. Using agents we can cover much more ground than we can do by hand. In return, they require more careful judgment of claims that sound much more confident. As long as you remember that judgments have real consequences, it’s a worthwhile transaction.