Sepolia incident | Ethereum Foundation Blog

This blog post exposes threats to the Ethereum network that existed from the Merge to the Dencun hard fork.

background

Prior to the merge, various message size limits were set for RPC communications to protect clients from denial of service (DOS) attacks. These restrictions, which apply to messages received via HTTP endpoints, are passed on to the Engine API, which plays an important role in connecting execution and consensus layer clients during block creation. The engine API’s participation in block generation allows block generation to exceed the RPC size limit for some clients but remain within an acceptable range for others.

If an attacker generates a message that exceeds a client’s size limit at the lowest settings while respecting gas limit requirements, and then waits for a block to be generated, a situation may arise where some clients consider that block. We consider it valid, but others reject it and issue an HTTP error code: “413: Content too large”.

effect

An attacker who can manipulate these messages can force the majority of nodes (=geth) to reject blocks that the minority accepts. These blocks will fork and the proposer will miss out on the reward.

Initially, we thought it would only be possible to create these blocks using a modified version of the builder or client. Geth has a default 128KB limit for transactions. This means that large transactions like the one being discussed will not be included in any geth node’s transaction pool. However, it was still possible to trigger the limit by having a client with a higher limit propose a block and have the CL request validation of this proposed larger block.

We proposed a solution that temporarily lowered the RPC limit for all clients to the lowest value (5 MB). This invalidates the block and causes most nodes to reject the block, thus very limiting the disruption an attacker can cause to the network.

However, on February 7, we discovered that it is possible to create a block that hits the 5MB limit with multiple transactions that are below the 128KB limit and do not exceed 30 million gas.

This is a bigger problem because attackers have realized that they can generate many high-value transactions and send them onto the network. Because he pays more than everyone else in the mempool, all nodes (even geth nodes) include attack transactions in their blocks, creating blocks that are not accepted by the majority of the network, resulting in many forks (all are considered ). valid by a small number of nodes) the chain is continuously re-formed.

By late February 7th, we had all come to the conclusion that raising the RPC limit was a safer alternative.

timeline

• 2024-02-06 13:00: Toni (EF), Pari (EF) and Justin (Besu) are attempting to submit a specifically refined transaction to the network. Transactions contribute up to 2.7 MB blocks when compressed quickly.
• 2024-02-06 13:25: The transaction should be valid, but Pari receives an error from the local Geth node.
• 2024-02-06 15:14: Justin put the transaction into a block and submitted it through the Besu client.
• 2024-02-06 20:46: Sam (EF) warns Pari (special thanks) Mystic Ryujin X), Toni and Alex talk about a particular Sepolia node they are having trouble with.
• 2024-02-06 21:05: Team checks again with Marius on Geth and checks for bugs.
• 2024-02-06 21:10: The gang gets together to debug.
• 2024-02-07 23:40: Decided to limit all clients to RPC request limit to 5MB.
• 2024-02-07 6:40: There may be a bigger problem, and we have discovered that the attack can be executed with transactions less than 128KB in size.
• 2024-02-07 10:00: Decided to increase RPC request limit for all clients.
• 2024-02-07 21:00: Fix merged into geth.
• 2024-02-09: Geth is released.

While Geth is the only client affected by this bug, other clients have also updated their defaults to be safe from this attack even with increased gas limits. The client team has indicated that there will be safe rpc restrictions in the next update.

Geth: v1.13.12

Nethermind: v1.25.4

believe: 24.1.2

Erigon: v2.58.0

Res: v0.1.0-alpha.18