p2p – How does block synchronization currently work in Bitcoin Core?

Since the introduction of header-first synchronization in Bitcoin Core 0.10, blockchain data structures can be viewed in three components:

• that much block header tree: A tree data structure with the genesis block as the root containing all the headers for the known forks of the chain. This tree structure is always connected. This means that you cannot add a header to a tree unless its parent is already in the tree.
• that much block data: For each item in the block header tree, there may or may not be corresponding block data. Blocks can be received (and requested) out of order as long as the corresponding header exists in the header tree. Because there may be gaps, there is no case in which a block can be saved unless its parent block is also saved. Of course, a block can only become an active chain tip if it is fully verified. This requires checking all previous blocks as well. Since the block file cleanup feature was introduced in Bitcoin Core 0.11, blocks can also be deleted again after they have been verified. In this case, that header will still remain in the tree.
• that much active chain tip (reference to an entry in the block header tree) UTXO set (or chainstate), which represents a fully verified block that is currently considered active (introduced in Bitcoin 0.8).

Now, to illustrate how synchronization works, there are three separate processes that each manage these data structures.

1. Header synchronization: The process of requesting, receiving, and storing headers that our peers have but that we do not have.
2. block synchronization: The process of requesting, receiving, and storing an entire block of data for the headers we already have. Our peers have it, but we don’t.
3. Enable blocking: Full verification of the blocks we have and corresponding changes to the active chain tip.

In reality, it all happens in parallel, but it’s easier to think of it as a sequence.

1. Header synchronization

Request headers. The first step in synchronization is learning about the headers your peers have. This is primarily done through transfers. getheaders When we connect or accept a connection, we respond to each peer (by sending a locator indicating the headers we already have). headers Messages containing up to 2000 new headers. When a total of 2000 headers are received, we send another header. getheaders To ask for more. This process continues until all headers the peer has to provide are obtained. When it falls far behind, it only requests one peer (the header sync peer) to minimize duplicate bandwidth usage, but when it gets close to the current timestamp, it requests all peers.

Connectionless header. When a header arrives that is not connected to an existing header tree (i.e., the parent of the received block is not yet in the tree) getheaders First, learn about all the headers in between. If the parent is not yet known, the header will not be processed in any case. This does not mean that it has been rejected (i.e., not permanently marked as invalid). They are simply ignored (as if they were not received at all). If all goes well, the missing parent header will arrive and be received again, at which point it will be processed normally.

Direct header presentation. Since the introduction of the direct header notification mechanism (negotiated using BIP130) sendheaders Messages) was added in Bitcoin Core 0.12.0, allowing our peers to send us. headers A message directly announcing a new active chain tip (see below). If this mechanism is not used (because the peer believes we are far behind, or because the peer does not support BIP130), it may send us a message. inv A message containing only the block hash. This too we getheaders In response to any headers we missed, including the hash just passed invEd to us.

Header validation. When the header arrives, it is validated as much as possible. These include syntax correctness, proof-of-work, difficulty scaling, rules for timestamps (greater than the median of the last 11 blocks and within the next 2 hours), and rules for version numbers (BIP34, BIP66, and BIP65). Please enter your requirements in the version number). If a header fails any of these, that header and all headers derived from it are ignored.

Header spam prevention. The block header is very small (81 bytes per block) header messages), must be protected from storing large numbers of low-difficulty items that an attacker could generate. Proof-of-work inherently makes production costs very high. good Although headers can be generated in bulk, an attacker can construct a chain of headers that branch somewhere early in history (e.g. right after the generation block) when the difficulty is low, without having to do a significant amount of work. This can ultimately lead to memory issues for the node if the node allows these headers into its block header tree. Historically, checkpoints were used to prevent this, but starting with Bitcoin Core 24.0, a more comprehensive mechanism called header presynchronization was used. This means downloading the header twice. Firstly, if the header undergoes a significant amount of work to make sure it forms a chain (but not yet added to the block header tree), and secondly, if it is downloaded again. If it matches what was sent in the first step, it is added to the block header tree. You can learn more about this mechanism in this answer.

2. Block synchronization

Blocks allowed. Once you know about the header, you can download the block. Typically a block is Only Requests and grants whether the tip is part of a header chain that has at least as many accumulated operations as the currently active chain tip. Blocks that are not in the header tree are never accepted (they become orphan blocks), and blocks from forks that do not have a competitive cumulative workload in the header chain are also not accepted.

Request mechanism. Blocks are requested using: getdata Message to which the peer responds block message(getdatas is also used for transaction fetching, but we ignore it here.) To determine which peer these requests are sent to, we use inv and headers From the notices our colleagues send us, we can find out what the last block header is in common with them. We generally assume that the peer that tells us a block may actually provide that block data (at least if they set the block). NODE_NETWORK service flag or NODE_NETWORK_LIMITED (for sufficiently recent blocks).

Request reservation. real getdata Among all block header tree entries that are not permanently marked as invalid, requests for missing blocks are sent along the path from the genesis block to the header tree tip with the highest cumulative operation (which can be distinguished from the active chain tip). It’s possible. Requests are distributed to all peers with each block in a round-robin fashion, with a limit of 16 requested but not yet received blocks per peer. We never request blocks more than 1024 blocks ahead of the currently active chain tip, to limit how out-of-order blocks can be received (this is important for block file cleanup). As long as a block request has not been processed, the same block will not be requested again from another peer to avoid duplicating bandwidth usage. If a peer takes too long to respond to a block request, the connection is lost and the request is sent elsewhere.

Unrequested block. You can receive blocks without requesting them. Bitcoin Core will not send unsolicited blocks, but some other P2P clients may. When this happens, we treat the headers inside the block as if they were received via the block. headers After sending the message approx. If the If you want to make a request after seeing the header, you can also process the block directly. Otherwise it is ignored (i.e. it acts as if only the header was received).

Compact Block Notice. Bitcoin Core 0.13.0 added support for BIP152 compact blocks. In compact blocks, blocks are transmitted without full transaction data, but instead contain short (48-bit) salted hashes of individual transactions. Upon receiving a compressed block, the receiver attempts to reconstruct the entire block using the transactions in memory and then requests missing transactions. Details are beyond the scope here, but the compact block includes a feature called high-bandwidth mode. When activated, a limited number of peers announce Block us directly using: cmpctblock Message, skip round trip using headers/inv and getdata Saves waiting time. In this case, the headers contained in the compact block are first reprocessed, and then the compact block is processed if the block appears to be very close to the active chain tip (up to 2 blocks ahead).

Block verification. When a block is processed, all applicable consensus rules are performed, including syntax validity, transaction Merkle root recalculation, transaction finality, BIP34 (“Height of Coinbase”), SegWit commitment, and maximum block weight. If these checks fail, the block is either ignored or, depending on the type of error, marked as permanently invalid. If the check is successful, the block data is saved to disk. Scripts, double spending, inflation, etc. cannot be verified yet.

getblocks Based synchronization. Before header-first synchronization was introduced, other mechanisms were used. Instead of getheaders Similar to this to start syncing. getblocks The message has been sent, and the peer responds as follows: inv Contains up to 500 block hashes, getdataand go further getblocks Continue. Bitcoin Core no longer uses this mechanism as of version 0.10, but it is still supported for peers who want to synchronize in this way (i.e. Bitcoin Core responds with: getblocks No message is sent.)

3. Activate block

Finally, once we have a block, we can determine what the active chain tip is and fully verify the associated block.

Select active chain tip. The rule is that at every point we aim to make the active chain tip:

1. Of all the blocks in the block header tree that have full data for all ancestors and that have full data and are not (yet) permanently marked as invalid…
2. Only items with the highest cumulative effort are considered.
3. Among them, select the one that received the entire data first.

Move towards selection. If at any point the result of this selection is not equal to the currently active chain tip, progression towards the selection begins. First, there are blocks that are part of the active chain but not in the selected chain. disconnected In reverse order, the blocks of selections that are not in the active chain are: connected Forward in order. If a disconnection is involved during the activation process, it is called a disconnection. reorganization.

Full block verification. The connection involves performing full validation of all consensus rules, including script validation, subsidy verification, and double-spending verification, which require access to the UTXO set. Since we keep the UTXO set only at the active chain tip, concatenation involves removing all the inputs of the block from the UTXO set and adding the outputs. Unlinking does the opposite, removing all outputs added to the unlinked block and adding back all used inputs. If validation fails here, the block will be permanently marked as invalid, which may cause the above selection logic to change its idea of ​​what the active chain tip is.

Announcing a new chain tip. Whenever the activation process stabilizes, peers will be notified of new active chain tips. When only a few blocks are connected and peers support BIP130 headers A message is sent with the headers of all connected blocks. Otherwise we revert to sending. inv Here’s a new tip. We will notify you when a peer has selected us as a high-bandwidth compact block peer. before Full verification is completed under circumstances permitted by BIP152 (again as a means of reducing latency).