Setting Block_fetch_horizon For NEAR & MPC Indexes

by Alex Johnson 51 views

In this article, we'll dive deep into the significance of the block_fetch_horizon setting within the context of NEAR and MPC (Multi-Party Computation) indexes. We'll explore a real-world scenario where a panic in an MPC node was potentially linked to state synchronization issues, and how adjusting block_fetch_horizon can be a crucial step in preventing such problems. Whether you're a seasoned blockchain developer or just starting your journey, this guide will provide you with the insights you need to effectively manage your NEAR and MPC node configurations.

Background: The MPC Node Panic and State Synchronization

Recently, a panic was observed in an MPC node, which raised concerns about the stability and reliability of the system. The initial investigation pointed towards a potential root cause related to the node's synchronization state. In blockchain networks, nodes need to stay synchronized with the latest state of the chain to accurately process transactions and maintain consensus. When a node falls out of sync, it can lead to various issues, including data inconsistencies and, in severe cases, panics.

Razvan Barbascu, a key contributor, suggested that the node's downtime exceeding one epoch might have triggered a state synchronization process. In such cases, the node attempts to resynchronize by potentially wiping existing data and re-downloading the blockchain state. This process, while necessary for recovery, can be disruptive and time-consuming. To prevent unwanted state synchronization, a crucial parameter comes into play: the block_fetch_horizon.

Decoding the Error Message

The error logs provided offer valuable clues about the nature of the problem. The following excerpts highlight the key issues:

2025-11-24T08:11:33.048042854Z 2025-11-24T08:11:33.046158Z ERROR indexer: Failed to build StreamerMessage. Skipping. block_height=890 streamer_message=Err(String("Block either has never been observed on the node or has been garbage collected: BLOCK: 7F2y36VG6hKAJeiUkNhXSDXZGzRsn4VU87e1Yu7B8CH3"))
2025-11-24T08:11:33.069516286Z
2025-11-24T08:11:33.069584820Z thread 'tokio-runtime-worker' panicked at /usr/local/cargo/git/checkouts/nearcore-86558fdb18093f53/49aeec5/chain/indexer/src/streamer/mod.rs:200:42:
2025-11-24T08:11:33.069593409Z `receipt` must be present at this moment

The first error message indicates that the node failed to build a StreamerMessage for block height 890. This suggests that the node was unable to retrieve the block data, possibly because it was either never observed or had been garbage collected. The subsequent panic message, "receipt must be present at this moment," further implies a critical data access failure during the indexing process.

These errors point towards a scenario where the node's view of the blockchain state is inconsistent or incomplete, leading to the observed panic. This is where the block_fetch_horizon setting becomes essential.

What is block_fetch_horizon?

The block_fetch_horizon is a configuration parameter that dictates how far back in the blockchain history a node is willing to fetch blocks. It essentially defines the window of blocks that the node actively keeps track of. This setting is crucial for several reasons:

  • Preventing Unnecessary State Syncs: By setting a sufficiently large block_fetch_horizon, you can prevent the node from triggering a full state sync if it experiences temporary downtime or network issues. If the node can still fetch blocks within the defined horizon, it can catch up without resorting to a complete data wipe and resynchronization.
  • Optimizing Resource Usage: A smaller block_fetch_horizon can reduce the amount of storage space and network bandwidth required by the node. However, this comes at the risk of triggering more frequent state syncs if the node falls behind.
  • Ensuring Data Availability: For indexers and other applications that rely on historical blockchain data, the block_fetch_horizon determines the range of blocks that can be reliably accessed. A larger horizon ensures that more historical data is readily available.

The Recommendation: Setting a Large block_fetch_horizon

To mitigate the risk of state synchronization issues, Razvan Barbascu recommended setting the block_fetch_horizon to a very large value, specifically suggesting 99mn (99 million). This large horizon effectively tells the node to maintain a vast window of blockchain history, reducing the likelihood of triggering a state sync due to minor disruptions.

While a larger block_fetch_horizon generally improves stability, it's essential to consider the trade-offs. A larger horizon requires more storage space and can potentially increase network bandwidth usage. Therefore, the optimal value depends on the specific use case and the available resources.

How to Set block_fetch_horizon in NEAR and MPC Nodes

The process of setting the block_fetch_horizon typically involves modifying the node's configuration file. The exact file name and location may vary depending on the specific node implementation and deployment environment. However, the general steps are as follows:

  1. Locate the Configuration File: Identify the configuration file for your NEAR or MPC node. Common names include config.json, node.toml, or similar.
  2. Open the Configuration File: Use a text editor to open the configuration file.
  3. Find the block_fetch_horizon Setting: Search for the block_fetch_horizon parameter within the file. It might be located in a specific section related to syncing or data storage.
  4. Modify the Value: Change the value of block_fetch_horizon to your desired value. In this case, we recommend setting it to 99mn.
  5. Save the Configuration File: Save the changes you made to the configuration file.
  6. Restart the Node: Restart your NEAR or MPC node for the changes to take effect. This is crucial for the new configuration to be loaded and applied.

Example Configuration Snippet (Illustrative)

While the exact syntax may vary, here's an illustrative example of how the block_fetch_horizon setting might appear in a configuration file:

{
  "sync": {
    "block_fetch_horizon": "99mn"
  }
}

Important: Always consult the official documentation for your specific NEAR or MPC node implementation for the most accurate and up-to-date instructions on configuring block_fetch_horizon.

User Story: Running MPC Nodes in Localnet

The user story presented highlights a common desire among developers: to run MPC nodes in a localnet environment. A localnet is a private, isolated blockchain network used for development and testing purposes. Running MPC nodes in a localnet allows developers to experiment with MPC functionalities, test integrations, and debug issues without affecting the main network.

Setting an appropriate block_fetch_horizon is particularly important in localnet environments. Localnets often experience periods of inactivity or rapid block generation, which can lead to synchronization challenges for nodes. By setting a large block_fetch_horizon, developers can ensure that their MPC nodes remain synchronized even during these fluctuations.

Acceptance Criteria: Verifying the Configuration

The acceptance criteria mentioned, "update block_fetch_horizon to 10000 and hope it works :-)," underscores the importance of verifying that the configuration change has been successfully applied and is functioning as expected. While simply setting the value is a necessary step, it's crucial to confirm that the node is indeed honoring the new block_fetch_horizon.

Here are some ways to verify the configuration:

  • Check Node Logs: Monitor the node's logs for messages related to block_fetch_horizon or syncing behavior. The logs might indicate whether the node is using the configured horizon and whether it's experiencing any syncing issues.
  • Monitor Node Metrics: If your node exposes metrics, look for metrics related to block fetching or syncing. These metrics can provide insights into the node's syncing behavior and whether it's staying within the configured horizon.
  • Simulate Downtime: Simulate a brief period of downtime for your node and observe whether it can catch up without triggering a full state sync. This is a practical way to test the effectiveness of the block_fetch_horizon setting.

Additional Resources and Notes

The provided Zulip chat discussion link (#community-support > Help with state sync / connection setup for MPC node inside @ 💬) offers valuable context and insights into the original issue. It's highly recommended to review this discussion for a deeper understanding of the problem and the proposed solution.

Key Takeaway: The block_fetch_horizon setting is a critical parameter for managing state synchronization in NEAR and MPC nodes. Setting a sufficiently large horizon, such as 99mn, can significantly improve node stability and prevent unnecessary state syncs. However, it's essential to consider the trade-offs in terms of storage space and network bandwidth. Always consult the official documentation for your specific node implementation for the most accurate configuration instructions.

Conclusion

In conclusion, understanding and correctly configuring the block_fetch_horizon is crucial for maintaining the stability and reliability of NEAR and MPC nodes. By setting an appropriate horizon, you can prevent unwanted state synchronizations, optimize resource usage, and ensure data availability. We hope this article has provided you with the knowledge and insights you need to effectively manage this important parameter. Remember to always consult the official documentation and community resources for the most up-to-date information and best practices.

For further information on NEAR protocol and its configuration parameters, please visit the official NEAR Documentation.