Both Arabica and Mocha halted across multiple occasions starting on October 30th and lasting between 10 minutes to several hours until the patch on Nov 4th (v6.2.2-mocha). The cause was a very subtle timing bug inherited in cometbft in which the consensus timer could occasionally trigger earlier than it was set to. In v6, when we introduced a minimum time before sending precommit votes in order to keep block time constant, this timing bug would cause the precommit to trigger earlier than expected. When this happened, the function would determine it was too soon and thus ignore the call, causing that particular node to freeze until it saw 2/3+ precommit votes from the rest of the network. To halt the network would require the timing bug to affect 1/3+ of voting power at the same height. While the bug was present in v6 as early as August 27th, it wasn't triggered until October 30th. It has been patched in v6.2.2-mocha and v6.2.2-arabica. This did not affect Mainnet.
Resolved
Both Arabica and Mocha halted across multiple occasions starting on October 30th and lasting between 10 minutes to several hours until the patch on Nov 4th (v6.2.2-mocha). The cause was a very subtle timing bug inherited in cometbft in which the consensus timer could occasionally trigger earlier than it was set to. In v6, when we introduced a minimum time before sending precommit votes in order to keep block time constant, this timing bug would cause the precommit to trigger earlier than expected. When this happened, the function would determine it was too soon and thus ignore the call, causing that particular node to freeze until it saw 2/3+ precommit votes from the rest of the network. To halt the network would require the timing bug to affect 1/3+ of voting power at the same height. While the bug was present in v6 as early as August 27th, it wasn't triggered until October 30th. It has been patched in v6.2.2-mocha and v6.2.2-arabica. This did not affect Mainnet.
Monitoring
Mocha is producing blocks as of 8687994 at 02:59 UTC.
Investigating
Mocha has stalled at block 8687830.
Please upgrade to the latest versions of celestia-app for both Arabica and Mocha to resolve the issue. Details can be found in the incident report:
https://status.celestia.org/incidents/01K98GBXSK6ZYP0HM6XH7255Y1
Investigating
Mocha has stalled at 8687711 and is being investigated.
Investigating
Mocha experienced instability and a ~29-minute halt between blocks 8653674 and 8653675 that's currently being investigated. Separately, fixes for slashing and peering have been applied, and performance has improved. Related investigations are tracked in #6124 and #6125, and appear connected to Arabica issue #6106.
