An anomaly was said to have caused nearly half of Cardano’s nodes to disconnect and restart over the weekend, with the root cause still under intensive investigation.
Cardano stake pool operators (SPO) together with users have been reporting nearly half of the nodes for the network shortly went offline over the weekend.
Based on a January 22 post shared on the Telegram SPO for Input Output Global, the engineering and research fintech behind the Cardano blockchain, an anomaly resulted in 50% of Cardano nodes disconnecting and restarting.
Last night during the anomaly on the #Cardano network, the entire network did not go down. There was a brief period of degradation. Most nodes impacted had gracefully recovered. No network restart was required. pic.twitter.com/FupQXk2otV
— Rick McCracken DIGI (@RichardMcCrackn) January 22, 2023
While explaining the sudden disruption, the post stated:
“This appears to have been triggered by a transient anomaly causing two reactions in the node, some disconnected from a peer, and others threw an exception and restarted.”
Despite a momentary degradation, the Cardano network recovered without any external intervention. As highlighted in the post, “such transient issues” were considered within the node design and consensus and “the systems behaved exactly as expected.”
During that anomaly that happened between block 8300569 and 8300570, block production allegedly kept going but was slowed for several minutes and the “impact was low, akin to the delays that occur during normal operations.”
“Most nodes automatically recovered, depending on the SPO of choice.”
At the time of publication, the root cause of the anomaly and resulting node disconnections and restarts are still under investigation. The official announcement said:
“We’re now investigating the root cause for this anomalous behavior and implementing further logging measures alongside our regular monitoring procedures.”
The co-founder of Node Shark and a Cardano SPO, Tom Stokes, reported in a January 22 post that more than half of the listed nodes were impacted. He also shared a chart showing where the network sync dropped from 100% to around 40% for more than 300 reporting nodes.
A few hours ago over half of all #Cardano nodes went offline. This is why decentralization matters. pic.twitter.com/NXDVyKy8ep
— Tom Stokes (@eUTxO_pro) January 22, 2023
Based on the Stokes chart, the network sync recovered back to about 87% after the drop but did not instantly return to its previous level of 100%. Another SPO reported similar issues to Stoke based on a January 22 post, but declared “Some SPOs saw no impact.”
They said:
“Others had relays and BPs restart. SPOs, Devs, and IOG are in Discord debugging atm. No root cause yet.”
