Solana network failure cause: known vulnerability leads to infinite loop

A post-mortem report on this week's Solana network outage points to a known vulnerability as the cause of the outage. On Tuesday, the Solana network was down for five hours. Validators restarted the network just before 10 a.m. ET.The post-mortem report, written by Anza, concluded that an infinite loop vulnerability caused validators to stall on a certain block. The infinite loop caused the validator to repeat an action endlessly, and the validator was unable to function, thus preventing the network from processing more transactions. Specifically, Solana's Just-In-Time (JIT) compilation cache encountered a vulnerability that resulted in the recompilation of an infinite loop for some older programs, thereby monopolising network resources and aborting operations. As a result, the compiler, which compiles code in real-time to improve performance, became a bottleneck. To address this issue, the Solana team deployed a fix in the new release that resolves the vulnerability and prevents such loops. This issue is consistent with a vulnerability found in the recent Devnet outage investigation, for which a patch will be deployed soon.The Solana security team first noticed this issue in April 2022.The Solana security team is also working on a new release that will be released in the next few months. The report explains, "This fix removes the prerequisites needed to trigger the vulnerability, which is a simpler solution. A more complete fix will be included with further improvements to LoadedPrograms and allow for stabilisation in the regular release cycle." This is the sixth major outage for Solana since its inception.Solana's last outage occurred in February of last year.