- On Saturday night, millions of transactions per second flooded the network
- Validators crashed while trying to clear up transactions in abandoned blocks
Bots linked to a new NFT project running on Solana caused a seven-hour network outrage on April 30, project developers told CoinDesk. During that time, no new blocks were produced by the network.
The blockchain processes 2,700 transactions per second (tps) on average according to explorers. As per developer documents, the upper limit is 710,000 tps on a standard network.
Millions of Transactions Flooded the Network
On Saturday night, millions of transactions per second flooded the network. As a result, network validators ran out of memory and crashed, developers said. The official Twitter account of the ecosystem stated:
Solana’s Mainnet Beta cluster suffered a 7 hour outage caused by stalled consensus. Below is an investigation into the cause of the outage, and the steps core developers are taking to increase network resiliency and stability.
Developers wrote:
An enormous amount of inbound transactions (6 million per second) flooded the network, surpassing 100 Gbps of traffic at individual nodes. There is no evidence of a denial of service attack, but instead evidence indicates bots tried to programmatically win a new NFT being minted using the popular Candy Machine program.
Validators crashed while trying to clear up transactions in other, abandoned blocks. They had to evaluate an excessive number of total forks, and this couldn’t happen automatically. Manual intervention was needed to resolve the issue.
Bots Swarmed Candy Machine
As reported, bots had swarmed Candy Machine, a popular NFT minting tool. The purpose was to take part in a new NFT project mint.
There ended up being a lot of spam on the network because that mint did not involve a Dutch auction, which is the typical process crypto projects follow. Instead, it involved a fixed price for users, which encouraged bots to make a colossal number of transactions to win the mint.
Intermittent Congestion Issues
In the meantime, developers pointed out Solana’s multiple congestion issues in the past months. Most of these were caused by bot activity targeted at NFT mints.
What is Solana Doing About All This?
The upcoming v1.10 release, which is stabilizing on testnet at the moment, includes memory use improvements to extend the period, over which validator nodes can endure stalled or slow consensus. This would help prevent or at least mitigate issues like those seen on April 30.
Developers add that Solana will introduce fees for prioritizing transactions. This could help prevent network crashes due to a single project gone wrong.
