Erasure Coding
A technique that splits data into fragments and adds redundant fragments, so the original can be rebuilt even if many fragments are lost. Decentralised storage networks use it to survive node failures without storing full copies everywhere.
Also known as: erasure code, forward error correction
Erasure coding is how modern decentralised storage keeps data alive without brute-force replication. Instead of storing several full copies of a file on different machines, the network encodes the file into a larger set of fragments, of which only a subset is needed to reconstruct the original. A common scheme splits data into k data fragments and adds m redundant fragments, producing k+m in total, where any k of them suffice to rebuild the file. Lose up to m fragments and nothing is lost.
The appeal is storage efficiency. Full replication of a file across a network that tolerates one third of nodes failing costs roughly 25 times the raw file size once you account for the safety margin. Classic Reed-Solomon erasure coding brings that down to around 3 times. Newer schemes push it further: Walrus, the Sui-based storage network, uses a two-dimensional scheme called RedStuff that achieves its fault tolerance at about 4.5 times the raw size, far cheaper than replication while still surviving the loss of up to two thirds of its fragments.
The trade-off is compute and coordination. Encoding and decoding cost CPU cycles, and reconstructing a lost fragment can require reading fragments from many nodes. Well-designed systems minimise this: self-healing recovery rebuilds only the missing fragments, and reads the minimum needed rather than the whole file. For DeAI, erasure coding is the primitive that makes verifiable, always-available storage cheap enough to hold model weights, datasets, and agent memory without trusting a single provider.