From Chunks to Blocks: Accelerating Uploads and Downloads on the Hub
The article argues that content-defined chunking (CDC) is only a means to speed up data movement, not the ultimate goal of deduplication. To scale, the team moves from per-chunk transfers to aggregation: blocks up to 64MB reduce CAS entries by ~1000x, while shards map files to blocks and detect changes, enabling faster uploads/downloads.