From Chunks to Blocks: Accelerating Uploads and Downloads on the Hub

The article argues that content-defined chunking (CDC) is only a means to speed up data movement, not the ultimate goal of deduplication. To scale, the team moves from per-chunk transfers to aggregation: blocks up to 64MB reduce CAS entries by ~1000x, while shards map files to blocks and detect changes, enabling faster uploads/downloads.

Source

HuggingFace Blog

Ingested

FEB 12, 2025 · 19:10

Editorial score

4.0 / 5