We could save petabytes of cache storage with Zstandard and Pingora (blog.cloudflare.com)

I've done similar things for large container images. My format allows for using FastCDC to chunk files, but there's a tradeoff between number of shared chunks and between number of HTTP requests. I keep it turned off by default.

I'd be curious whether block level de-duping would add value too in their case. You effectively achieved that to some degree with storing the deltas.

You may or may not be familiar with the content defined chunking family of algorithms like FastCDC.

https://joshleeb.com/posts/chunking.html

https://www.usenix.org/conference/atc16/technical-sessions/p...

On the side I'm working on an extension to git-lfs to use fastcdc for both storage and transmission of artifacts to drastically reduce size and make git-lfs more practical for more things.

NPM packages doesn’t take up that much disk space for me with standard pnpm deduplication. The much bigger offender for me is Rust target dir: when people talk about vibe coding in Rust for performance, what they don’t mention (at least I’ve hardly ever seen it mentioned) is every trivial little tool pushes 1GB on disk and anything slightly nontrivial easily racks up multi-GB. Which hurts when you have lots of vibed small tools. I wonder if anyone bothered to look into this problem.

I was thinking this. Zstd is widely supported in browsers, over 80% right now and will increase over time:

https://caniuse.com/?search=zstd

They already do this, at least for paid accounts. You can even decide what compression model you want them to serve on your behalf.

I assume the entire resource needs to be decompressed first, then indexed into, served, and discarded. Well, actually, you could just decompress up to the end of the range.

> I don't see how that's possible if the cache no longer stores the uncompressed data.

Zstd has a seekable format for frames, similar to pigz --independent works.

[1] - https://github.com/facebook/zstd/blob/dev/contrib/seekable_f...

Idk but btrfs and zfs manage to pull it off

Seekable OCI (SOCI) uses an index so I imagine that's an option (real byte range a-b maps to compressed range x-y). Presumably you'd still need to read the header and some additional pieces

they said they didn't change the behavior for range requests. So, it'll still be the basic no compression. (eg, server side decompression)

I would compress it all, and then selectively recompress at higher compression levels depending on the link, read frequency, diversity and capabilities of the clients.

Zstd 3 to 5 is nearly free in terms of not bottlenecking disk or network. Zstd 12 to 19 gives amazing compression results and still result in speedups when reading from disk. It really is a wonderful all purpose compressor.

One of the nice things about Zstd is if you try to compress an already compressed stream, it short circuits. So even if you are given say HVEC MP4 and run zstd -19 on it, it will "compress" immediately and not DOS your pipeline.

I agree! I came to the comment section to say exactly this. In any cache hierarchy you want to put colder content in cheaper but slower storage. Here, compression is the cheaper but slower form of storage.

That part is a little bit confusing.

I think they probably don't care about storage on the devices that do the compressing and are optimizing for quickly pushing hot content to edge locations. So the compression at the source saves bandwidth during the pushing to edge phase and allows the edges to hold more (reducing churn, further saving bandwidth back to the source).

Put a different way, they're trying to make cache evictions cheaper (less bandwidth to refill) and less likely (bigger cache on same disk size)

"Ignore cold assets" makes more sense with that framing

Although if that's the case, the CPU statement still is a bit confusing.

When I worked on a large CDN the content popularity distribution was heavily skewed. Think 20-40% of throughput from top 1% if content, and 80-90% from the top 10%. Anything outside of that had a very low probability if ever being read again in the effective cache lifetime.

Then the effective cost of scaling CPU > RAM > storage > network due to power & space limitations. Spending extra processing time on 50% of your content would be wasted effort as its never read again.

And yes, increasing effective storage might increase cache width/lifetime, but its not by that much to dramatically inprove access rates. Especially when most content by unique bytes is compressed audio/video in the first place.

Yeah. It's worth looking at your own code and infrastructure as well. In my previous job there were dozens of opportunities to realise big savings by swapping out gzip for other compression schemes -- usually zstd, occasionally lz4 or bz2 or brotli. Huge assets took less hard drive space, took less time to download, and took less time to decompress. The differences were not small, and resulted in appreciable improvements both in infra cost and dev productivity.