H.264 Is Magic (2016) (sidbala.com)

434 pointsby ea01618 hours ago149 comments

tgtweak 16 hours ago

Had the pleasure of working with Jason Garrett-Glaser (better known as DarkShikari) circa 2011 on a patch for x264 to better utilize more cores (slice-threaded lookahead). At the time he was working with Sony to optimize their PlayStation game streaming service, and this had the potential to lower encoding latency (critical to game streaming) even on same-core systems by spreading the frame over more cores to do more parallel work, and doing more intelligent scheduling and convergence after that tile-based work was reconverged into a frame. At the time we were using servers with 48 physical cores to do transcoding, a bit of a rarity in a single NUMA space back then and there were no instances to test this on so I flamegraph traced a few encodes at very high thread parallel counts and send back the instrumented profiles (from a custom compiled version) showing the core work being done and he used that to optimize the motion estimation code which worked on the first shot to perfectly level out cpu usage and increase encoding speed across ~24 cores (was previously topping out around 8 and running one core hot iirc). I got a look into how exotic and complex this project was, particularly in terms how low level and advanced things like motion estimation, quantizing and perceptual compression were... it was and still is sort of magic how many smart techniques are baked into something that we barely pay attention to. They were, even at that time, working on x265.

His blog, Diary of an x264 developer, was a great read too for anyone wanting to get a vantage point on how crazy that codebase was.

[1]https://github.com/mstorsjo/x264/blob/master/doc/threads.txt

(edit: was Gaikai that was acquired by Sony in 2012 and is the basis for PlayStation Now)

mcoliver 15 hours ago

Very cool. Took a similar approach to optimizing openutv by looking at flamegraphs. UTV is a fork of openrv which is an older release of the commercial RV. Like many things it uses ffmpeg and also native readers and writers for certain formats. Allowed me to get it to 8k@60fps buttery smooth playback with hardware acceleration. If you are still in the video/imaging space would love your feedback on UTV. openutv.com

drmpeg 13 hours ago

Using multiple CPUs for encoding goes way back. The early (1994) C-Cube MPEG-1 and MPEG-2 real-time encoders for DirecTV were built with 8 chips running in parallel (out of necessity since they were only 60 MHz processors) to encode 704x480. Each chip encoded a slice of the image and an auxiliary bus was used to send reconstruction video between chips so that motion vectors could cross chip image boundaries.

Here's a picture of the VME bus development board. It was a two board sandwich to support 10 chips in parallel (only one board pictured).

https://www.w6rz.net/scorpio.jpg

The first real-time HDTV encoders for terrestrial digital television (ATSC) were built by Harris using an IBM encoder chip. It took 6 chips to encode 1920x1080. The IBM chips didn't have an auxiliary bus, so you could see motion vectors limited to chip image boundaries (in this case, 6 tiles arranged like so).

  -------------------------
  |       |       |       |
  |   1   |   2   |   3   |
  |       |       |       |
  -------------------------
  |       |       |       |
  |   4   |   5   |   6   |
  |       |       |       |
  -------------------------

tgtweak 12 hours ago

Yeah at this time intel was just releasing their quick sync tech and a few vendors were putting 4 intel cpus on a pcie board and using it for video encoding! Matrox also had a considerable piece of this business doing video compression and editing/compositing using bespoke processors and boards.

Despite being more "power efficient" - both nvidia and intel's hardware solutions left a lot to be desired in terms of final quality for the same bitrate, compared to x264 that seemed to just put out substantially better quality video for the same bitrate even at the same h264 profile levels. There are still lots of x264 design elements, some even verbatim ported, into modern av1 and hevc encoders.

drmpeg 7 hours ago

Today I learned about Intel Quick Sync. Thanks. I retired from video compression in 2011, so I've been out of the loop for a long time.

https://en.wikipedia.org/wiki/Intel_Quick_Sync_Video

rufo 4 hours ago

As someone doing a lot of DVD transcoding at the time, I remember being excited for Quick Sync on my new Sandy Bridge build - only to find that, while fast, it had absolutely _horrendous_ quality. I revisited it on my previous main Plex server build, which I think was a Haswell, and it was still unusable for transcoding compared to x264.

It took Nvidia until the encoder they bundled into 20xx series cards and Intel until the 6th or 7th generation of Quick Sync, but their hardware encoders are _finally_ at a point you can get decent quality at reasonable bitrates. Even so, x264 (and x265!) beat the pants off of them if one can spare the time and electricity; they're still that far ahead.

pwython 18 hours ago

Ten years later, 2026 magic: AV1

https://engineering.fb.com/2026/06/22/video-engineering/adop...

And soon AV2

https://av2.aomedia.org/

HumblyTossed 18 hours ago

And maybe ten more and enough hardware will support it that it will be mainstream.

Dylan16807 17 hours ago

AV1? It's mainstream already. Lots of hardware supports it and you don't need hardware support for many other cases.

alt227 17 hours ago

Mainstream? The most popular hardware streaming device (Nvidia shield) does not even support it, so I wouldnt call it mainstream.

Firerouge 17 hours ago

Isn't the latest shield hardware 7 year old at this point? Only one year newer than AV1 itself

duped 17 hours ago

Niche luxury products aren't exactly mainstream either

alt227 15 hours ago

I guess it depends on your definition of the words niche and luxury.

judge2020 17 hours ago

Nvidia Shield is hardly the most popular anything. Roku has 43% of the market and Amazon has 35%.

Would be interesting to see if we could segment third-party from TV-embedded (see quote), but the metrics from the article seem to reflect actual consumer usage, as most consumers just use what comes in the TV they buy nowadays.

> This reflects a consolidated approach and eliminates yet another device in the home, and it’s catching on, as just 34% consider streaming media players as their primary viewing device. Meanwhile, 56% said the same for smart TVs.

https://www.cepro.com/news/roku-amazon-lead-streaming-media-...

alt227 15 hours ago

Interesting you bring up Roku as a main comparison, as only the Roku Ultra device has AV1 support and that is via DASH only:

https://developer.roku.com/dev/docs/media

So if you are saying Roku has 43% of the market, I would consider that as support for my point that AV1 is not that widely supported.

judge2020 14 hours ago

Very fair. Looks like Amazon is a bit better with AV1 support starting in 2021 with the Fire TV Stick 4k Max - 1st Gen[0], with a single absense of it on the seemingly uber-budget friendly $35 "Fire TV Stick HD Alexa Voice Remote (2024)".

That's still a sliver of the market, but the more royalty-free* codecs we use, the better.

0: https://developer.amazon.com/docs/device-specs/device-specif...

*: https://arstechnica.com/gadgets/2026/03/av1s-open-royalty-fr...

Rockjodd 15 hours ago

> Nvidia Shield is hardly the most popular anything.

This is so wrong..

> From Jellyfin blog: https://jellywatch.app/blog/jellyfin-nvidia-shield-tv-setup-...

>> The NVIDIA Shield TV (Pro and Tube) is widely considered the best Android TV streaming device for self-hosted media servers in 2026. While the Apple TV 4K wins on UI polish and the Onn 4K Pro wins on price, the Shield wins on something most other devices can't match: lossless audio bitstream passthrough. TrueHD, TrueHD Atmos, DTS-HD MA, DTS:X - the Shield passes them all to your AV receiver untouched.

>> Combined with hardware decode for H.264, H.265, VP9, and AV1, plus an 8-year-old Tegra X1+ chip that still gets driver updates, the Shield is a genuine "set it and forget it" Jellyfin client.

Nvidia shield is a device that simply works and passes the audio untouched to my sound receiver and I enjoy it __everyday__.

EDIT: <trying to fix formatting>

john_strinlai 14 hours ago

the jellyfin quotes are saying "best", which is significantly different from "most popular", which is what judge2020 said.

MadameMinty 17 hours ago

av1_nvenc exists therefore it's mainstream :)

fishgoesblub 17 hours ago

I would not consider the ancient Nvidia shield the "most popular" streaming device.

sanfus_grimmer 17 hours ago

Nvidia hasn't released a new model of the Shield since 2019. The current Apple TV model, Google TV model, Roku model, Fire TV, and Onn models support AV1.

croes 16 hours ago

My $100 NUC has AV1 hardware decoder support.

Maybe a device from 2019 is a bad data point

bubblethink 14 hours ago

Shield is a 10 year old device at this point. It's essentially the 2017 hw that they've refreshed a couple of times with minor changes. Wait for the actual shield next for proper support.

cmckn 6 hours ago

You can’t be serious, I bet 9 out of 10 owners of streaming devices haven’t even heard of the Nvidia Shield.

Fr0styMatt88 4 hours ago

The Shield is dead though and a long-time abandoned, right?

I remember the Shield going from fantastic to a horrible mess software wise over the span of a single Android generation (maybe 2019 or 2020), constant ARC weirdness, etc.

serallak 37 minutes ago

Last official update was in February this year, so not really abandoned.

But no new models since 2019.

GeekyBear 16 hours ago

Sorry, but YouTube began thermal throttling on a Firestick the moment they started forcing AV1 regardless of the presence of on-device hardware support.

Dylan16807 16 hours ago

Then that particular model doesn't support it properly. I don't think that changes my point.

GeekyBear 15 hours ago

If the common hardware customers already have fails when you attempt to use a newer codec (but works fine otherwise), that can't be said to be a mainstream codec yet.

Dylan16807 15 hours ago

Your particular model is not representative of "the common hardware" as a whole. And it can be mainstream even when a significant fraction of devices can't use it. AV1 is a third of Netflix streams and most YouTube streams, that's more than enough to qualify as mainstream.

GeekyBear 15 hours ago

It's on most of YouTube streams because Google forces it on devices that lack hardware support, even if it breaks the functionality of the device.

Mainstream support doesn't have to be forced.

Dylan16807 14 hours ago

Even if half the devices were broken, it would still be more than a third of streams and mainstream.

A lot less than half of devices are broken when playing YouTube, I assure you.

If they stopped forcing it, it would still be the best choice most of the time. It doesn't "have to be" forced to be a major choice. The forcing changes the exact percentage but not the overall picture.

mrob 14 hours ago

>forcing AV1 regardless of the presence of on-device hardware support

I'm posting from a machine without AV1 acceleration. I disabled AV1 in Firefox's about:config and Youtube falls back to VP9.

pxoe 17 hours ago

AV1 decode works well enough even on quite old devices, and it has more favorable quality and size, individually or when considering both. I prefer it to h264 (or vp9 or even h265) at this point, and there isn't a particular need for hardware decode to benefit from it. On Youtube (probably the biggest and most accessible source of av1 video), when comparing size and quality on different codecs (h264/vp9/av1), av1 is either gonna have much smaller size while keeping some level of quality, or have better quality at similar size, or even have both be better.

paulryanrogers 15 hours ago

Firesticks are quite popular. I'd like to see what percentage in the wild actually does decode AV1 in hardware, and without stuttering or buffering problems. Even my newer devices sometimes choke on h265.

OGWhales 13 hours ago

I had an iphone 13 and av1 playback was pretty bad, could technically be done with some apps but never worked well. New iphone it works quite well.

cubefox 16 hours ago

The first Apple chips with AV1 support only came out in 2023. So the majority of Apple hardware out there doesn't support it. It will be a long time till support is at >95%.

Gigachad 13 hours ago

I’ve found that YouTube just uses it anyway in software decoding. 4k AV1 video seems to use about 10% cpu on my M1.

shevy-java 17 hours ago

We need one final video codec.

Otherwise we will have this:

https://xkcd.com/927/

Dylan16807 17 hours ago

At any point in time there's only a couple video codecs you should be using. When we can repeatedly shave off 30% of video bandwidth, there's no reason to stop making one or two new ones every decade.

Edit: And you can also look at the video codec landscape as having two major standards that put out new versions sometimes. The MPEG/h.26x series and the VPx/AVx series. It's not quite that simple but it's close.

aktenlage 17 hours ago

There is no need to have a final one. It's no problem to keep supporting older codecs and add new ones in new products, as long as there's a "common ground" of widely supported codecs.

taneq 14 hours ago

I don’t think we’re far off a “final” video codec, in the sense that jpg and mp3 and zip are effectively “final” image and sound and data codecs. Sure, there are newer codecs with higher compression foot a given quality level, but at some point it’s good enough to not bother upgrading, especially when the new version costs exponentially more for negligible gain.

…maybe THIS one will be The Final Codec… doodoodoo dooo

thunderfork 10 hours ago

Re: mp3, Opus is a pretty big improvement. In my experience I've found that Opus is pretty reliably transparent at 96kbit/s, whereas LAME (which really squeezes mp3 for all it's got) needs ~130kbit/s

I think encode costs are higher (although I'm struggling to find any actual data) but in my experience this is totally negligible in practice.

Of course, don't go converting all your mp3s to Opus, but if you're running a music streaming server or whatever, Opus is pretty sweet... and probably close to "as good as it gets" until/unless we figure out some crazy new tricks.

ChadNauseam 9 hours ago

> jpg and mp3 and zip are effectively “final” image and sound and data codecs

In popular consciousness, maybe. When people think image they thing jpg. But most websites you use aren't delivering images as jpg. Most sites care about saving money on bandwidth and their site loading fast, and because of that they tend to use webp and avif, which are much smaller for the same quality.

aidenn0 4 hours ago

jpg: webp, avif, jxl. The first two are adopted because of non-negligible gains, the latter because of features missing in jpg.

mp3: Both AAC and opus have non-negligible gains and both are fairly widely used.

zip[1]: lz4, zstd, and lzma2 dominate at least gzip's implementation of deflate: For any given setting of gzip, there is a setting for at least one of those three that is both faster and smaller.

1: Not a codec; the zip format specifies a dozen methods for encoding. I'm assuming you mean Deflate, which is the most common compression method associated with zip.

echelon 17 hours ago

I feel as though AI is going to blow these all out of the water soon.

And then we'll get AI-designed hardware codecs for them.

cowlevel 17 hours ago

Or the AI will be the codec - intelligence is compression.

cubefox 16 hours ago

Yes, "neural" or "learned" video codecs are considered the future. They already have competitive compression ratios, but last thing I heard was that they are still much slower than conventional codecs.

qmr 17 hours ago

That's all well and good, but let me know when I can buy a used $50 NUC or pi or such that has hw decoding of the same for my TV front ends.

kccqzy 16 hours ago

Intel has supported native hardware decoder for AV1 since Tiger Lake. That’s quite old. You could buy a used NUC with Tiger Lake. Even when new a NUC with 1115G4 or 1135G7 isn’t expensive. I found one with damaged WiFi for $20. https://www.ebay.com/itm/366767313411 You can buy it now.

ssl-3 15 hours ago

I went to the link to buy a computer (a NUC11TNKi3) with this capability for $20, but all it would let me do is place a bid on it; I cannot buy it now.

It looks like they normally sell for around $200: https://www.ebay.com/sch/i.html?_nkw=NUC11TNKi3&LH_Sold=1

Dylan16807 15 hours ago

Do you really care if your NUC uses an extra 2 watts to do software decoding?

ender341341 9 hours ago

Lots of media devices that don't have AV1 decoders tend to have issues decoding high resolution AV1 in real time so you either get pauses to buffer or dropped frames.

Dylan16807 8 hours ago

Yes but if it says "NUC" I doubt you'll have that issue.

NavinF 14 hours ago

Cheaper to get a long fiber optic HDMI cable and connect your TV to your PC GPU. That's what I do. You get the best uncompressed video (typ 4K 120Hz or 8K 60Hz), zero lag so it can be used for stuff other than video, and don't have to maintain another box

a3w 16 hours ago

2018 AV1 demo

https://news.ycombinator.com/item?id=16796127

walrus01 2 hours ago

If you really want to see magic, take a test sample of YUV420 video and AV1 encode it to an extremely low bitrate at like 720p, 24 fps to simulate something like the jerky handheld front-phone-cam in the example URL. Keep lowering the bitrate until the quality become unacceptable. Then compare it to the same bitrate/approx total file size in H264 and how bad it looks.

Dwedit 16 hours ago

One uncommonly used feature of H.264 is support for lossless video. It's a bit hard to use due to tooling (not easy to discover and select the feature), and is sometimes incompatible with web browsers, but you can get actual lossless video using a mainstream video codec. You can verify that it really is lossless, and doesn't have any conversion step where it's quantized to 8-bit YUV.

"x264vfw" is one tool which supports this format, even inside of an AVI file. So older software that uses Video For Windows can create those files.

ledoge 16 hours ago

FYI, both VP9 and AV1 support lossless as part of their base profiles, so lossless files using those codecs should be compatible with all implementations (at least in theory). Though those profiles only support 4:2:0, so it depends on your use case if that really counts as lossless.

Dylan16807 15 hours ago

I struggle to think of a situation where the blur of chroma subsampling is acceptable but very high quality or prores isn't good enough, I need lossless.

Dwedit 15 hours ago

4:2:0 is not lossless. 4:4:4 with quantizing down to 8-bit YCbCr isn't lossless either. But 10-bit 4:4:4 is reversible back to 8-bit RGB without loss.

thunderfork 11 hours ago

There's a distinction to be made between lossless (compression) and lossless (bit depth/gamma/yadda yadda); typically the format conversion isn't considered part of the "lossy/lossless" distinction.

Definitely not something to be forgotten if you're expecting to compress lossless RGB, although this is pretty rare outside of niche-ish use cases (afaik game capture is basically the only place where people are likely to be using RGB and not YCbCr? could be wrong!)

Dwedit 9 hours ago

Animation production (2D or 3D) can also have raw RGB frames.

thunderfork 7 hours ago

Ah, I hadn't considered that but that totally makes sense!

aidenn0 4 hours ago

Also (if you round correctly) there is no generation loss in the format conversion. You can round-trip between them as many times as you want and you only get the loss of the initial RGB->YCbCr.

doublepg23 16 hours ago

What is the purpose of lossless H.264 vs. ProRes or the like?

Is it similar to lossless audio were there's not a ton of optimization you can do?

freeone3000 16 hours ago

Lots of hardware requires specifically h.264 as a transit; many of the MPEG standards are built around it. So this lets you get MPEG-PS, or lossless BDs, or some hardware/specialized software decoders for machines that otherwise couldn’t keep up (like set top boxes).

shellac 16 hours ago

I've attempted to get researchers to use lossless H.264 to encode CT scans. They typically use directories full of tiffs. Even with intra-slice compression (i.e. tiff compression) they are pretty huge. But H.264 will let you recover the original images to the individual bit, and save a huge amount of space thanks to inter-slice similarities.

They are still using directories of tiff, though.

jedbrooke 14 hours ago

I’d imagine the main drawback of using H.264 here is that you’d still have to decode to get the frames back. This is similar to how even though many consumer grade cameras will output natively to h.264, if you want to edit the videos from it you’d usually have to convert it to prores or something first (the editor would usually do this automatically behind the scenes)

I’m not sure if lossless h.264 is I only or not, but if it uses inter-frame compression then it’s difficult to randomly access arbitrary frames. For any arbitrary frame you’d have to find the nearest key frame, then interpolate from there based on the inter-frame compression metadata.

ammo1662 3 hours ago

It's acceptable to use the nearest key frame when randomly accessing arbitrary frames.

The real problem is that you'd have to rewrite the entire app stack to use H.264 or AV1 for storing the image stack generated by CT.

CT images are 16-bit monochrome, and the pixel values have real physical meaning (Hounsfield Units). So you'd have to prove that your compression tool can reproduce the original images 1:1. The H.264 encoder would also need to be modified to support 16-bit pixels.

And you'd need to modify the post-processing tools to accept your new file format. Otherwise users will still have to convert it back to DICOM or some other commonly accepted format before they can use their post-processing software.

So unless you're a CT vendor, you'll have a hard time pushing this into a hospital.

Sopel 16 hours ago

prores is not lossless. It's also an intraframe codec so compression is not great.

ErroneousBosh 12 hours ago

It depends what you mean by "not great". Prores's compression is far better than say H.264, if you actually want usable footage for editing.

Sopel 16 hours ago

you can easily record lossless h264 via obs and any ffmpeg-backed player will play it just fine

the days where the OS matters for video playback are long gone

aidenn0 4 hours ago

x265 supports lossless as well (though I believe it's not part of the H.265 standard). It's what I use for video capture.

Hasz 17 hours ago

I really like the idea that compression is intelligence, there are so many tricks an interesting ideas related to it. A bunch of them are described in the article, but you can imagine a long tail of many more inscrutable tricks chained together to get really incredible shrinkage in data.

Hutter prize (I think mentioned in a 3b1b video somewhere else in this thread) is a great example. Given this was written before advent of LLM, I would love to see someone throw openAI/Anthropic levels of compute at a similar problem to see what the swarm can produce.

https://arxiv.org/html/2404.09937v1

augment_me 17 hours ago

The newest submissions to Hutter Prize made crazy jumps by just replacing the LSTM with a transformer and training it.

35'000$ for a prompt and some training is a nice chunk of cash.

Next steps are replacing the encoders and all the other human-biased compression parts by just raw data and training, probably another 20-30% is up for grabs.

kouru225 16 hours ago

I remember reading an article on here by one of the founders of MPEG about why he was leaving MPEG after several decades, and part of it was that modern AI is indistinguishable from compression and he wanted to explore new compression algorithms using the recent explosion of knowledge

HelloUsername 18 hours ago

Previous discussions:

4.nov.2016 https://news.ycombinator.com/item?id=12871403 219 comments

17.mar.2022 https://news.ycombinator.com/item?id=30710574 219 comments

24.may.2019 https://news.ycombinator.com/item?id=19997813 180 comments

14.jun.2024 https://news.ycombinator.com/item?id=40681306 205 comments

tgtweak 15 hours ago

It's still magic

ksec 17 hours ago

H.264 High Profile is close to Patent Free. [1] I often wonder how much more efficiency could we get if we were to increase block size outside of spec.

[1] https://meta.wikimedia.org/wiki/Have_the_patents_for_H.264_M...

MadameMinty 17 hours ago

Worth pointing out that PNG is also just awful and should probably die like BMP died. WebP was already available at the time this article was written:

PNG: 1039993 bytes - baseline

optimized*: 541539

WebP: 398974 - lossless mode, of course.

JPEG XL: 310778 - lossless, quick, and easy.

*after monumental compute effort; still probably some room for improvement but the effort is ridiculous.

ashleyn 17 hours ago

WebP stands on its own merits but it's profoundly annoying that as of Pop OS 22.04 I still have to open them up in Firefox instead of Eye of Gnome. Compatibility issues like this are littered everywhere. This laziness on the part of supporting WebP is a significant driver of continued use of PNG.

MadameMinty 17 hours ago

Yes, it's very disheartening. We're approaching two decades of WebP, after all. At this point it's not even worth implementing its support because JXL is superior in every use case.

Dwedit 17 hours ago

Not really, lossless WEBP still decodes much faster than lossless JXL, so if you need to decode the images faster, you use WEBP. Yes, you could also speculatively decode the JXL images to hide the decode time.

For lossy, neither is in a good place, since lossy AVIF beats both lossy JXL and lossy WEBP. Lossless AVIF is a joke and barely competes with PNG.

cubefox 16 hours ago

I think the case for JPEG XL is not that it's the best at everything, but that it is both quite good overall while also supporting basically every obscure image format feature under the sun. It's the potential USB of image formats.

ChadNauseam 9 hours ago

> while also supporting basically every obscure image format feature under the sun

Except for gain maps :)

I don't really mind because I don't use them anyway. I prefer to use the client's tonemapper. But they are a useful feature that's not supported by (chrome's implementation of) jxl

cubefox 2 hours ago

They are supported by JPEG XL (libjxl ≥ 0.11), Chrome just doesn't support them yet. See https://issues.chromium.org/issues/476652050

MadameMinty 16 hours ago

JXL decoding can still be optimized and had significantly less time for it, so let's not be so sure if that statement holds. Last update brought much improvement at just the area you mention.

And AVIF compression ratio beats JXL only conditionally. E.g. my photo library had AVIF at 29% and JXL at 25% the input size at the same perceptual quality (SSIM2=83). Sometimes it's on par. Sometimes AVIF wins. JXL wins almost always. Not to mention you can losslessly, reversibly transcode lossy JPG files to JXL, getting significant size reduction for free.

Joel_Mckay 11 hours ago

Try this, as it should support animated webp features in eom. =3

sudo apt install eom webp-pixbuf-loader libimlib2-dev libwebp-dev pkg-config libwebp-dev libgdk-pixbuf2.0-dev libgtk-3-dev

SchemaLoad 10 hours ago

Open a bug report on Eye of Gnome. WebP has no issues opening on all of the native MacOS tooling. Since this is an open source program it should be fairly easy to submit a PR to bring in webP support.

masoudd 5 hours ago

Eye of Gnome can open webp. You need webp-pixbuf-loader package installed.

diericx 5 hours ago

Classic

aidenn0 4 hours ago

That's crazy; gwenview (The KDE image viewer) has supported webp for quite some time.

abhinavk 2 hours ago

Gnome is pretty ahead on JXL support though. They even ship their wallpapers in it.

MadameMinty 17 hours ago

For completeness, my background job of the highest imaginable PNG compression yielded optimized: 528313 bytes. Took only several minutes...

F3nd0 17 hours ago

With JPEG XL finally being allowed into mainstream web browsers, we’ll hopefully see more native platform support and be able to slowly phase every older codec out, since JPEG XL does very well at more or less everything (except for animation, compared to video codecs).

MadameMinty 16 hours ago

I hope so, but I disagree with the size of the "except": JXL performs better than APNG and GIF, and still supports transparency and lossless encode. Video codecs struggle here. So even with animation it still depends!

There's also a case for AVIF in some use cases (for this image it performs poorly). But if I had to choose one, I'd choose JXL.

abhinavk 2 hours ago

Gnome ships wallpapers in JPEG XL. Decoding support has landed but export support is either sparse or untuned.

spiralpolitik 6 hours ago

WebP is limited to 16383x16383. I've run into this limit a handful of times and I would expect it to become more of an issue as screen pixel densities increase.

tvelichkov 17 hours ago

This reminds me that I still haven't watched `part 2` of a very nice video from 3Blue1Brown - "Compression is Intelligence". YouTube suggestions, definitely failed me with this one.

johnwalkr 14 hours ago

I've shared this article many times. I work in space and the vast majority of the time, new people insist that all images should be raw. After all, you go to great expense to get somewhere in the name of science, you don't want to discard any information. And that's true in some cases but typically not for a whole mission or even a whole image if you can crop a specific part. Usually 100 or 1000 images instead of one is more valuable even if the compression is lossy.

That's just image compression though. Very few space-grade cameras support video compression. In a lot of cases it works well even though you think you're after a series of still images.

jwrallie 9 hours ago

It’s mostly mission dependent as you mentioned, if the value of some images are different than others, compressing several pictures and selecting the best to get in raw is a good strategy, and you can also crop them to get interesting areas as mentioned.

It can also depends on the downlink bandwidth available and power/energy budget, so the optimal strategy will differ.

It can be misleading at times though, for example some interesting feature may happen to be just a compression artifact. A cloud looking like some strange human-made object for example, happened to us one time.

userbinator 4 hours ago

~370 MB/sec of raw data.[...]This is next to impossible to deal with.[...]You can't move it anywhere fast. Even SSDs have trouble dumping this straight from RAM to Disk

Always interesting to see what happened in ten years --- 370MB/s is well below what the slowest NVMe SSD can do today, and easily achieved over a 5Gbps Ethernet connection.

Incidentally, H.264 patents have either expired or will be expiring soon: https://meta.wikimedia.org/wiki/Have_the_patents_for_H.264_M...

jacquesm 3 hours ago

On average, yes. But without hiccups: only very few NVMe drives can do this over really long runs without stalls. I've been working a lot with high rate SDR earlier this year and it took a lot of testing of different drives to find one that did not have this issue. I was quite surprised because you'd think that the underlying mechanism, if it works at all should be able to work sustainably but once every 10 minutes or so there would be a (brief) stall which would cause the SDR to start dropping data because it wasn't serviced fast enough. The only one that didn't do this was the 'Lexar SSD NM1090 PRO 4TB', which was surprising because it was one of the cheaper ones that I tried.

tombert 15 hours ago

I remember reading this article when it was new, and I left fully agreeing, but more broadly: modern video compression is magic.

Like, it's baffling to me how much we can compress high-quality video nowadays. HEVC and AV1 and VP9 are all also very cool and they still sort of break my brain. I understand how they work but it doesn't feel like it should compress as well as it does.

Latty 15 hours ago

I've done AV1 encodes where I was convinced the conversion must have failed in some way for it to be that small, scrubbed through it and it all looks fine, watched through the entire thing to confirm that wasn't the case, just waiting for the shoe to drop, but then to find out, no, it's really just that good.

bugake 7 hours ago

In the end, YouTube still loses frames above 1080p, even if you have fiber-optic internet, the ultimate CPU and GPU, and loads of RAM, just because YouTube's code is garbage.

juanfra91 9 hours ago

I work with video sites and I'd never understood why an hour-long video could be so small. Saving only what changes between frames explained it all. It also explains why videos of rain or confetti look pixelated.

yashk1010 3 hours ago

Reading human written article (No LLM) is soo fun. Can't find it anywhere these days.

Dwedit 17 hours ago

One year left before the "last major patent" expires.

kouru225 16 hours ago

Wait is this true? Will I finally be able to use h264 on da Vinci resolve for free?

paulryanrogers 13 hours ago

I am right now. Though perhaps not with GPU acceleration.

bugake 7 hours ago

It is time to make a standard neural network for video, which guesses the next frame and/or applies super-resolution like DLSS does, so the necessary information to transmit is reduced to the minimum necessary for DLSS to reproduce the image without loss.

ummonk 12 hours ago

I wish the video presented had been one with, say, 5x higher bitrate, as it would have sufficed to make the point without presenting a video with very glaring compression artifacts.

dizzant 15 hours ago

I'm surprised that 10 years later, the article still teaches an incorrect conceptual understanding of "quantization".

The context heavily implies that quantization is "discarding information" generally (e.g. clipping, masking). The wikipedia article he links in the footnote disagrees, and so do I:

> quantization is the process of mapping input values from a large set (often a continuous set) to output values in a (countable) smaller set, often with a finite number of elements.

Quantization is a function over values, or the image itself. Clipping and masking, by contrast, are functions over the space/frequency dimensions, i.e. the pixel indices ("preimage"). A rule like "apply a circular mask" or "drop the first quadrant" does not depend on the values!

The most extreme example of quantization is a threshold filter. Regardless of location, output "1" if the pixel is brighter than X, or "0" otherwise. This depends only on the value, not on the location. He also misses the opportunity to point out that chroma subsampling is another application of quantization, this time to chroma values (drop the low-order bits).

Codecs like H.264 and JPEG use spatially-varying quantization rules to effectively discard more information at high frequencies, so his explanation of H.264 is spot on, but a non-expert encountering quantization for the first time would probably come away with the wrong idea. For an article explicitly targeted to non-experts that goes out of its way to introduce the term, it's a pretty important oversight.

solarkraft 15 hours ago

Consider me confused. Isn’t quantizing a value, e.g. reducing the number of bits in an int, a form of discarding information from that value? Isn’t your comparison just about discarding information on another level (discarding entire values)?

edflsafoiewq 13 hours ago

I think the GP's point is, if you don't know what quantization is, the article gives the impression it consists of discarding some of the values, which is where the compression comes from. This is a good "0-th order" intuition, but quantization is a softening of this; instead of fully retaining or fully discarding a value, there is a sliding scale of how much information to discard. If you already know what quantization is, it's easy to see how discarding is a special case, but the article gives the impression discarding = quantization.

dizzant 11 hours ago

Exactly correct. Quantizing is like rounding. This is the same notion as in quantum mechanics: the values aren’t any real number (a huge input space), but rather integer multiples of some constant (a small output space). Quantization is “rounding” to the nearest allowed output.

To build on the example used in the article, you could quantize an image by dropping the lower half of the bits of every pixel value. This discards half the information by mapping the larger input image space (e.g. 8 bits per pixel) to a smaller output image space (e.g. 4 bits per pixel), effectively scaling/rounding the original values.

The fact that higher frequencies can be quantized more intensely with relatively little image quality loss is a completely separate idea from quantization itself. In this context, high frequency masking as described in the article could be accurately described as a spatially dependent quantization to zero-bit output values. But the quantization part is the mapping, not the shape of the mask, and I think the article is misleading about the distinction.

In real codecs, there usually isn’t a hard boundary like a mask, but instead an empirical table of how many bits to keep based on the perceptual value of the information at each frequency.

CharlesW 15 hours ago

The definition you quoted explains why quantization discards information: multiple distinct input values map to the same output value. Once that happens, you can no longer determine the original input from the output alone. The distinctions between those inputs are the information that has been lost.

dizzant 12 hours ago

I’m not saying that quantization doesn’t lose information. Instead, I’m saying that every step to lose information is not quantization. Functions on the preimage (clipping, masking) are not quantization.

sacredcows 16 hours ago

This reminds me of that old nursery rhyme. Billy was a chemist's son, but Billy is no more, what he thought was H2O was H.264

rishabhd 17 hours ago

It is tradition to re-read this every year. Along with the m1 blackmagic one.

jruohonen 16 hours ago

Ref.:

https://news.ycombinator.com/item?id=38099633

amelius 16 hours ago

I wonder if an AI can make a better codec, and if we would think of it as "magic".

tgtweak 15 hours ago

There are some interesting advances in neural video encoders and decoders - I suspect it will look a lot like the neural models in DLSS 5 for compression -> decompression/reconstitution. There have been some lossless text-compression neural models built that seem to surpass even high level compression algorithms so with time those could get there.

Dylan16807 15 hours ago

The fact that you said 5 worries me.

bugake 7 hours ago

Transmission cost of an interpolated frame: 0 with DLSS. But adding a little extra information could help reduce artifacts.

Sure, DLSS uses game engine information, such as motion vectors and depth, which is unavailable for video. But neural networks can estimate motion directly from video frames and interpolate without that information. Providing additional data to fix artifacts would cost little.

whall6 17 hours ago

Socrates was certainly a pioneer of conveying information efficiently

Dan42 15 hours ago

Well, that was disappointing. Everything in that article is generic to any video compression codec, even MPEG. I thought this would discuss why H.264 compresses so much more than predecessors, but not a word about that. Should have been titled "video compression is magic".

dzdt 18 hours ago

Article is dated (2016).