So in practicality, you’re going to want an arbitrary limit on this (the article suggests as much). But if you place a limit on it then you’ve got one implementation of the standard that can decode certain characters and another that can’t. Better to have one standard that puts a hard limit on the number of bytes and another standard that uses more bytes and so on.
UTF-8000: Unlimited UTF-8 (utf-8000.jb2170.com)
49 pointsby vismit20004 hours ago28 comments
2shortplanks an hour ago
On a practical matter, it seems like a bad idea to have codepoints that can take up to an arbitrary number of bytes - this just screams buffer overflow problems.
flohofwoe an hour ago
OTH UTF-8 is just one variable-length stream encoding among many others (RLE, LBE128, etc...).
mqus 15 minutes ago
Some ideas of what to do with this space:
- fully-customizable emojis (think of a RPG-like character customization screen)
- heck, why not full jpegs/gifs?
- some unicode programming script (running Doom)
- ?
That said, some very minor (HN-style) nitpick:
> Otherwise for an n byte code unit this is (5n+1) / 8n, that is 5n+1 content bits out of a total of 8n bits from n bytes. We can rewrite this as (5/8) + 1/(8n) which moderately quickly approaches 5/8 = 62.5%. It is nice that this limit is nonzero and does not depend on n.
Isn't a limit by definition no longer dependent on n?
jeroenhd 6 minutes ago
U+E000–U+F8FF, U+F0000–U+FFFFD, and U+100000–U+10FFFD can already provide you with your own emoji, as that range has been reserved for private use. Extending the range further might make sense if you need even more space in your program, but that's a lot of space already.
Sharlin an hour ago
UTF-8 originally supported up to six-byte encodings (see eg. RFC 2279), but it was restricted to four bytes in 2003 in order to match UTF-16 constraints :(
delamon 33 minutes ago
We still have about 85% of codepoint space unused. Hopefully, by the time it becomes a proble, UTF-16 will be long dead
nasso_dev 22 minutes ago
i hope so too, but UTF-16 being used by languages such as java and javascript makes me fear it might be here to stay.... i hope im wrong
lukasgelbmann 14 minutes ago
Self-synchronization in UTF-8 is intuitively a great thing to have, yet I don’t remember actively relying on it ever. Does anyone have a good example of when it‘s useful?
Another related nice property that UTF-8 has: substring search reduces to bytestring substring search. I.e. given two Unicode strings in UTF-8 encoding, you can check if one is a substring of the other by just treating them as bytestrings and checking if one bytestring is a substring of the other bytestring. This is a stronger property than self-synchronization: UTF-8 has it, but UTF-8000 doesn’t.
sph 2 hours ago
> UTF-8000 is in no way endorsed by or representative of the Unicode Consortium.
Not until they decide to expand the emoji range, allocate space for all past and future fictional languages, as well as birdsong and dog barks.
Someone at the consortium is rubbing their hands with glee with all the newfound space.
But honestly, cool hack! If you invent a method to encode large numbers into bytes, why limit yourself to 24-bit numbers?
flohofwoe an hour ago
> ...24-bit numbers?
Technically current UTF-8 only goes up to 21 bits (that's the current UNICODE range), for the encoding itself that is an arbitrary limit though, with the 'single lead byte' method of traditional UTF-8 it could go up to 36 bits "payload".
achille 2 hours ago
> Ken Thompson: "...i really dont think it is useful. it is like replacing ipv6 with ipv50"
yyyk an hour ago
Just limit it to 8 bytes at which point you always do 'know the number of follow on bytes' from the first byte.
Nobody needs more than 4.47 trillion characters. (famous last words)
rsanheim 43 minutes ago
This is like TIME CUBE (https://en.wikipedia.org/wiki/Time_Cube) but for encoding nerds.
edit: replaced false prophet site with wiki link for the original one and true timecube.com...lost to the sands of ...time.
Dwedit 2 hours ago
FF bytes are an easy way to identify an invalid UTF-8 file. This idea doesn't have that property.
sph 2 hours ago
True, but not all non-UTF8 bytestrings contain 0xFF bytes, so it’s not very useful in practice.
da_chicken an hour ago
Yes, I agree.
It's more common for programs that say they support UTF-8 to not really do so at all. It wasn't that long ago that "UTF-8" support was often just single byte, so it was little more than ASCII. Even now it's common for programs to choke on the optional BOM. Yes, it is redundant, congratulations. The spec still explicitly allows it. Three and four byte character support is still not the best, too.
flohofwoe an hour ago
> "UTF-8" support was often just single byte, so it was little more than ASCII
"Single byte UTF-8" is ASCII. That's one of its most important properties.
> Even now it's common for programs to choke on the optional BOM
And they should... BOMs (and especially the hilarious UTF-8 BOM) are strictly a legacy Microsoft/Windows thing and should be abolished along with "extended" 8-bit ASCII encodings and UCS-2/UTF-16 (only UTF-32 makes sense, but should only be used at runtime to allow random access on UNICODE code points, but not for data exchange.
flohofwoe an hour ago
It's still a joy to see how frigging elegant and extensible the UTF-8 specification is. And even without the esoteric 0xFF lead byte, the regular UTF-8 encoding with a 0xFE lead byte (11111110) would still have plenty of headroom (36 bits) compared to the current 21 bits for UNICODE.
beeforpork 41 minutes ago
As are FE, FD, FC, FB, FA, F9, F8, F7, F6 and F5.
mrlonglong an hour ago
I love it.
Some day we'll need this when we finally realise we are not alone in the universe. Alien glyphs ftw.
jibal an hour ago
The letter from Ken Thompson is spot on:
"your first 2 extensions (5 and 6 bytes) were clearly envisioned. the standard (up to 4 bytes) was created to cover the size of unicode. i thought any more description would be a waste of paper. i think your extension from 7 to 8 bytes is a little hoaky. i requires reading the whole string rather than "knowing" the number of follow on bytes. so, i think the only thing new is the 7 byte version.
i appreciate the mail, but i really dont think it is useful. it is like replacing ipv6 with ipv50."
flohofwoe 2 hours ago
Phew, and I was worried that we'd be running out of UNICODE space for new emojis ;)
beeforpork 36 minutes ago
It would be nice if the first paragraph gave a motivation for this. Why did you start this?
I mean, I can derive from the text that this can encode arbitrarily large integers. So what? What's the point? I also understand that this is for fun.
Grimeton an hour ago
More like WTF-8.
Sharlin an hour ago
No, WTF-8 is its own encoding: https://wtf-8.codeberg.page/
snvzz 2 hours ago
No project is ever safe from complicators.
This is why we need the KISS enforcers.
flohofwoe an hour ago
But the nice thing about UTF-8 is that this proposal isn't really a "complication", but a minimal and natural extension of the original idea to allow more than one lead byte.