Sure, in some cases it makes sense to make the tradeoff of not handling more obscure characters. But if the tradeoff is made, the encoding should not be called UTF-8.
"UTF-8 (UCS Transformation Format—8-bit[1]) is a variable-width encoding that can represent every character in the Unicode character set," says Wikipedia. The UTF-8 implementation in MySQL does not meet this definition because it cannot represent every character in the Unicode character set.
When MySQL first implemented UTF-8 they probably did support every Unicode character... because there were less than 64K Unicode characters. Then Unicode/UTF-8 was redefined out from under them.
> there were less than 64K Unicode characters. Then Unicode/UTF-8 was redefined out from under them.
Unicode 2.0 introduced multiple planes, i.e. more than 65536 characters. That was in 1996. If that was the case, then MySQL has had more than one-and-a-half decades to introduce multiple planes and seems to have done so less than a year ago. I disagree with being 'redefined out from under them', when it was defined a year after MySQL started, at a time when it probably didn't even have Unicode support yet anyway.
Interesting to note is that MySQL was first released in 1995. Which means that only for one year of its existence were there less than 65536 characters.
Yes, but at the time, UTF-8 could encode up to 31 bits per character using six-byte sequences. It has since been restricted to four-byte sequences at the longest.
> But when did people really start using more than the 16bit unicode chars?
1996.
China even made it a legal requirement for computer systems in 2000, through mandating GB 18030.
There's the Private Use Area if nothing else. There is NO excuse to not support anything other than the BMP. Adding support is trivial unless you have been using UTF-16 in the erroneous belief that it's two bytes long always (in which case you've really been using UCS-2).
According to Wikipedia, the original version of UTF-8 supported >4 byte characters, and was later restricted to 4 bytes by RFC 3629 in November 2003, seven months before MySQL 4.1 was released with Unicode support. (There were 96,447 Unicode characters at that time.)
"UTF-8 (UCS Transformation Format—8-bit[1]) is a variable-width encoding that can represent every character in the Unicode character set," says Wikipedia. The UTF-8 implementation in MySQL does not meet this definition because it cannot represent every character in the Unicode character set.