More specifically, Unicode is one of those things (like bignum support) where Lua's design for embedding assumes that if it's a priority, your project will already have settled on a library for it, so anything Lua came bundled with would just get in the way. The GMP library, for example, is almost four times larger than the whole Lua runtime.
Besides, Lua strings are interned raw byte-arrays (like atoms in Erlang or Lisp atoms), and can have \0s and other arbitrary binary data. While the standard string library is ignorant of Unicode, there's nothing preventing extra libraries from providing Unicode-aware string operations.
Lua also left out native regular expression support, because it would be larger than the rest of the standard libraries put together. You can just load LPEG if it matters.
I wouldn't have a problem with external unicode support if there was one "standard" recommended library that had a strong community behind it. My whole point is that that doesn't appear to exist.
Sure. This is a real issue for new Lua users, though in the long run it's not a showstopper.
For roughly half the people using Lua, library choices have already been decided by the big C++ (or whatever) project they're adding Lua to, so the question never arises in the first place. That's the primary use case for Lua. It's a really nice standalone language, too, but it was designed for embedding. Many of its strengths (such as the small, orthogonal, clean design) come from that focus, but so do some quirks. If you use Lua as a standalone language, you'll occasionally need to do some digging for libraries, particularly if you're not comfortable with C. The situation has been improving significantly over the last year or two, though.
I don't have a specific Unicode library recommendation, but would suggest checking the mailing list archives (http://lua-users.org/lists/lua-l/).
Besides, Lua strings are interned raw byte-arrays (like atoms in Erlang or Lisp atoms), and can have \0s and other arbitrary binary data. While the standard string library is ignorant of Unicode, there's nothing preventing extra libraries from providing Unicode-aware string operations.
Lua also left out native regular expression support, because it would be larger than the rest of the standard libraries put together. You can just load LPEG if it matters.