More Unicode support

Version 4.05 adds more support for Unicode, in particular UTF-8 encoded strings.

1. A check of the code revealed that there were various places where the UTF-8 flag was not honoured when compiling regular expressions. These have been fixed.
   
2. UTF-8 support was inadvertently removed from version 4.03 - it is now back in.
   
3. You can specify UTF-8 codes by using regular expressions, for example:
   
   
match: ^\x{198}\x{19c}$

   
   This matches the (hex) Unicode characters 0198 and 019c.
   
4. For testing triggers there is now a "Insert Unicode" button in the "Debug Simulated World Output" dialog. This lets you easily insert UTF-8 sequences into your debug strings. For example, by using that I quickly converted the characters 0198 and 019C into \C6\98\C6\9C for testing the trigger.
   
5. New script routines for Lua scripting:
   
   
   • utf8encode - encodes a sequence of Unicode numbers into UTF-8.
     
   • utf8decode - decodes a UTF-8 string into its appropriate Unicode numbers
     
   • utf8valid - checks if a UTF-8 string is valid
   
   

It is difficult for me to extensively test these, as I generally use the "normal" Western characters, but these changes seem to be a step in the right direction.

You can look up Unicode characters at:

www.unicode.org

For example, this page has charts for various languages:

http://www.unicode.org/charts/

I should also point out that although the Unicode Insertion tool which has been recently added to MUSHclient purports to support Unicode characters in the range (hex) 0 to 7FFFFFFF, there are a couple of restrictions on that:

• Unicode support in Windows is, as far as I know, implemented as unsigned 16-bit characters, so the maximum practical size is FFFF (65535 in decimal).
  
• According to www.unicode.org the largest code point in the UTF-8 encoding is hex 10FFFF.

The vast majority of Unicode characters appear in the range 0000 to FFFF, so this is not a major restriction. It appears that some of the more obscure scripts (for example, Byzantine Musical Symbols, or Old Persian) require characters in the higher range.

You would probably have some difficulty in finding a typeface that would represent such characters on the screen.

Version 4.06 modifies this behaviour a little bit.

I thought it would be nice to have utils.utf8encode and utils.utf8decode works symmetrically, so I have replaced the call to the Windows MultiByteToWideChar function with a hand-written function that actually handles all numbers in the range 0 to 7FFFFFFF. The largest numbers take 6 bytes to store a single number. For example, the number 0x7FFFFFFF is represented as \FD\BF\BF\BF\BF\BF.

Also, the function utils.utf8valid returns the length of a UTF-8 string (if it is valid) rather than just true, so that you can use it to find the length of a UTF-8 sequence.

Also there is a minor problem in version 4.05, in that you cannot encode 0x00 because that was used as a string terminator. The routines have been reworked so that even 0x00 can be encoded / decoded.

Validity tests

The test for validity of UTF-8 is fairly rigorous. First, let's look at the UTF-8 possible bit patterns:



Thus, sequence should follow the general pattern above. However there is one more requirement, namely that each Unicode point should be encoded in one and only one way. Thus, this is not a valid way of encoding the number 1:



Although it follows the general pattern (2nd item from the top), the number 1 should be encoded as:



This restriction is to stop people "slipping through" things that are not valid (for example, swear words) by encoding them in non-standard ways.

Here are (hex) examples of ways UTF-8 strings might fail:

• C0 --> too short, needs another byte
• C080 --> could have been encoded in a shorter way (00)
• C0C0 --> 11xxxxxx must be followed by 10xxxxxx
• C000 --> ditto
• 80 --> pattern 10xxxxxx must be preceded by 11xxxxxx
• FDBFBFBFBF --> too short
• FC80BFBFBFBF --> could have been encoded in a shorter way (F0BFBFBF)
• FE --> the byte FE can never occur in UTF-8 (11111110)
• FF --> the byte FF can never occur in UTF-8 (11111111)