Overview

A Little Background About Character Sets

During the 1980s other expanded character sets were developed to allow larger sets of characters in a single set. Work focused on 8-bit character sets with 256 characters in them. Unfortunately, many parts of the BSD system were coded with assumptions that a character was 7 bits, so a major effort was undertaken during the late 1980s and 1990s to make the system `8-bit clean'. This involved changes to the system itself and to many utilties. The result was that FreeBSD evolved to use the 8-bit ISO8859 character set, which is still the default today. In ISO8859, the 8 bits allow for codes from 0..255 or hex 0x0..0xff. The US-ASCII characters occupy the same 0x0..0x7f positions and an additional selection of common western European accent characters are on the 0x80..0xff positions. At the time, this was a great improvement as documents could now be written in many more languages. Unfortunately however, 128 extra western European characters doesn't by any means include all European characters and is therefore not sufficient even for many Europeans (it lacks eastern European characters, for example, not to mention Greek). As a result, several alternate ISO8859 character sets were developed, known as ISO8859-1, ISO8859-2, ISO8859-3 etc, each with a different set of characters in the 0x80-0xff range. ISO8859-1 contains western European characters, ISO8859-2 contains eastern European characters, and there are variants with Cyrilic, Greek, Arabic and Hebrew characters too. A document or message, however, can only use characters from one character set, so combining Hebrew with French in the same document would still pose a problem. And, ISO8859 doesn't handle Asian character sets or languages which have more than 256 characters in general use and which therefore won't fit in ISO8859's 8-bit range.

In today's increasingly global world, it is becoming more and more common to have multi-lingual documents and to want to share them with folk all over the world. In fact, a simple list of peoples' names often poses a problem if those names would normally be written in differing character sets. It's also common to exchange messages and documents with people from many parts of the world, and while multi-lingual documents might not be necessary all the time, having to continually switch character sets for work with different people becomes tedious and annoying. The expanded 8-bit ISO8859 character sets don't solve the problem. A third evolution of character sets has therefore taken place.

Unicode encodes characters as multi-byte values, supporting a much larger range of characters. Encoding of all the world's languages is supported within Unicode, as are various specialized characters such as mathematical characters, music symbols, Braille and so on. Unicode isn't a merely an extension from 8-bit to 16-bit character set, but instead allows for characters with 24-bits or 32-bits or more. Even within Unicode, there are alternative possible character encodings, but one such encoding, UTF-8, is commonly accepted as a good base for handling most of the world's languages without excessive overhead. Unicode's UTF-8 encoding is supported on platforms such as Linux and Mac OS/X and Windows, too, which is an important aspect of making it useful for message and document interchange.

To accomodate Unicode, the system and utilities must, once more, be adapted, this time to support varying length characters. This support is known as wide character support and has been implemeted extensively throughout FreeBSD and many of the base system utilties and ports. Converting a FreeBSD system to use Unicode involves enabling this wide character support in system libraries and utilities where it is needed. It also involves telling the system to interpret characters as Unicode sequences and to convert keyboard keystrokes to Unicode character values.

Fixing an error in the UTF-8/LC_CTYPE file

There is an error in the FreeBSD distributed UTF-8/LC_CTYPE file. This error lists the sets of combining characters, that is characters that overstrike other characters, sometimes known as non-spacing characters, with a space size of 1. They need to be listed as size 0.

Fixing a bug in hexdump(1)/od(1)

The hexdump(1) and od(1) utilities have code which assumes that all characters have a width of at least 1. There's an assert check for this and hexdump/od can fail if this isn't the case. Having applied the above lcctype.diff, this assertion is no longer correct.

$LANG and $LC_CTYPE

First, set either your LANG or LC_CTYPE environment variable to a value which includes the UTF-8 encoding. A list of locales which support UTF-8 can be found using:

    $ ls -d /usr/share/locale/*.UTF-8

    $ export LANG=en_US.UTF-8

    % setenv LANG en_US.UTF-8

    $ export LC_CTYPE=en_US.UTF-8
    $ export LC_COLLATE=POSIX

You should also put these settings in your ~/.profile or ~/.login startup file. This can also be done on a system-wide basis in the /etc/profile and /etc/login files.

    # pkg_add -rv libiconv

    $ iconv -f iso8859-1 -t utf-8 file >file.new

Keyboard Input

To do this on a per-user basis, put a command such as this one in your .xinitrc or .xsession X11 startup file.

    setxkbmap "us,us(intl)" -option \
	"grp:rctrl_toggle,lv3:ralt_switch"

You can have more than two maps. If you regularly use several languages, you may want a separate map for each language. When writing in English, for example, I set my map to the us one. When writing in French or German though, I use RCtrl to toggle to the us(intl) map and toggle back to us again once I'm done. If I used additional languages, I'd add additional maps for those languages. Additional RCtrl presses could then take me to an eastern European map, or Greek, Hebrew, Cyrillic, or one of the Asian ones - whatever I needed.

You need to memorize the layout of the characters on the keyboard for each map. In the us mapping characters are typed as labeled on the US keyboard.

In the us(intl) mapping, the accent keys `, ', ", ^ and a few others are deadkeys. This means that each must be typed and followed by a suitable letter to obtain the accented version of the letter. To type an acute-e, é, for example, type ´ followed by e and to enter an a-circonflex, â, use ^ followed by a. To obtain the ASCII character for each of those accents themselves, the accent key is typed twice. In addition, a number of western European characters can be typed directly by using the RAlt key as follows:

    ~~ !¹ @˝ #¯ $£ %  ^Þ &  *˛ (˘ )° _  +÷ |¦ <ÿ
    `` 1¡ 2² 3³ 4¤ 5€ 6¼ 7½ 8¾ 9‘ 0’ -¥ =× \¬ <ÿ

         QÄ WÅ EÉ R® TÞ YÜ UÚ IÍ OÓ PÖ {« }» |¦
         qä wå eé r® tþ yü uú ií oó pö [« ]» \¬

           AÁ S§ DÐ FF GG HH JJ KK LØ :° ¨"
           aá sß dð ff gg hh jj kk lø ;¶ ´'

             ZÆ XØ C¢ VV BB NÑ Mµ <Ç >ˇ ? 
             zæ xø c© vv bb nñ mµ ,ç .˙ /¿

• 1 is entered using 1
• ! is entered using Shift-1
• ¡ is entered using RAlt-1
• ¹ is entered using Shift-RAlt-1

• € is entered using RAlt-5

The us(intl) map has the standard alphanumeric and punctuation keys in their normal positions and addditional accented characters are typed using RAlt as a shift key. However, in many maps, standard alphanumeric and punctuation keys also move to new places. You'll need to know where characters are in each mapping and a printout of each of your maps' layouts is useful.

xterm

    # cd /usr/ports/x11/xterm
    # make -DWITH_WIDE_CHARS
    # make install

XTerm Fonts

    -misc-fixed-medium-r-normal--18-120-100-100-c-90-iso10646-1
    -misc-fixed-medium-r-normal--15-140-75-75-c-90-iso10646-1
    -misc-fixed-medium-r-normal--14-130-75-75-c-70-iso10646-1

    $ grep 10646 /usr/local/lib/X11/fonts/*/fonts.dir
or (on older systems):
    $ grep 10646 /usr/X11R6/lib/X11/fonts/*/fonts.dir

Choose a suitable font, add lines such as the following to your ~/.Xdefaults file:

    xterm*vt100.font:
	-misc-fixed-medium-r-normal--15-140-75-75-c-90-iso10646-1
    xterm*vt100.utf8Fonts.font:
	-misc-fixed-medium-r-normal--15-140-75-75-c-90-iso10646-1

Prerequisite Wide-Character Libraries

libutf-8 library

Port converters/libutf-8 can be installed without changes from the ports or as a pre-compiled package:

    # pkg_add -rv libutf-8

ncursesw library

The ncurses library is part of the base system, and as of March 9th, 2007, 7-current includes wide character support as standard. Since April 7th, 2007 6-stable also includes this as standard.

However the 5.x version still uses an older version of ncurses which doesn't have wide character support. An alternate version in the ports tree, devel/ncurses does have support for wide characters. Since May 22nd, 2006, wide character support is enabled in this port by default, so it's simply necessary to build the port or install a pre-compiled package:

    # pkg_add -rv ncurses

    # cd /usr/ports/devel/ncurses
    edit the Makefile and uncoment the
	--enable-widec keyword already there
    # make
    # make install

Utilities

bash

claws

csh

cups

egrep

emacs

ex

fgrep

grep

groff/grops

A simple solution is to install the japanese/groff port which does include wide character support. However note that this is a slightly older version of groff. The system groff is renamed to _groff to ensure that the version from the port is what is used.

    # cd /usr/ports/japanese/groff
    # make
    # make install
    # mv /usr/bin/groff /usr/bin/_groff

joe

ls

mutt

For the mail/mutt 1.4.2 stable version of mutt:

    # cd /usr/ports/mail/mutt
    Edit the Makefile to change the dependency on
	ncurses.5 to ncursesw.5
    # make -DWITH_NCURSES_PORT
    # make -DWITH_NCURSES_PORT install
or
    # make -DWITH_NCURSES_PORT WITH_SGML_DOCS=no
    # make -DWITH_NCURSES_PORT WITH_SGML_DOCS=no \
	install

    # make -DWITH_MUTT_NCURSES_PORT
    # make -DWITH_MUTT_NCURSES_PORT install

    set send_charset="US-ASCII:UTF-8"
or just:
    set send_charset="UTF-8"

If your incoming mail is not displayed properly, check that you have the LANG environment variable properly set (as described above) or configure mutt:

    set charset="UTF-8"

    export TERM=xterm-color

nvi

    # cd /usr/ports/editors/nvi-devel
    edit the Makefile and add the following:

	CONFIGURE_ARGS+=        --enable-widechar
    # make
    # make install
    # mv /usr/bin/vi /usr/bin/_vi
    # mv /usr/bin/view /usr/bin/_view
    # mv /usr/bin/ex /usr/bin/_ex
    # mv /usr/bin/nvi /usr/bin/_nvi
    # mv /usr/bin/nview /usr/bin/_nview
    # mv /usr/bin/nex /usr/bin/_nex

• There's a problem using :r to read in a file or piping data through a command: whereas the :e does open a Unicode file OK, the :r or pipe read doesn't. In practice, this means if you use !!foo to pipe the buffer to a program and back, it will come back with all the UTF-8 characters broken.
• There's also a problem if vi encounters a non UTF-8 char in the input (for example if you try to edit an ISO8859 file containing 8-th bit characters). Vi then gives a "conversion error" on that line and omits the entire line. If you directly address a line with an invalid character, or worse, if the first line in the file contains such a character, vi will hang. If this happens, you'll need to start a new xterm and use "kill -9" on the hung vi process from there. It would be better if vi were to simply display the unconvertable sequences using hex notation.
• Possibly related to the above problem is that there is a Unicode character called BOM for Byte Order Mark which is often found at the start of Unicode files to indicate the specific Unicode encoding and byte ordering. In UTF-8 this is the sequence \xef\xbb\xbf or, in octal, \0357\0273\0277. Despite being a legitimate Unicode sequence, vi fails to recognize it and will hang if a file containing this sequence is opened.
• A problem exists with Unicode in filenames. If you say, for example, "$ vi euro€.txt", vi displays "euro\xe2\x82\xac.txt: new file: line 1" on its status line. While the dislpay of the filename is broken, vi does still save the file with the correct Unicode character.
• Trying to turn off an option using :set noxxx doesn't work in this version of vi. If you try, no matter what option you try to turn off, you get the message: noprint="".
• The flash option seems to be enabled from the start, and because of the problem above, cannot be disabled.
• Recover mode doesn't work. The probem here seems to simply be that on receipt of a signal, such as SIGHUP, the /var/tmp/vi.recover/recover.XXXXX mail file is not being created. If one happens to be there, recovery does actually take place.

sh

sylpheed

tcsh

vi

zsh

Apache Web Server

In the absence of such clues, apache uses the encoding specified in its configuration file as the default. Once you have converted all files with 8-bit data in them to UTF-8, you'll probably want to change this default. For the www/apache2 version, in the configuration file, /usr/local/etc/apache2/http.conf look for and change:

    AddDefaultCharset ISO-8859-1
to
    AddDefaultCharset UTF-8

    #Include etc/apache22/extra/httpd-languages.conf
to
    Include etc/apache22/extra/httpd-languages.conf

    #
    # Specify a default charset for all pages sent
    # out. This is always a good idea and opens the
    # door for future internationalisation of your
    # web site, should you ever want it. Specifying
    # it as a default does little harm; as the
    # standard dictates that a page is in iso-8859-1
    # (latin1) unless specified otherwise i.e. you
    # are merely stating the obvious. There are also
    # some security reasons in browsers, related to
    # javascript and URL parsing which encourage you
    # to always set a default char set.
    #
    AddDefaultCharset UTF-8

For UTF-8 support a font definition containing a reasonable set of characters as well as a set of suitable keyboard maps are needed.

The sysutils/jfbterm port provides a framebuffered console which does have UTF-8 support.