Editor's note: Perl guru Tom Christiansen created and maintains a list of 44 recipes for working with Unicode in Perl 5. Perl.com is pleased to serialize this list over the coming weeks.

? 17: Make ?le I/O default to utf8

If you've ever had the misfortune of seeing the Unicode warning "wide character in print", you may have realized that something forgot to set the appropriate Unicode-capable encoding on a filehandle somewhere in your program. Remember that the rule of Unicode handling in Perl is "always encode and decode at the edges of your program".

You can easily Decode STDIN, STDOUT, and STDERR as UTF-8 by default or Decode STDIN, STDOUT, and STDERR per local settings as a default, or you can use binmode to set the encoding on a specific filehandle.

Alternately, you can set the default encoding on all filehandles through the entire program, or on a lexical basis. As documented in perldoc perlrun, the -C flag and the PERL_UNICODE environment variable are available. Use the D option to make all filehandles default to UTF-8 encoding. That is, files opened without an encoding argument will be in UTF-8:

     $ perl -CD ...
     # or
     $ export PERL_UNICODE=D

The open pragma configures the default encoding of all filehandle operations in its lexical scope:

     use open qw(:utf8);

Note that the open pragma is currently incompatible with the autodie pragma.

Editor's note: Perl guru Tom Christiansen created and maintains a list of 44 recipes for working with Unicode in Perl 5. Perl.com is pleased to serialize this list over the coming weeks.

? 16: Declare STD{IN,OUT,ERR} to be in locale encoding

Always convert to and from your desired encoding at the edges of your programs. This includes the standard filehandles STDIN, STDOUT, and STDERR. While it may be most common for modern operating systems to support UTF-8 in filehandle settings, you may need to use other encodings.

Perl can respect your current locale settings for its default filehandles. Start by installing the Encode::Locale module from the CPAN.

    # cpan -i Encode::Locale
    use Encode;
    use Encode::Locale;

    # or as a stream for binmode or open
    binmode STDIN,  ":encoding(console_in)"  if -t STDIN;
    binmode STDOUT, ":encoding(console_out)" if -t STDOUT;
    binmode STDERR, ":encoding(console_out)" if -t STDERR;

The Encode::Locale module allows you to use "whatever encoding the attached terminal expects" for input and output filehandles attached to terminals. It also allows you to specify "whatever encoding the file system uses for file names"; see the documentation for more.

Editor's note: Perl guru Tom Christiansen created and maintains a list of 44 recipes for working with Unicode in Perl 5. Perl.com is pleased to serialize this list over the coming weeks.

? 15: Declare STD{IN,OUT,ERR} to be UTF-8

Always convert to and from your desired encoding at the edges of your programs. This includes the standard filehandles STDIN, STDOUT, and STDERR.

As documented in perldoc perlrun, the PERL_UNICODE environment variable or the -C command-line flag allow you to tell Perl to encode and decode from and to these filehandles as UTF-8, with the S option:

     $ perl -CS ...
     # or
     $ export PERL_UNICODE=S

Within your program, the open pragma allows you to set the default encoding of these filehandles all at once:

     use open qw(:std :utf8);

Because Perl uses IO layers to implement encoding and decoding, you may also use the binmode operator on filehandles directly:

     binmode(STDIN,  ":utf8");
     binmode(STDOUT, ":utf8");
     binmode(STDERR, ":utf8");