Unicode and Encodings

The golden rule: decode input at the boundaries (files, network, database), work with character strings internally, and encode output at the boundaries.

Common bugs: "Wide character in print" (output not encoded), double encoding (é mojibake), forgetting use utf8 for literal non-ASCII source.

perldoc perlunicode, perlunitut, perlunifaq, and Tom Christiansen's "Unicode in Perl" talks are essential reading.

use v5.36;
use utf8;                                   # the source code itself is UTF-8
use open qw(:std :encoding(UTF-8));         # STDIN/STDOUT/STDERR and default open layers use UTF-8

my $s = 'naïve café ☕';
say length $s;                              # characters, not bytes
say uc $s;
say scalar reverse $s;

use Encode qw(encode decode);
my $bytes = encode('UTF-8', $s);            # characters → bytes
my $chars = decode('UTF-8', $bytes);        # bytes → characters
say length $bytes;                          # byte count
use Encode qw(encode_utf8 decode_utf8);

say "\N{U+1F600}";                          # character by code point
say "\N{SNOWMAN}";                          # character by name (charnames loads automatically since 5.16)
say sprintf 'U+%04X', ord 'é';
say 'letters only' if $s =~ /^[\p{L}\s☕]+$/;

use Unicode::Normalize qw(NFC NFD);
say NFC("e\x{301}") eq "\x{e9}" ? 'same after NFC' : 'different';
say fc('STRASSE') eq fc('straße') ? 'equal (casefold)' : 'not equal';

use Unicode::GCString;                      # grapheme-aware length (CPAN)
my @graphemes = "👨‍👩‍👧e\x{301}" =~ /\X/g;   # \X matches an extended grapheme cluster
say scalar @graphemes;            
…