Unicode and mbstring: Bytes, Characters, and Safe Text Handling

Understand the distinction between bytes and characters in UTF-8 text, use the mbstring extension (mb_strlen, mb_substr, mb_strtolower, and related functions) to operate correctly on multi-byte strings, and handle encoding detection, normalization, and safe truncation without corrupting user-visible text.

PHP's native string functions -- strlen(), substr(), strtoupper() -- operate on raw bytes, not characters, which works perfectly for ASCII text (where one byte is always one character) but silently breaks on anything containing accented letters, non-Latin scripts, or emoji, all of which UTF-8 represents using more than one byte per character. This module covers what UTF-8 actually is at the byte level, the difference between bytes, code points, and grapheme clusters, and how the mbstring ('multi-byte string') extension gives you character-aware equivalents of PHP's standard string functions -- along with normalization, encoding detection, and the specific discipline of truncating user-visible text without splitting a character in half.

Bytes, Code Points, and Grapheme Clusters

UTF-8 is a variable-length encoding: it represents each Unicode code point using between 1 and 4 bytes. ASCII characters (English letters, digits, basic punctuation) use exactly 1 byte each, and UTF-8 was deliberately designed so those bytes are identical to plain ASCII -- which is why UTF-8 text that happens to be pure English looks and behaves identically to old-fashioned ASCII text. But an accented letter like 'é' takes 2 bytes, many CJK (Chinese/Japanese/Korean) characters take 3 bytes, and characters outside the Basic Multilingual Plane -- including most emoji -- take 4 bytes. This is the entire reason strlen('café') returns 5 in PHP, not 4: the 'é' contributes 2 bytes to the byte count, even though a human reading the word sees 4 characters.

A code point is a single Unicode-assigned numeric identifier for a character (U+0041 for 'A', U+00E9 for 'é', U+1F600 for a grinning-face emoji). Encoding is what turns a sequence of code points into actual bytes for storage or transmission (UTF-8 being the encoding used almost everywhere on the modern web); mb_strlen() counts code points correctly regardless of how many bytes each one takes in the chosen encoding.

A grapheme cluster is the layer above even code points: what a human actually perceives as 'one character' on screen. Most of the time one code point is one grapheme, but not always -- an emoji with a skin-tone modifier, a flag made of two regional-indicator code points, or a base letter combined with a separate accent mark (like 'e' followed by a combining acute accent, rather than the single precomposed 'é' code point) can all be multiple code points that together render as a single visual glyph. mb_strlen() with the default byte-oriented mode counts code points, not grapheme clusters, so for text that may contain combining characters or complex emoji, PHP's separate intl extension (with grapheme_strlen() and friends) is the tool built specifically for grapheme-aware counting -- a distinction that matters for genuinely correct 'character count' displays involving combining marks or compound emoji.

• UTF-8 is variable-length: 1 byte for ASCII, up to 4 bytes for characters outside the Basic Multilingual Plane (many emoji included). • strlen() counts bytes; on non-ASCII UTF-8 text, byte count and human-perceived character count diverge. • A code point is Unicode's numeric identity for a character; mb_strlen() correctly counts code points, not raw bytes. • A grapheme cluster is what a human sees as 'one character' -- sometimes made of multiple code points (combining marks, complex emoji). • For true grapheme-aware operations (not just code-point-aware), PHP's intl extension (grapheme_strlen(), etc.) is the correct tool.

mbstring Functions: The Character-Aware Equivalents

mb_strlen($string, $encoding = 'UTF-8') returns the number of characters (code points) in a string under the given encoding, rather than the number of raw bytes strlen() would return -- mb_strlen('café') correctly returns 4. mb_substr($string, $start, $length) extracts a substring by character position rather than byte position, so mb_substr('café', 0, 3) correctly returns 'caf' (three whole characters) instead of risking a byte offset that lands in the middle of the 2-byte 'é'. mb_strtolower() and mb_strtoupper() correctly case-fold characters outside the ASCII range (like 'É' to 'é'), which strtolower()/strtoupper() -- being byte-oriented and originally locale-dependent for the ASCII range only -- do not reliably handle for non-ASCII letters.

Most mbstring functions accept an explicit $encoding parameter, but when omitted they fall back to the internal encoding configured via mb_internal_encoding() (which should be set to 'UTF-8' for any modern application, and is UTF-8 by default in current PHP versions). Passing the encoding explicitly at each call site is more verbose but removes any dependence on ambient configuration -- valuable in a library that might run inside an application whose internal encoding was changed by something else entirely.

mb_str_split($string, $length = 1) splits a string into an array of individual characters (or fixed-length chunks of characters) the way str_split() does for bytes, and is the character-safe way to iterate 'one character at a time' over text that might contain multi-byte characters -- iterating str_split()'s byte-level output over UTF-8 text will split multi-byte characters across array entries, corrupting them.

• mb_strlen($s) -- character (code point) count, not byte count. • mb_substr($s, $start, $length) -- extracts by character position, never splitting a multi-byte character. • mb_strtolower() / mb_strtoupper() -- correctly case-fold non-ASCII letters (é, ñ, etc.), unlike the byte-oriented originals. • mb_internal_encoding() sets the default encoding mbstring functions fall back to when $encoding is omitted -- keep it 'UTF-8'. • mb_str_split($s) -- splits into an array of whole characters, safe for multi-byte text (str_split() splits by raw byte and corrupts multi-byte characters).

Encoding Detection and Normalization

Not all text a PHP application receives is guaranteed to already be valid UTF-8 -- a file upload, a legacy database column, or an external API can hand you bytes in a different encoding (ISO-8859-1/Latin-1, Windows-1252, Shift-JIS) or, worse, bytes that are simply invalid as any encoding due to corruption or a bug upstream. mb_detect_encoding($string, $encodingList) attempts to guess which of the listed candidate encodings the byte sequence is valid as, returning the first one that fits (or false if none do) -- it's a heuristic, not a guarantee, and is far more reliable when the actual source's likely encoding is known and passed explicitly as a short candidate list rather than mbstring's very broad default list.

mb_check_encoding($string, 'UTF-8') answers a narrower, more reliable question: is this byte sequence valid UTF-8, yes or no? This is the right tool for validating that user-submitted or externally-sourced text is safe to treat as UTF-8 before doing anything else with it -- storing invalid byte sequences in a UTF-8 database column, or feeding them to functions expecting valid UTF-8, is a common source of mangled-looking text (or outright errors) surfacing much later, far from where the bad bytes actually entered the system.

Normalization addresses a different problem: the same visual character can often be represented by more than one distinct sequence of code points -- 'é' can be one precomposed code point (U+00E9), or 'e' (U+0065) followed by a separate combining acute accent (U+0301), and these two byte sequences are not equal as strings even though they render identically and mean the same thing to a human. Unicode defines several normalization forms (NFC, NFD, NFKC, NFKD) that convert text into one canonical representation, which matters enormously for correctness whenever you compare, search, deduplicate, or hash user-entered text -- two 'identical-looking' usernames or search terms can silently fail an == comparison if one arrived pre-composed and the other decomposed. PHP's intl extension provides Normalizer::normalize($string, Normalizer::FORM_C) for this; mbstring itself does not perform normalization.

• mb_detect_encoding() is a heuristic guess among candidate encodings -- more reliable with a short, specific candidate list than a broad default one. • mb_check_encoding($s, 'UTF-8') validates that a byte sequence is well-formed UTF-8 -- do this before trusting external/uploaded text as UTF-8. • The same visual character can have multiple valid Unicode representations (precomposed vs. combining-mark sequences). • Normalization (NFC/NFD/NFKC/NFKD, via the intl extension's Normalizer class) converts text to one canonical form before comparison/storage. • Skipping normalization can make visually-identical strings fail equality checks, breaking search, deduplication, and uniqueness constraints.

Unicode-Aware Regex and Safe Truncation

PCRE patterns need the u modifier to treat both the pattern and the subject string as UTF-8 rather than raw bytes -- without it, a character class like [À-ÿ] or a quantifier applied to \X (a full grapheme cluster) will misinterpret multi-byte sequences as multiple separate single-byte characters, matching incorrectly or, on genuinely invalid UTF-8, failing the whole match outright. With the u modifier, \d, \w, and \s also become Unicode-aware by default (matching digits/letters from any script, not only ASCII 0-9/a-z), and a shorthand like \p{L} becomes available to match 'any Unicode letter' from any script, which is essential for validating names or text in a genuinely international application rather than one that silently only supports English.

Truncating user-visible text safely -- for previews, summaries, or fixed-width display -- means cutting at a character boundary, never a byte boundary. Calling substr($text, 0, 100) on UTF-8 text can land its cut point in the middle of a multi-byte character's bytes, producing an invalid, often visibly broken UTF-8 sequence (commonly rendered as replacement-character mojibake, like a diamond with a question mark) right at the cut point. mb_substr($text, 0, 100) instead counts 100 whole characters and can never split one apart, which is why it -- not substr() -- is the correct function for any user-facing truncation.

Truncation by character count still isn't the same as truncation by rendered width or by grapheme count when text contains combining marks or complex emoji, but for the overwhelming majority of real-world UTF-8 text (which is composed of precomposed, single-code-point characters), mb_substr() gets truncation right where substr() actively corrupts it -- making it the essential, non-negotiable baseline for any function that shortens text a user will actually read.

• The /u modifier makes a PCRE pattern (and \d/\w/\s within it) Unicode-aware instead of byte-oriented. • \p{L} (with /u) matches any Unicode letter from any script -- essential for validating international names/text. • substr() truncates by byte offset and can split a multi-byte character in half, producing corrupted/invalid UTF-8. • mb_substr() truncates by character count, always landing on a whole-character boundary. • Even mb_substr() truncation isn't grapheme-cluster-perfect for combining marks/complex emoji, but it is correct for the vast majority of real text and strictly safer than byte-based substr().

Character-Safe Length and Substring

strlen()/substr() are byte-oriented -- correct only for pure-ASCII text. mb_strlen()/mb_substr() are character-oriented -- correct for any UTF-8 text, ASCII included. Set mb_internal_encoding('UTF-8') once, early, if relying on the default-encoding fallback.

$text = 'café';

echo strlen($text) . "\n";      // 5 -- BYTE count ('é' is 2 bytes in UTF-8)
echo mb_strlen($text) . "\n";   // 4 -- CHARACTER count (correct)

echo substr($text, 0, 3) . "\n";    // may cut mid-character -- unsafe
echo mb_substr($text, 0, 3) . "\n"; // 'caf' -- always a whole-character boundary

Code Examples

For pure-ASCII 'hello', strlen() and mb_strlen() agree (5 and 5), because every character is exactly one byte. For every string containing an accented letter or a CJK character, the byte count strlen() reports is larger than the actual number of characters a person reading the text would count -- '東京' is 2 characters but 6 bytes (3 bytes each in UTF-8), which strlen() reports as 6. Expected result: café bytes=5 chars=4 naïve bytes=6 chars=5 東京 bytes=6 chars=2 jalapeño bytes=9 chars=8 hello bytes=5 chars=5 strlen($name), mb_strlen($name) safePreview() counts and cuts by character, so the 12-character truncation of 'Le café était...' always lands on a whole character regardless of how many bytes the accented letters take. The raw substr(..., 0, 12) call below it cuts at exactly byte 12, which can (depending on where accents fall) slice a multi-byte character's bytes apart, producing an invalid, visibly broken tail character. The exact broken output of the byte-based version depends on where the cut byte offset falls relative to a multi-byte character's byte boundaries -- sometimes it's fine by luck, but it's never guaranteed to be, which is the whole problem.

return mb_substr($text, 0, $maxChars) . '…';

mb_check_encoding() rejects malformed byte sequences before anything else touches them. Normalizer::normalize(..., FORM_C) then converts both the precomposed and the decomposed representations of the same visual name to the identical canonical byte sequence, so a direct === comparison (or a database UNIQUE constraint, or a search index) treats them as the same name -- which they are, to any human reading them. Expected result: bool(false) bool(true)

$normalized = Normalizer::normalize($raw, Normalizer::FORM_C);

\p{L} matches a letter from ANY Unicode script -- Latin, accented Latin, CJK, and beyond -- which is what lets 'José García' and '田中太郎' both validate correctly, unlike an ASCII-only [A-Za-z] class that would reject every accented or non-Latin name outright. The /u modifier is what enables \p{L} at all and ensures the multi-byte characters in the subject string are interpreted as whole characters, not raw bytes, while matching. Expected result: Grace Hopper: valid José García: valid 田中太郎: valid O'Brien: valid user_123: invalid return preg_match('/^[\p{L} \'-]{1,50}$/u', $name) === 1;

Common mistake: Using [A-Za-z] (no \p{L}, no /u) to validate a 'name' field -- it silently rejects every legitimately named user whose name contains an accented letter or a non-Latin script, which in a genuinely international application is a real, user-facing bug, not an edge case.

<?php
declare(strict_types=1);

mb_internal_encoding('UTF-8');

$names = ['café', 'naïve', '東京', 'jalapeño', 'hello'];

foreach ($names as $name) {
    printf(
        "%-10s bytes=%d  chars=%d\n",
        $name,
        strlen($name),
        mb_strlen($name),
    );
}

<?php
declare(strict_types=1);

function safePreview(string $text, int $maxChars): string
{
    if (mb_strlen($text) <= $maxChars) {
        return $text;
    }

    return mb_substr($text, 0, $maxChars) . '…';
}

$article = 'Le café était délicieux et l’ambiance était formidable.';

echo safePreview($article, 12) . "\n";
// Compare with the byte-based (unsafe) version below, which can corrupt a character:
echo substr($article, 0, 12) . "\n";

<?php
declare(strict_types=1);

function sanitizeSubmittedName(string $raw): string
{
    if (!mb_check_encoding($raw, 'UTF-8')) {
        throw new InvalidArgumentException('Name is not valid UTF-8');
    }

    // Normalize to NFC so visually-identical names compare/hash consistently,
    // regardless of whether the client sent precomposed or decomposed code points.
    $normalized = Normalizer::normalize($raw, Normalizer::FORM_C);
    if ($normalized === false) {
        throw new InvalidArgumentException('Name failed Unicode normalization');
    }

    return mb_substr(trim($normalized), 0, 100); // enforce a character-safe max length
}

// 'é' as one precomposed code point vs. 'e' + combining acute accent -- visually identical:
$precomposed = "Jos\u{00E9}";
$decomposed  = "Jose\u{0301}";

var_dump($precomposed === $decomposed); // false -- different byte sequences
var_dump(
    sanitizeSubmittedName($precomposed) === sanitizeSubmittedName($decomposed)
); // true -- normalization made them equal

<?php
declare(strict_types=1);

function isValidDisplayName(string $name): bool
{
    // \p{L} matches any Unicode letter from any script; /u makes the pattern
    // and subject Unicode-aware instead of byte-oriented.
    return preg_match('/^[\p{L} \'-]{1,50}$/u', $name) === 1;
}

foreach (['Grace Hopper', 'José García', '田中太郎', 'O\'Brien', 'user_123'] as $name) {
    $ok = isValidDisplayName($name) ? 'valid' : 'invalid';
    echo "{$name}: {$ok}\n";
}

One Character, Multiple Layers: 'é' in UTF-8

Normalization must happen before comparison or counting -- otherwise two visually-identical strings can disagree at every layer.

Grapheme cluster:  'é'                (what a person sees: ONE character)
                     |
Code point(s):     U+00E9                (ONE Unicode code point, precomposed form)
                     |                    -- OR, decomposed form --
                   U+0065 + U+0301        ('e' + combining acute accent: TWO code points)
                     |
UTF-8 bytes:       0xC3 0xA9              (precomposed: 2 bytes)
                     |                     -- vs --
                   0x65 0xCC 0x81         (decomposed: 3 bytes total)

strlen()      counts BYTES        -> 2 (precomposed) or 3 (decomposed) -- inconsistent!
mb_strlen()   counts CODE POINTS  -> 1 (precomposed) or 2 (decomposed) -- still inconsistent!
Normalizer    -> canonical form FIRST, then both representations agree.

Important Rules

Never use strlen()/substr()/strtolower()/strtoupper() on text that might contain non-ASCII characters -- use the mb_* equivalents.

Set mb_internal_encoding('UTF-8') early (or always pass $encoding explicitly) so mbstring functions behave predictably.

Validate external/uploaded text with mb_check_encoding($s, 'UTF-8') before trusting it as UTF-8.

Normalize text (Normalizer::normalize(..., Normalizer::FORM_C)) before comparing, deduplicating, or uniquely constraining user-entered strings.

Always truncate user-visible text with mb_substr(), never substr(), to avoid splitting a multi-byte character.

Add the /u modifier (and prefer \p{L}/\p{N} over [A-Za-z0-9]) to any regex that validates genuinely international text.

Why This Matters in Production

Byte-vs-character bugs are among the most common, most quietly-shipped defects in internationalized applications: a 'first name' field that silently truncates or rejects legitimate international names, a search feature where a precomposed and a decomposed spelling of the same name don't match each other, or a text preview that ends every so often in a visibly broken character glyph right at the cut point. None of these show up in testing done only with English test data, which is exactly why they slip past code review and QA so often -- and why explicitly testing with accented, CJK, and emoji strings (not just 'hello world') is a genuinely important habit for any application with real, international users.

$preview = substr($comment, 0, 100); // byte-based truncation on user-submitted UTF-8 text

$preview = mb_substr($comment, 0, 100); // character-based truncation, never splits a multi-byte character substr() cuts at an exact byte offset, which can land in the middle of a multi-byte UTF-8 character (2 to 4 bytes long) whenever the cut point happens to fall inside one -- producing an invalid byte sequence that typically renders as a broken glyph right at the truncation point. mb_substr() counts whole characters, so the cut always falls on a valid character boundary.

if (strlen($username) > 20) { throw new InvalidArgumentException('Username too long'); } // rejects perfectly reasonable non-ASCII usernames if (mb_strlen($username) > 20) { throw new InvalidArgumentException('Username too long'); } // measures actual character count

A username like 'José García' is 11 characters but 12 bytes; a longer non-Latin name can diverge even further between byte count and character count. Using strlen() for a 'max 20 characters' rule effectively enforces a stricter (and unfair) limit on any user whose name contains non-ASCII characters, while ASCII-only names get the full intended limit.

[multiple-choice] Why does strlen('café') return 5 instead of 4? A. strlen() has a bug with accented letters B. strlen() counts raw bytes, and 'é' takes 2 bytes to encode in UTF-8, so the byte count (5) is one more than the actual character count (4) C. The string literal itself contains a hidden trailing space D. PHP strings are always padded to an even byte length

B. strlen() counts raw bytes, and 'é' takes 2 bytes to encode in UTF-8, so the byte count (5) is one more than the actual character count (4) — UTF-8 is a variable-length encoding -- ASCII characters take 1 byte, but 'é' takes 2 bytes. strlen() is byte-oriented, so it reports 5 total bytes (3 ASCII letters at 1 byte each + 'é' at 2 bytes), even though a human reading 'café' sees exactly 4 characters.

[multiple-choice] Why is mb_substr() the correct choice over substr() for truncating user-facing text? A. mb_substr() is simply faster in every case B. substr() cuts at a raw byte offset and can split a multi-byte character in half, producing invalid/corrupted UTF-8; mb_substr() cuts by character count, always landing on a whole-character boundary C. substr() cannot be used on strings longer than 255 bytes D. mb_substr() automatically translates the text

B. substr() cuts at a raw byte offset and can split a multi-byte character in half, producing invalid/corrupted UTF-8; mb_substr() cuts by character count, always landing on a whole-character boundary — Because UTF-8 characters can span multiple bytes, an arbitrary byte offset from substr() may fall inside one of those multi-byte sequences, breaking it. mb_substr() operates on character (code point) positions instead, so it can never produce that kind of corruption.

[multiple-choice] Why might Normalizer::normalize() be necessary even after confirming two strings are both valid UTF-8? A. Valid UTF-8 guarantees the same character is always represented by the identical byte sequence B. The same visual character (e.g. 'é') can be validly represented by different code point sequences (one precomposed code point, or a base letter plus a combining accent mark) -- these are both valid UTF-8 but not byte-for-byte equal, so a direct comparison can wrongly treat identical-looking text as different C. Normalization is only needed for non-UTF-8 encodings D. mb_check_encoding() already performs normalization internally

B. The same visual character (e.g. 'é') can be validly represented by different code point sequences (one precomposed code point, or a base letter plus a combining accent mark) -- these are both valid UTF-8 but not byte-for-byte equal, so a direct comparison can wrongly treat identical-looking text as different — Unicode allows more than one valid code point sequence for many visually identical characters (precomposed vs. combining-mark decomposition). Both are valid UTF-8, but they are different byte sequences -- only normalizing to a single canonical form (like NFC) before comparing, searching, or enforcing uniqueness makes such strings correctly equal.

Write a function truncateWords(string $text, int $maxChars): string that safely truncates $text to at most $maxChars characters (using mb_substr, never substr), and if truncation actually occurred, trims any trailing partial word and appends an ellipsis ('…') so the result never ends mid-word.

First check mb_strlen($text) against $maxChars to see if truncation is even needed. If it is, take mb_substr($text, 0, $maxChars), then use mb_strrpos() to find the last space in that substring and cut there before appending the ellipsis.