Unicode & Text Encoding

Every piece of text your program handles is, underneath, a sequence of numbers — understanding how Python's str maps to those numbers, and how bytes on disk map back, prevents an entire category of bugs.

A Python str is a sequence of Unicode code points — abstract numbers that identify characters, independent of how they're stored as bytes. Encoding converts a str into bytes using a specific scheme (like UTF-8); decoding does the reverse. Confusing text and bytes, or assuming an encoding without stating one, is one of the most common sources of subtle, environment-dependent bugs in real-world Python programs.

Unicode Code Points

Unicode assigns every character (letters, digits, punctuation, emoji, and more) a unique integer called a code point, conventionally written as U+XXXX in hexadecimal. There are over a million possible code points, of which a large fraction are assigned to real characters across essentially every writing system in use. ord(char) returns a single character's code point as an int; chr(number) does the reverse, building a one-character string from a code point.

ord("A")        # 65
ord("€")        # 8364
chr(65)          # 'A'
chr(0x1F600)     # '😀'  (U+1F600, GRINNING FACE)

Python's str Is a Sequence of Code Points

Since Python 3, str is fundamentally text — a sequence of Unicode code points — with no inherent byte representation until you explicitly encode it. This is a deliberate language design: it means len(), slicing, and iteration on a str operate in units of code points, not bytes, so text handling is correct regardless of what encoding the text will eventually be stored or transmitted as.

s = "café"
len(s)          # 4 -- four code points, not bytes
list(s)          # ['c', 'a', 'f', 'é']

# CPython implementation detail: internally CPython chooses among
# several fixed-width storage representations (Latin-1, UCS-2, UCS-4)
# per string based on its widest code point (PEP 393), purely as an
# optimization -- this is not part of the language's guarantees.

Text Versus Bytes: Two Different Types

str and bytes are distinct, non-interchangeable types in Python 3. A str represents text (code points); a bytes object represents raw 8-bit values (0-255) with no inherent notion of 'characters' at all. Mixing them directly — concatenating a str and a bytes, or comparing them — raises TypeError. This strictness is intentional: it forces you to be explicit about the encoding boundary between the two.

text = "hi"
data = b"hi"
type(text), type(data)     # (<class 'str'>, <class 'bytes'>)

try:
    text + data
except TypeError as e:
    print(e)   # can only concatenate str (not "bytes") to str

Encoding: str -> bytes

str.encode(encoding='utf-8', errors='strict') converts text into bytes using the named encoding. UTF-8 is the dominant, recommended default: it represents every Unicode code point, is backward-compatible with 7-bit ASCII (any ASCII text is valid UTF-8 unchanged), and uses a variable number of bytes per character (1 to 4) so common Western text stays compact.

"café".encode("utf-8")    # b'caf\xc3\xa9'  -- 'é' takes 2 bytes in UTF-8
"café".encode("utf-8")[-1]  # 169 (the second byte of the 2-byte 'é')
len("café".encode("utf-8"))  # 5 bytes for 4 characters

"hi".encode()               # b'hi' -- pure ASCII text is identical in UTF-8

Decoding: bytes -> str

bytes.decode(encoding='utf-8', errors='strict') reverses the process, interpreting a sequence of bytes as text according to the named encoding. Decoding with the *wrong* encoding either raises an error or, worse, silently produces the wrong characters (mojibake) — there is no way to recover the original encoding from the bytes alone unless it's stated or detected some other way.

data = "café".encode("utf-8")
data.decode("utf-8")     # 'café' -- round-trips correctly

# Decoding UTF-8 bytes as latin-1 doesn't error, but silently misreads them:
data.decode("latin-1")   # 'café' -- wrong, but no exception was raised

Common Encoding Errors

UnicodeDecodeError happens when bytes don't form a valid sequence for the declared encoding (for example, decoding arbitrary binary data as UTF-8). UnicodeEncodeError happens when a str contains a code point that the target encoding can't represent (for example, encoding an emoji as ascii). Both are subclasses of UnicodeError, itself a ValueError.

try:
    b"\xff\xfe".decode("utf-8")
except UnicodeDecodeError as e:
    print(e)   # invalid start byte

try:
    "café".encode("ascii")
except UnicodeEncodeError as e:
    print(e)   # 'ascii' codec can't encode character '\xe9' ...

The errors Parameter

Both encode() and decode() accept an errors argument controlling how to handle characters or bytes that can't be converted: 'strict' (default, raises), 'ignore' (silently drops the offending data — usually a poor default since it loses information), 'replace' (substitutes a placeholder, '\ufffd' for decode or '?' for encode), and 'backslashreplace' (produces a Python-escape-style representation, useful for debugging).

"café".encode("ascii", errors="ignore")          # b'caf'
"café".encode("ascii", errors="replace")         # b'caf?'
"café".encode("ascii", errors="backslashreplace") # b'caf\\xe9'

Reading and Writing Text Files: Always Name the Encoding

When opening a file in text mode, Python decodes/encodes automatically — but the encoding it assumes if you don't specify one is platform-dependent (CPython implementation detail: it defaults to locale.getencoding(), which varies by OS and locale configuration). Always pass encoding="utf-8" explicitly for reproducible, portable behavior, especially for files you intend to share across machines.

Opening a file in binary mode ('rb'/'wb') skips decoding entirely and gives you raw bytes — the right choice for non-text data, or when you need to inspect/handle the encoding yourself.

from pathlib import Path

path = Path("notes.txt")
path.write_text("café ☕\n", encoding="utf-8")
text = path.read_text(encoding="utf-8")
print(text)

Unicode Normalization (Concept)

Some characters can be represented in more than one way: an accented letter like 'é' can be a single code point (U+00E9) or a base letter 'e' followed by a combining accent mark (U+0065 U+0301). These look identical when rendered but compare as *unequal* strings and have different lengths. Unicode normalization (via the standard unicodedata module's normalize() function) converts text to one of four canonical forms (NFC, NFD, NFKC, NFKD) so equivalent text compares equal — important for reliable string comparison, search, and deduplication of user-entered or externally-sourced text.

import unicodedata

composed = "caf\u00e9"          # 'é' as one code point
decomposed = "cafe\u0301"       # 'e' + combining acute accent

composed == decomposed           # False -- different code point sequences!
len(composed), len(decomposed)   # (4, 5)

unicodedata.normalize("NFC", decomposed) == composed   # True

Practical Guidance

Treat 'the Unicode sandwich' as the goal: decode bytes to str as early as possible on input (reading a file, a network response, user input), work with str throughout your program's logic, and encode back to bytes as late as possible on output. Avoid mixing str and bytes in the same variable or function's logic — Python's strict separation exists precisely to catch that mistake early via TypeError rather than letting it corrupt data silently.

Key Takeaways

str is text (a sequence of Unicode code points); bytes is raw binary data — they are distinct types that cannot be mixed directly.

str.encode() converts text to bytes; bytes.decode() converts bytes back to text; both require naming an encoding (default utf-8).

UTF-8 is the standard, recommended encoding: ASCII-compatible, represents every code point, variable-width.

Decoding with the wrong encoding may raise UnicodeDecodeError or silently produce wrong (but valid-looking) characters — there's no way to detect the 'right' encoding from bytes alone.

Always pass an explicit encoding= when opening text files; the platform default is not guaranteed and varies by system.

Visually identical text can differ in code point sequence; use unicodedata.normalize() before comparing untrusted or user-entered strings.

open("data.txt").read() # relies on the platform's default encoding open("data.txt", encoding="utf-8").read()

Without an explicit encoding, text-mode file I/O uses the platform's locale-dependent default, which can differ between machines and silently corrupt non-ASCII text or raise UnicodeDecodeError on another system.

response_bytes.decode("latin-1") # 'works' without erroring, but text is garbled response_bytes.decode("utf-8") # use the encoding the source actually used

Many single-byte encodings like latin-1 can decode almost any byte sequence without raising an error, which means a wrong-encoding bug can go unnoticed -- the output looks like text, just the wrong text (mojibake).

name + name_bytes # TypeError: str + bytes name + name_bytes.decode("utf-8") str and bytes cannot be concatenated directly; decode the bytes to str (or encode the str to bytes) first, depending on which type you actually want the result to be.

if user_input_a == user_input_b: # both render as 'café' but differ in composition import unicodedata if unicodedata.normalize('NFC', user_input_a) == unicodedata.normalize('NFC', user_input_b):

Two strings can look identical when displayed but contain different Unicode code point sequences (composed vs. decomposed accents). Normalize both sides before comparing text that may come from different input sources.

What is the fundamental difference between a Python str and a Python bytes object?

str is a sequence of Unicode code points -- abstract text with no inherent byte representation. bytes is a sequence of raw 8-bit integers (0-255) with no inherent notion of characters. Converting between them always requires stating an encoding, via encode() (str to bytes) or decode() (bytes to str).

Why is UTF-8 generally the recommended default encoding?

It can represent every Unicode code point, it is backward-compatible with 7-bit ASCII (plain ASCII text is unchanged in UTF-8), and it is variable-width so common text stays compact -- making it a safe, near-universal default for interchange.

What's the difference between UnicodeDecodeError and UnicodeEncodeError?

UnicodeDecodeError occurs when converting bytes to str and the bytes don't form a valid sequence for the declared encoding. UnicodeEncodeError occurs when converting str to bytes and the string contains a code point the target encoding cannot represent.

Why might decoding bytes with the wrong encoding NOT raise an exception, and why is that dangerous?

Some encodings (like latin-1) map every possible byte value to some character, so almost any byte sequence decodes 'successfully' -- just to the wrong text. This is dangerous because the bug produces no error, only silently corrupted (mojibake) output that may not be noticed until much later.

Why can two strings that display identically still compare as unequal with ==?

Unicode allows some characters to be represented in more than one code point sequence (e.g., a precomposed accented letter versus a base letter plus a combining accent mark). These render the same but are different sequences of code points, so == (which compares code points) returns False unless the strings are normalized to the same form first.

What does ord('A') return, and what does chr() do relative to it?

ord('A') returns 65, the Unicode code point for 'A'. chr() is the inverse: chr(65) returns the one-character string 'A'.

Why should you avoid mixing str and bytes values in the same piece of logic?

Python 3 deliberately does not allow implicit coercion between str and bytes -- operations like concatenation or comparison between them raise TypeError immediately. This is intentional: it surfaces encoding-boundary bugs as an explicit, early error instead of letting mismatched data flow silently through a program.

Write a function safe_decode(data, encoding='utf-8') that decodes bytes to str, returning None instead of raising if the bytes are not valid for that encoding.

def safe_decode(data: bytes, encoding: str = "utf-8") -> str | None: try: return data.decode(encoding) except UnicodeDecodeError: return None print(safe_decode("café".encode("utf-8"))) # café print(safe_decode(b"\xff\xfe")) # None

Wrap the decode() call in a try/except UnicodeDecodeError, returning None on failure instead of letting the exception propagate -- a pattern useful when processing untrusted or unknown-source byte data.

What does this print, and why do the two lengths differ?

It prints '5 6'. len(s) counts Unicode code points (n-a-ï-v-e is 5 characters), while len(b) counts bytes -- the 'ï' character takes 2 bytes in UTF-8 (since its code point exceeds 127), so the byte string is one longer than the character string.

Write a function are_equivalent(a, b) that returns True if two strings represent the same text once Unicode-normalized to NFC form, even if their underlying code point sequences differ.

import unicodedata def are_equivalent(a: str, b: str) -> bool: return unicodedata.normalize("NFC", a) == unicodedata.normalize("NFC", b) print(are_equivalent("caf\u00e9", "cafe\u0301")) # True unicodedata.normalize('NFC',...) converts both strings to the same canonical composed form before comparing, so visually identical text with different underlying code point sequences (composed vs. decomposed accents) compares equal.

What does this print? (The euro sign U+20AC requires 3 bytes in UTF-8.)

It prints b'5\xe2\x82\xac' and then 4 -- the ASCII digit '5' takes 1 byte, and the euro sign takes 3 bytes in UTF-8, for a total of 4 bytes even though the original string has only 2 characters.

s = "naïve"
b = s.encode("utf-8")
print(len(s), len(b))

data = "5€".encode("utf-8")
print(data)
print(len(data))