Python Lexical Structure
Before Python parses statements, it breaks source text into tokens — identifiers, keywords, literals, operators, delimiters — governed by rules about indentation, whitespace, and line continuation that are stricter than most languages.
Lexical structure is the lowest layer of a language's grammar: how raw source text is broken into meaningful tokens before those tokens are assembled into statements and expressions. Python's lexical rules are distinctive because indentation itself is a token-level concept with syntactic weight, not just cosmetic whitespace. This topic surveys the token categories — identifiers, keywords, literals, operators, delimiters — and the whitespace/continuation/encoding rules around them.
What Is Lexical Structure?
Before Python can understand def area(r): return 3.14 * r 2 as a function definition, it must first split that text into tokens: the keyword def, the identifier area, delimiters ( and ), the identifier r, the delimiter ), the delimiter :, the keyword return, the literal 3.14, the identifier r, the operator , and the literal 2. This splitting process — tokenizing — is Python's lexical analysis, and it happens before parsing (which assembles tokens into a grammatical structure) and before execution.
Identifiers
An identifier is a name used for a variable, function, class, module, or other user-defined entity. Python identifiers must start with a letter (Unicode letters are allowed, not just ASCII) or an underscore, followed by any number of letters, digits, or underscores. Identifiers are case-sensitive (total and Total are different names).
Full identifier rules, naming conventions, and the special meaning of leading/trailing underscores are covered in depth in the Keywords & Identifiers topic.
total_price = 19.99 # valid
_private = 1 # valid — leading underscore has a naming convention (see below)
café = "coffee" # valid — Unicode letters are permitted
2fast = True # INVALID — cannot start with a digitKeywords
Keywords are identifier-shaped words reserved by the language itself — words like if, def, class, return, and, and None — which cannot be used as ordinary identifiers because they have fixed syntactic meaning. Python 3 also has a small set of "soft keywords" (like match, case, and _ used in structural pattern matching, and type used in type-alias statements) that are only special in specific contexts and remain valid identifiers elsewhere.
The full reserved-keyword list and soft-keyword behavior are covered in depth in the dedicated Keywords & Identifiers topic; this section only places keywords within the overall token taxonomy.
Literals
A literal is source-code notation for a fixed value of a particular type, written directly into the code rather than computed.
Numeric literals can also use underscores as visual separators for readability (1_000_000), and integer literals can be written in binary (0b101), octal (0o17), or hexadecimal (0x1F) in addition to decimal.
42 # int literal
3.14 # float literal
1 + 2j # complex literal
"hello" # str literal
'world' # str literal — single or double quotes are equivalent
b"bytes" # bytes literal
True, False # bool literals
None # the NoneType literal — the "no value" singleton
[1, 2, 3] # list literal
(1, 2, 3) # tuple literal
{1, 2, 3} # set literal
{"a": 1} # dict literalOperators
Operators are symbols that perform operations on values: arithmetic (+, -, *, /, //, %, **), comparison (==, !=, <, >, <=, >=), assignment (=, and the augmented forms +=, -=, etc.), logical (and, or, not — written as words, not symbols, in Python), bitwise (&, |, ^, ~, <<, >>), identity (is, is not), and membership (in, not in).
print(7 // 2) # 3 — floor division
print(7 % 2) # 1 — modulo
print(2 ** 10) # 1024 — exponentiation
print(3 is 3) # implementation-dependent for small ints; don't rely on it (see below)
print(3 in [1, 2, 3]) # True — membershipDelimiters
Delimiters are punctuation tokens that structure code without being operators themselves: parentheses () (grouping, calls, tuples), brackets [] (indexing, list literals), braces {} (dict/set literals), the colon : (introducing blocks and slices, and annotating types), the comma , (separating items), the dot . (attribute access), and the semicolon ; (statement separator, rarely used since Python relies on newlines instead).
numbers = [1, 2, 3] # brackets: list literal
first = numbers[0] # brackets: indexing
point = (1, 2) # parentheses: tuple
config = {"debug": True} # braces: dict literal
x = 1; y = 2 # semicolon: two statements on one line (discouraged by PEP 8)Indentation as Syntax
Where most languages treat indentation as purely cosmetic and use braces to define blocks, Python's tokenizer produces implicit INDENT and DEDENT tokens based on a line's leading whitespace, and these tokens are how the parser determines block structure. This makes indentation load-bearing at the lexical level, not just a formatting convention.
Block structure via indentation is covered from the program-structure angle in the Program Structure topic; this section frames it specifically as a lexical/tokenization concern.
Whitespace Rules
Outside of leading indentation (which is syntactically significant) and inside string literals (where whitespace is part of the literal's value), whitespace between tokens is generally insignificant to the parser — x=1 and x = 1 mean the same thing, though PEP 8 strongly prefers spaces around most operators for readability. A logical line normally ends at a physical newline, which is itself a meaningful token (it signals the end of a statement).
Line Continuation
A single logical line of code can span multiple physical lines two ways. Implicit continuation happens automatically inside any unclosed bracket — (), [], or {} — so multi-line expressions inside brackets need no special marker and are the preferred, more readable style. Explicit continuation uses a backslash \ as the very last character of a physical line to say "this statement continues on the next line" — usable outside brackets, but less preferred stylistically because it's fragile (trailing whitespace after the backslash breaks it silently) and harder to read.
# Implicit continuation (preferred) — inside parentheses
total = (
price
+ tax
+ shipping
)
# Explicit continuation — a backslash, used sparingly
total = price \
+ tax \
+ shippingSource Encoding Declaration
Python 3 source files are UTF-8 by default, so an explicit encoding declaration is rarely needed anymore (it was more common under Python 2, whose default was ASCII). If a file genuinely needs a different encoding, a special comment on the first or second line declares it, following the pattern from PEP 263: # -*- coding: <encoding-name> -*-. In ordinary modern Python 3 development targeting UTF-8, you can simply omit this — it exists mainly for legacy or unusual-encoding situations.
# -*- coding: utf-8 -*-
# (Rarely needed explicitly in Python 3, since UTF-8 is already the default.)f-String Lexical Considerations
An f-string (formatted string literal, prefixed with f or F, introduced in Python 3.6) embeds expressions directly inside string literals using {expression} syntax, evaluated and formatted at runtime when the f-string is evaluated. Lexically, the text inside the braces is itself parsed as a full Python expression — not just a variable name — which is a meaningful departure from how ordinary string literals are tokenized (where {} would just be literal characters).
Since Python 3.12, f-string parsing was formalized and relaxed (PEP 701) so that quote characters inside the {expression} portion can match the f-string's own outer quote character, and expressions can span multiple lines and include comments — removing several lexical restrictions that earlier Python versions imposed on what could appear inside the braces.
name = "Ada"
score = 97.456
print(f"{name} scored {score:.1f}%") # Ada scored 97.5%
print(f"{2 + 2}") # 4 — a full expression, not just a name
print(f"{{literal braces}}") # {literal braces} — doubled braces escape to a literal braceKey Takeaways
Lexical analysis breaks source text into tokens — identifiers, keywords, literals, operators, delimiters — before any parsing or execution happens.
Indentation is a lexical-level concern in Python: the tokenizer emits INDENT/DEDENT tokens from leading whitespace, and these drive block structure.
Ordinary whitespace between tokens (outside indentation and string literals) is not significant, but PEP 8 has strong style conventions for it anyway.
Line continuation is implicit inside unclosed brackets (preferred) or explicit via a trailing backslash (used sparingly).
Python 3 source defaults to UTF-8, making explicit encoding declarations mostly a legacy concern.
f-strings parse the contents of {} as full Python expressions, evaluated at runtime — not as plain literal text.
total = price + \ tax + \ shipping # trailing space after \ silently breaks continuation total = ( price + tax + shipping)
A backslash continuation must be the absolute last character on the line — any trailing whitespace after it breaks the continuation, often silently producing a confusing SyntaxError. Implicit continuation inside parentheses avoids this fragility entirely.
if True: x = 1 y = 2 # mixes a tab and spaces in the same block if True: x = 1 y = 2 # 4 spaces, consistently
Mixing tabs and spaces for indentation raises a TabError in Python 3. PEP 8 mandates spaces only, 4 per level.
print(f"{name" + "}") # trying to build braces from string concatenation print(f"{{literal}}") # doubled braces are the correct way to escape a literal brace in an f-string
Inside an f-string, { and } are special delimiters for embedded expressions; to include a literal brace character, double it ({{ or }}) rather than trying to concatenate it in from outside the literal.
What is the difference between lexical analysis and parsing?
Lexical analysis (tokenizing) breaks raw source text into a stream of meaningful tokens — identifiers, keywords, literals, operators, delimiters, and indentation markers. Parsing then takes that token stream and assembles it into a grammatical structure (statements, expressions) according to the language's grammar. Lexing happens first and feeds parsing.
Why is indentation described as a 'lexical' concern in Python rather than purely a formatting style?
Because Python's tokenizer itself produces INDENT and DEDENT tokens directly from a line's leading whitespace, and the parser relies on those tokens to determine block boundaries. This makes indentation part of the token stream, with real syntactic consequences (an IndentationError), not just a cosmetic convention checked later.
What are the two ways to continue a single logical line of Python code across multiple physical lines, and which is preferred?
Implicit continuation, which happens automatically for any code inside an unclosed bracket ((), [], or {}), and explicit continuation using a trailing backslash \ as the very last character of a line. Implicit (bracket-based) continuation is preferred because it's more readable and doesn't break silently from trailing whitespace.
Does Python 3 require an explicit source-encoding declaration comment in most modern code? Why or why not?
No, not in ordinary modern development — Python 3 source files default to UTF-8 encoding already. The # -*- coding: ... -*- declaration (PEP 263) is mainly a legacy mechanism for files that need a non-UTF-8 encoding, a more common situation under Python 2's ASCII default.
In an f-string like f"{price * 1.1:.2f}", what exactly gets parsed as a Python expression, and what doesn't?
Everything inside the curly braces — price * 1.1 — is parsed and evaluated as a full Python expression at runtime, including operators, function calls, or any valid expression. The :.2f portion after the colon is a format specification, not a Python expression. Everything outside the braces in the string is ordinary literal text.
List four categories of tokens that make up Python's lexical structure.
Any four of: identifiers (names), keywords (reserved words), literals (fixed values like 42, "text", True), operators (+, ==, and, is, in, etc.), and delimiters (punctuation like (), [], {}, :, ,, .).
What does this print, and why doesn't Python require a backslash for the multi-line expression?
It prints 6. Because the expression is inside unclosed parentheses, Python treats it as one continuous logical line automatically — implicit line continuation — regardless of how many physical lines it spans, so no backslash is needed.
Write a variable assignment using an integer literal for one million, formatted with underscore separators for readability.
population = 1_000_000 Underscores in numeric literals are purely a readability aid for the programmer — Python ignores them when parsing the value, so population equals exactly 1000000.
What does this print?
It prints: {Hello, World!} — the doubled {{ and }} are escaped to literal single braces, while the single {name} in between is evaluated as the expression name and substituted in.
Explain, in one or two sentences, why 2fast = True is not a valid Python identifier.
Python identifiers must begin with a letter (Unicode letters included) or an underscore — never a digit. Since 2fast starts with the digit 2, it's lexically invalid as an identifier and raises a SyntaxError.
x = (
1
+ 2
+ 3
)
print(x)name = "World"
print(f"{{Hello, {name}!}}")