Parsing Concepts
Turning raw text into structured data is a discipline of its own — tokenizing, validating, and choosing the right tool instead of forcing everything through regex.
Parsing is the process of taking unstructured or semi-structured text and turning it into structured data your program can work with: numbers, dates, nested objects, or an abstract syntax tree. This topic covers the conceptual building blocks -- tokenization, validation, and structured parsing -- and how to choose between regex, hand-written parsing, and parser libraries.
What Is Parsing
Parsing converts a sequence of characters into a structured representation according to some grammar or format -- turning "3 + 4 * 2" into an expression tree, '{"a": 1}' into a dict, or a log line into a record with named fields. Every parser answers two questions: is this input well-formed (validation), and if so, what structure does it represent (extraction).
Parsing sits between raw input (a string, a file, a network stream) and the rest of your program, which should generally never see unstructured text again once the parsing step is done -- the parser's output should be trustworthy structured data (dataclasses, dicts, custom objects) that downstream code can rely on without re-checking format.
Tokenization
Tokenization (lexical analysis) is the first stage of most parsing: breaking a raw character stream into a sequence of meaningful chunks called tokens -- numbers, identifiers, operators, punctuation, string literals -- while discarding or separating out irrelevant detail like whitespace. A tokenizer does not care about structure or nesting; it just classifies runs of characters.
Each token has a kind and a value; a downstream parser consumes this flat token stream rather than re-scanning raw characters.
import re
from dataclasses import dataclass
@dataclass
class Token:
kind: str
value: str
TOKEN_SPEC = [
("NUMBER", r"\d+(\.\d+)?"),
("PLUS", r"\+"),
("MINUS", r"-"),
("STAR", r"\*"),
("LPAREN", r"\("),
("RPAREN", r"\)"),
("SKIP", r"[ \t]+"),
]
TOKEN_RE = re.compile("|".join(f"(?P<{name}>{pat})" for name, pat in TOKEN_SPEC))
def tokenize(text: str) -> list[Token]:
tokens = []
for match in TOKEN_RE.finditer(text):
kind = match.lastgroup
if kind != "SKIP":
tokens.append(Token(kind, match.group()))
return tokens
for token in tokenize("12 + (3 * 4)"):
print(token)Validation
Validation checks whether input conforms to an expected format or grammar, often without necessarily extracting every piece of structure -- "is this a syntactically valid email address", "is this a well-formed date string". Validation and parsing overlap heavily: a full parse that succeeds is itself a validation, and a parser that raises or returns None/a sentinel on malformed input is doing validation as a side effect.
For simple format checks, the standard library often already validates for you as a byproduct of conversion -- int("abc") raises ValueError rather than silently returning a placeholder, and datetime.fromisoformat(...) raises on a malformed ISO string. Prefer "try to convert, handle the exception" over hand-rolled validation regex whenever a stdlib converter already exists for the format.
def parse_positive_int(text: str) -> int:
value = int(text) # raises ValueError on non-numeric text
if value <= 0:
raise ValueError(f"expected a positive integer, got {value}")
return value
for candidate in ["42", "-3", "abc"]:
try:
print(parse_positive_int(candidate))
except ValueError as exc:
print("invalid:", exc)Structured Parsing
Structured parsing produces a nested, structured result -- a tree, a dict, a list of records -- from tokens or raw text, respecting the format's grammar including any nesting. For well-known formats, the standard library already provides a structured parser, and reimplementing one with regex or manual string splitting is almost always a mistake:
Format Standard library parser
------ ------------------------
JSON json.loads / json.load
CSV csv.reader / csv.DictReader
XML xml.etree.ElementTree
INI-style configparser
URLs urllib.parse
command line argparse
Python code ast (via compile()/ast.parse)import json
raw = '{"name": "Ada", "skills": ["math", "engineering"]}'
data = json.loads(raw)
print(data["name"], data["skills"])Parser Libraries (Third-Party)
For custom grammars that don't map onto an existing stdlib format (a small config language, a query language, a domain-specific expression syntax), hand-writing a recursive-descent parser or reaching for a third-party parser library is the standard approach. pyparsing, lark, and ply are well-known third-party (not standard library) parsing toolkits that let you describe a grammar declaratively and get a parse tree back.
These are worth knowing exist, but are out of scope to teach in depth here -- the key point for this topic is recognizing when your problem has crossed from "a regex will do" into "this needs a real grammar and a parser".
Why Regex Fails on Nested Languages
Regular expressions describe regular languages, which have no concept of matching, counted nesting -- a regex cannot in general verify that parentheses are balanced or that HTML tags are properly closed, because doing so requires unbounded memory of how deep you are (a stack), which finite-state regex matching does not have. A regex can approximate simple, shallow, or fixed-depth nesting, but it breaks down as soon as depth is unbounded or content varies.
A real parser tracks a stack of open tags; regex has no equivalent memory, so nesting correctness is only ever coincidental.
Rule of thumb from the Regex Best Practices topic applies directly here: once a format has real nesting or a recursive grammar, stop trying to force it through regex and use json, an XML/HTML parser, or a grammar-based parser library instead.
import re
# This "looks like" it matches a tag pair, but fails on nested tags
# of the SAME name, overlapping tags, or attributes with '>' in them.
naive = re.compile(r"<(\w+)>(.*?)</\1>")
print(naive.findall("<b>bold <b>and nested</b> text</b>"))
# Only matches the inner pair correctly by accident; a real document
# with multiple nested/sibling tags breaks this quickly.A Minimal Recursive-Descent Parser
When no library fits, a small hand-written recursive-descent parser is often the clearest option: one function per grammar rule, each consuming tokens and calling other rule functions for nested structure. This naturally handles unbounded nesting because function calls form a real call stack.
# Parses simple arithmetic: NUMBER (('+' | '-') NUMBER)*
def parse_expression(tokens: list[Token]) -> float:
pos = 0
def parse_number() -> float:
nonlocal pos
tok = tokens[pos]
pos += 1
return float(tok.value)
result = parse_number()
while pos < len(tokens) and tokens[pos].kind in ("PLUS", "MINUS"):
op = tokens[pos].kind
pos += 1
rhs = parse_number()
result = result + rhs if op == "PLUS" else result - rhs
return result
print(parse_expression(tokenize("3 + 4 - 1")))Practical Usage
In everyday code, most parsing needs are met by an existing stdlib converter or format parser: int()/float() for numbers, datetime.fromisoformat() for dates, json/csv/configparser for structured formats, and argparse for command-line input. Reach for regex only for genuinely flat pattern matching (validating a shape, extracting a substring), and reach for a hand-written or library parser only when the format has real grammar/nesting and no stdlib module already covers it.
Edge Cases
Malformed input is the norm, not the exception, for anything parsing external data -- always decide up front whether a parse failure should raise, return None/a sentinel, or collect partial results with errors reported separately, and make that decision explicit rather than letting an unhandled exception surface deep in unrelated code.
Encoding matters before parsing even starts: text read from a file or socket must be decoded with the correct character encoding (commonly UTF-8) before token boundaries make sense; a wrong encoding can silently corrupt tokenization.
Whitespace and comment handling should be decided explicitly in tokenization (skip, or keep as tokens) -- inconsistent handling is a common source of subtly wrong parsers.
Complete Example
This is a flat-format use case (fixed field order, no nesting) where regex tokenizing/validating each line is appropriate; malformed lines are explicitly skipped rather than crashing the whole parse.
import re
from dataclasses import dataclass
@dataclass
class LogRecord:
date: str
level: str
message: str
LOG_PATTERN = re.compile(
r"^(?P<date>\d{4}-\d{2}-\d{2}) "
r"(?P<level>DEBUG|INFO|WARNING|ERROR) "
r"(?P<message>.+)$"
)
def parse_log(lines: list[str]) -> list[LogRecord]:
records = []
for line in lines:
match = LOG_PATTERN.match(line)
if match is None:
continue # explicit choice: skip malformed lines
records.append(LogRecord(**match.groupdict()))
return records
lines = [
"2024-03-15 ERROR disk usage above threshold",
"not a valid log line",
"2024-03-15 INFO service started",
]
for record in parse_log(lines):
print(record)Mental Model
Think of parsing as a pipeline: raw text -> tokens (lexical analysis) -> structure (syntactic analysis / grammar) -> validated, structured data your program trusts. Regex is good at the tokenizing stage and at flat validation; it is the wrong tool for the structural stage once nesting is involved -- that's where a stack-based parser (hand-written or from a library) takes over.
Quick Reference
Tokenization raw text -> flat sequence of typed tokens
Validation confirm input conforms to a format (often via try/except)
Structured parse tokens/text -> nested structure (tree, dict, records)
json/csv/xml/configparser/argparse -> use these before writing your own parser
Regex good for: flat patterns, tokenizing, simple validation
bad for: nested/recursive grammars (balanced delimiters, markup)
Recursive descent -> one function per grammar rule, handles unbounded nestingKey Takeaways
Parsing breaks into tokenization (flat lexical chunks) and structural parsing (nested/grammar-aware structure).
Validation is often a natural side effect of parsing -- prefer "try to convert, catch the exception" over separate hand-rolled validation when a converter already exists.
Use the standard library's dedicated parsers (json, csv, xml.etree.ElementTree, configparser, argparse) instead of reinventing them with regex or string splitting.
Regex cannot correctly express unbounded nesting; use a real parser (hand-written recursive descent, or a library like pyparsing/lark) once a format has recursive structure.
Decide explicitly how malformed input is handled -- raise, return a sentinel, or collect errors -- rather than letting failures surface as unrelated exceptions later.
re.findall(r"<(\w+)>(.*?)</\1>", html_text) from xml.etree import ElementTree as ET tree = ET.fromstring(html_text) # or a real HTML parser for non-XML HTML
Regex cannot reliably handle nested tags of the same name, self-closing tags, or attributes; a real parser tracks structure with a stack.
value = text.split(",")[2] # "parse" a CSV line by hand import csv row = next(csv.reader([text])) value = row[2]
Hand-splitting on commas breaks on quoted fields containing commas; csv.reader handles quoting, escaping, and edge cases correctly.
try: n = int(text) except Exception: n = 0 try: n = int(text) except ValueError: n = None # or re-raise / report, depending on the caller's needs
Catching bare Exception hides real bugs and silently substituting 0 hides parse failures as valid data; catch the specific exception and make failure visible.
What is the difference between tokenization and structured parsing?
Tokenization breaks raw text into a flat sequence of typed chunks (numbers, operators, identifiers) with no concept of nesting; structured parsing consumes those tokens (or the text directly) to build a nested structure according to a grammar.
Why is int(text) often a better validation tool than a hand-written regex for 'is this a valid integer'?
It performs the actual conversion and raises ValueError on invalid input as a natural side effect, so there's no separate validation step to keep in sync with the conversion logic, and it correctly handles details (leading +/-, whitespace) that a naive regex might miss.
Name three standard-library modules that provide structured parsers for common formats.
Any three of: json, csv, xml.etree.ElementTree, configparser, argparse, urllib.parse.
Why can't a regular expression correctly validate that parentheses are balanced in an arbitrary-length expression?
Regular expressions describe regular languages, which have no memory of nesting depth (no stack); verifying balance requires counting unboundedly, which needs a stack-based (context-free) parser instead.
What are pyparsing, lark, and ply, and how should they be described in documentation?
They are third-party (not standard library) Python parser-generator/toolkit libraries for building grammar-based parsers; any mention of them should be clearly labeled as third-party.
In a log-parsing pipeline, what are three reasonable ways to handle a malformed line, and why must the choice be explicit?
Raise an exception and stop, skip the line and continue, or collect it into a separate list of errors alongside successfully parsed records. The choice must be explicit because silently doing any one of these without deciding can hide data loss or crash unrelated code far from the actual bad input.
Why does a recursive-descent parser naturally handle unbounded nesting where regex cannot?
Because each grammar rule is implemented as a function, and functions calling each other build a real call stack in the language runtime -- that stack is exactly the unbounded memory needed to track nesting depth, which finite-state regex matching lacks.
Write a function tokenize_numbers_and_ops(text) that returns a list of tokens (as strings) for a simple arithmetic expression containing integers, '+', '-', '*', '/', and spaces (spaces are skipped).
import re def tokenize_numbers_and_ops(text: str) -> list[str]: pattern = re.compile(r"\d+|[+\-*/]|\s+") tokens = [] for match in pattern.finditer(text): piece = match.group() if not piece.isspace(): tokens.append(piece) return tokens print(tokenize_numbers_and_ops("12 + 5 * 3"))
['12', '+', '5', '*', '3'] -- whitespace matches are found but filtered out before appending, leaving only meaningful tokens.
What does this print, and why?
'invalid'. A trailing comma before the closing brace is not valid JSON syntax, so json.loads raises JSONDecodeError, which is caught and handled instead of propagating.
Write a function parse_key_value(text) that parses lines of the form 'key=value' (one per line) into a dict, skipping blank lines and lines without an '=' sign.
def parse_key_value(text: str) -> dict[str, str]: result = {} for line in text.splitlines(): line = line.strip() if not line or "=" not in line: continue key, _, value = line.partition("=") result[key.strip()] = value.strip() return result sample = "name = Ada\n\nrole=engineer\nmalformed line" print(parse_key_value(sample))
{'name': 'Ada', 'role': 'engineer'} -- blank lines and the line without '=' are skipped; str.partition cleanly splits on the first '=' without needing regex.
import json
try:
data = json.loads('{"a": 1,}')
except json.JSONDecodeError as e:
print("invalid")
else:
print(data)