Regular Expressions
Pattern-matching text with the re module — a compact language for describing, searching, validating, and transforming strings that goes far beyond plain substring search.
Regular expressions describe patterns of text — not literal strings, but shapes like "three digits, a hyphen, then four digits" or "any word starting with a capital letter". Python's re module compiles and applies these patterns for searching, matching, validating, and substituting text. Regular expressions are a small language of their own; this topic covers the practical core you'll use in almost every real script, not the full formal grammar.
What Is a Regular Expression
A regular expression ("regex") is a pattern string written in a specialized mini-language for matching text. Python exposes this through the re module, using patterns written as ordinary Python strings — almost always as raw strings (r"...") so backslashes in the pattern aren't also interpreted as Python string escapes.
Always prefer raw string literals (r"...") for regex patterns. Without the r prefix, a pattern like "\d" risks Python itself interpreting \d as an (invalid, so left alone with a DeprecationWarning) escape sequence before re ever sees it — r"\d" avoids that ambiguity entirely and is the idiomatic choice.
import re
text = "Order #4471 shipped on 2024-03-15"
match = re.search(r"\d{4}-\d{2}-\d{2}", text)
print(match.group()) # 2024-03-15Basic Syntax: search, match, fullmatch
re.search(pattern, string) scans the whole string for the first location where the pattern matches, anywhere. re.match(pattern, string) only checks for a match starting at position 0. re.fullmatch(pattern, string) requires the entire string to match the pattern, start to end. All three return a Match object on success or None on failure.
A common validation mistake is using re.match (or even re.search) when re.fullmatch is what's actually needed — match/search succeed on a pattern found anywhere at or after the start, even if trailing characters don't match anything.
import re
text = " hello world"
print(re.search(r"hello", text)) # matches — found anywhere in the string
print(re.match(r"hello", text)) # None — text doesn't start with "hello" (leading spaces)
print(re.match(r"\s*hello", text)) # matches — pattern accounts for the leading spaces
print(re.fullmatch(r"\d+", "12345")) # matches — entire string is digits
print(re.fullmatch(r"\d+", "12345 ")) # None — trailing space breaks a full matchCharacter Classes
A character class matches any one character from a set. [abc] matches a, b, or c. [a-z] matches any lowercase letter via a range. [^abc] (caret first) negates the class, matching anything except those characters. Several shorthand classes cover common cases: \d (digit, equivalent to [0-9] for ASCII), \w (word character: letters, digits, underscore), \s (whitespace), and their uppercase negations \D, \W, \S.
By default, \d, \w, and \s match Unicode categories (e.g. \w matches accented letters too), not just ASCII — this is the Python 3 default (str patterns are Unicode-aware unless re.ASCII is passed).
import re
print(re.findall(r"[aeiou]", "hello world")) # ['e', 'o', 'o']
print(re.findall(r"[^aeiou\s]", "hello world")) # non-vowel, non-space chars
print(re.findall(r"\d", "room 42, floor 3")) # ['4', '2', '3']
print(re.findall(r"\w+", "hello, world!")) # ['hello', 'world']Quantifiers
Quantifiers control how many times the preceding element must repeat. * means zero or more, + means one or more, ? means zero or one (optional), and {m,n} means between m and n repetitions (either bound can be omitted: {2,} means "2 or more", {,5} means "up to 5"). All of these are greedy by default — they match as much as possible — but each has a lazy (?-suffixed) counterpart that matches as little as possible.
import re
print(re.findall(r"ab*", "a ab abb abbb")) # ['a', 'ab', 'abb', 'abbb'] — zero or more b's
print(re.findall(r"ab+", "a ab abb abbb")) # ['ab', 'abb', 'abbb'] — one or more (excludes 'a')
print(re.findall(r"colou?r", "color colour")) # ['color', 'colour']
print(re.findall(r"\d{3,5}", "12 1234 123456")) # ['1234', '12345'] — 3 to 5 digits
# Greedy vs lazy
html = "<b>bold</b>"
print(re.findall(r"<.*>", html)) # ['<b>bold</b>'] — greedy: matches as much as possible
print(re.findall(r"<.*?>", html)) # ['<b>', '</b>'] — lazy: matches as little as possibleAnchors
Anchors match a position rather than a character. ^ matches the start of the string (or start of a line, with re.MULTILINE); $ matches the end of the string (or end of a line, with re.MULTILINE); \b matches a word boundary (the transition between a \w and non-\w character); \B matches a non-word-boundary.
import re
print(bool(re.match(r"^Hello", "Hello world"))) # True
print(bool(re.search(r"world$", "Hello world"))) # True
print(re.findall(r"\bcat\b", "cat concatenate cats")) # ['cat'] — 'cat' as a whole word only
print(re.findall(r"\bcat", "cat concatenate cats")) # ['cat', 'cat'] — 'cat' at the START of a word
text = "line one\nline two"
print(re.findall(r"^line", text)) # ['line'] — only the string start, by default
print(re.findall(r"^line", text, flags=re.MULTILINE)) # ['line', 'line'] — start of each lineGroups and Capturing
Parentheses (...) create a capturing group — a sub-match you can retrieve separately, and that quantifiers can apply to as a unit. Match.group(n) retrieves the n-th group (group(0) or group() is the whole match); Match.groups() returns all captured groups as a tuple. Named groups, written (?P<name>...), let you retrieve captures by name instead of position — much more readable for patterns with several groups.
A non-capturing group, (?:...), groups a sub-pattern (e.g. for applying a quantifier to it, or for alternation) without creating a retrievable capture — useful for keeping groups() output limited to the captures you actually care about.
import re
m = re.search(r"(\d{4})-(\d{2})-(\d{2})", "Date: 2024-03-15")
print(m.group()) # 2024-03-15 (whole match)
print(m.group(1)) # 2024
print(m.groups()) # ('2024', '03', '15')
m2 = re.search(r"(?P<year>\d{4})-(?P<month>\d{2})-(?P<day>\d{2})", "Date: 2024-03-15")
print(m2.group("year")) # 2024
print(m2.groupdict()) # {'year': '2024', 'month': '03', 'day': '15'}import re
# (?:https?|ftp) groups the alternation without capturing it separately
m = re.search(r"(?:https?|ftp)://(?P<host>[\w.]+)", "https://example.com/path")
print(m.group("host")) # example.com — only the named group is capturedAlternation and Grouping Precedence
| matches either the pattern to its left or its right, and applies to the widest possible surrounding expression, so it's almost always used inside a group to scope it correctly.
import re
print(re.findall(r"cat|dog", "I have a cat and a dog")) # ['cat', 'dog']
print(re.fullmatch(r"gray|grey", "grey")) # matches
print(re.fullmatch(r"gr(a|e)y", "grey")) # matches, group captures 'e'Flags
Flags modify matching behavior globally for a pattern. The most common: re.IGNORECASE (re.I) for case-insensitive matching, re.MULTILINE (re.M) so ^/$ match at each line boundary rather than only the string's start/end, re.DOTALL (re.S) so . also matches newlines (by default . matches any character except \n), and re.VERBOSE (re.X) to allow whitespace and comments in the pattern for readability.
import re
print(re.findall(r"python", "Python PYTHON python", flags=re.IGNORECASE))
# ['Python', 'PYTHON', 'python']
pattern = re.compile(r"""
(?P<area>\d{3}) # area code
-
(?P<num>\d{4}) # local number
""", re.VERBOSE)
m = pattern.search("call 555-1234 now")
print(m.group("area"), m.group("num")) # 555 1234Compilation: re.compile
re.compile(pattern, flags=...) precompiles a pattern into a Pattern object with the same matching methods (.search(), .match(), .findall(), etc.) as the module-level functions. Compiling once and reusing the Pattern object is more efficient when the same pattern is applied repeatedly (e.g. inside a loop over many lines), and it keeps the pattern (and its flags) named and defined in one place.
CPython implementation detail: re internally caches a limited number of recently used uncompiled patterns automatically, so calling re.search(pattern, ...) repeatedly with the same pattern string isn't as slow as it might seem. Explicit re.compile is still preferred for a pattern used heavily or defined for reuse across a module, for clarity and to avoid depending on that cache's size.
import re
EMAIL_RE = re.compile(r"[\w.+-]+@[\w-]+\.[\w.-]+")
def find_emails(text: str) -> list[str]:
return EMAIL_RE.findall(text)
print(find_emails("contact [email protected] or [email protected]"))
# ['[email protected]', '[email protected]']findall and finditer
re.findall(pattern, string) returns all non-overlapping matches as a list — plain strings if the pattern has no groups (or one group), or tuples of group values if it has multiple groups. re.finditer(pattern, string) returns an iterator of Match objects instead, which is more memory-efficient for many matches and gives you full match objects (position, all groups) rather than just captured strings.
import re
text = "2024-03-15, 2023-11-02"
print(re.findall(r"\d{4}-\d{2}-\d{2}", text)) # ['2024-03-15', '2023-11-02']
print(re.findall(r"(\d{4})-(\d{2})-(\d{2})", text)) # [('2024','03','15'), ('2023','11','02')]
for m in re.finditer(r"\d{4}-\d{2}-\d{2}", text):
print(m.group(), "at position", m.start())Substitution: sub and subn
re.sub(pattern, replacement, string) replaces every non-overlapping match with replacement, returning the new string. The replacement can be a plain string (with \1, \g<name> etc. referencing captured groups) or a function receiving each Match object and returning its replacement text — useful when the substitution needs computed logic, not just a fixed template. re.subn does the same but also returns the number of substitutions made.
import re
text = "John Smith, Jane Doe"
print(re.sub(r"(\w+) (\w+)", r"\2 \1", text))
# Smith John, Doe Jane — swaps first/last name using group backreferences
def upper_match(m: re.Match) -> str:
return m.group().upper()
print(re.sub(r"\berror\b", upper_match, "error: file not found, no error here"))
# ERROR: file not found, no ERROR here
result, n = re.subn(r"\s+", " ", "too many spaces")
print(result, n) # 'too many spaces' 2Splitting with re.split
re.split(pattern, string) splits a string wherever the pattern matches, similar to str.split() but with a pattern instead of a fixed separator — useful when the separator itself is variable (multiple kinds of whitespace, optional punctuation).
import re
print(re.split(r"[,;]\s*", "a, b;c, d")) # ['a', 'b', 'c', 'd']
print(re.split(r"\s+", " too many spaces "))
# ['', 'too', 'many', 'spaces', ''] — note leading/trailing empty strings when the pattern
# matches at the very start/end of the stringPractical Usage: Validation Patterns
A frequent, tempting misuse is trying to validate structurally complex formats (like a fully RFC-compliant email address) with a single perfect regex — this is notoriously hard to get exactly right and often not worth the complexity. A pragmatic, "good enough" pattern combined with fullmatch (to require the whole string to conform) covers most real validation needs.
import re
def looks_like_email(s: str) -> bool:
return bool(re.fullmatch(r"[\w.+-]+@[\w-]+\.[a-zA-Z]{2,}", s))
print(looks_like_email("[email protected]")) # True
print(looks_like_email("not-an-email")) # False
print(looks_like_email("[email protected] ")) # False — fullmatch rejects the trailing spaceEdge Cases and Performance Notes
Certain patterns — heavily nested quantifiers like (a+)+ applied to a string that ultimately fails to match — can trigger catastrophic backtracking, where matching time grows exponentially with input length. This is a real denial-of-service risk if patterns are ever applied to untrusted, attacker-controlled input. Prefer simpler, more specific patterns, avoid nested unbounded quantifiers where possible, and be cautious about running complex regexes against untrusted strings of unbounded length.
A pattern like r"(a+)+$" applied to "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!"
can take exponential time to fail, because the engine tries an
enormous number of ways to partition the a's between the inner
and outer quantifier before concluding there's no match.Key Takeaways
Write regex patterns as raw strings (r"...") to avoid conflicts between Python's string escapes and regex escapes.
search finds a match anywhere, match only at the start, fullmatch requires the entire string to match — use fullmatch for validation.
Quantifiers are greedy by default (match as much as possible); a trailing ? makes them lazy (match as little as possible).
Capturing groups (...) retrieve sub-matches by position; named groups (?P<name>...) retrieve them by name; (?:...) groups without capturing.
re.compile precompiles a pattern for efficient, readable reuse, especially inside loops or when defined once for a whole module.
findall/finditer find all matches; sub/subn replace matches (with a string template or a function); split divides a string on pattern matches.
Watch for catastrophic backtracking from nested unbounded quantifiers, especially on untrusted input.
pattern = "\d{3}-\d{4}" # not a raw string pattern = r"\d{3}-\d{4}"
Without the r prefix, Python's own string-escape processing runs first, which can silently misinterpret backslash sequences before the regex engine ever sees the pattern; raw strings avoid the ambiguity.
if re.match(r"\d+", user_input): # matches even if extra trailing garbage follows accept(user_input) if re.fullmatch(r"\d+", user_input): accept(user_input) re.match only requires the pattern to match starting at position 0 — trailing characters after the match are ignored. Validating that the ENTIRE input conforms requires re.fullmatch.
html = "<b>bold</b> and <i>italic</i>" re.findall(r"<.*>", html) # ['<b>bold</b> and <i>italic</i>'] — greedy spans too far re.findall(r"<.*?>", html) # ['<b>', '</b>', '<i>', '</i>'] — lazy stops at the first '>'
Greedy quantifiers match as much as possible, so .* can span across multiple tags to the last '>' in the string; the lazy .*? variant stops at the nearest match instead.
for line in huge_file: re.search(r"complex_pattern_defined_inline", line) # re-parses the pattern every call
PATTERN = re.compile(r"complex_pattern_defined_inline") for line in huge_file: PATTERN.search(line)
Compiling the pattern once outside the loop (and reusing the Pattern object) is clearer and avoids relying on re's internal pattern cache for performance in a hot loop.
What is the difference between re.search, re.match, and re.fullmatch?
re.search finds a match anywhere in the string. re.match only checks for a match starting at position 0 (but doesn't require the match to reach the end). re.fullmatch requires the pattern to match the entire string from start to end — the right choice for validation.
Why should regex patterns almost always be written as raw strings?
Without the r prefix, Python processes backslash escape sequences in the string literal before the regex engine sees it, which can cause the pattern's intended backslash sequences (like \d or \b) to be misinterpreted or trigger warnings. Raw strings pass backslashes through unchanged to the regex engine.
What is the difference between a greedy quantifier like .* and its lazy counterpart .*?
A greedy quantifier matches as much text as possible while still allowing the overall pattern to succeed, potentially spanning far more than intended (e.g. across multiple tags in HTML). The lazy version (with a trailing ?) matches as little as possible, stopping at the nearest point that lets the rest of the pattern succeed.
How do named groups differ from ordinary numbered capturing groups, and why are they useful?
A named group, (?P<name>...), can be retrieved by a descriptive name via match.group('name') or match.groupdict(), instead of only by numeric position via match.group(n). This makes patterns with several groups much more readable and less fragile if group order changes.
When and why would you use re.compile instead of calling re.search/re.match directly with a pattern string each time?
re.compile precompiles a pattern once into a reusable Pattern object, which is more efficient and clearer when the same pattern is applied repeatedly (e.g. in a loop over many lines) than repeatedly passing the raw pattern string to a module-level function.
What is catastrophic backtracking, and what kind of pattern is prone to it?
It's a worst-case scenario where a regex engine's backtracking search takes exponential time relative to input length, typically caused by nested or ambiguous unbounded quantifiers (like (a+)+) applied to input that ultimately fails to match — a real performance and denial-of-service risk on untrusted input.
What's the difference between re.findall and re.finditer?
re.findall returns a list of all non-overlapping matches — plain strings (or tuples of groups, if the pattern has multiple groups). re.finditer returns an iterator of full Match objects instead, giving access to position and all groups per match, and is more memory-efficient for a large number of matches.
Write a function that extracts all hashtags (a '#' followed by one or more word characters) from a string of text, returning them without the leading '#'.
import re HASHTAG_RE = re.compile(r"#(\w+)") def extract_hashtags(text: str) -> list[str]: return HASHTAG_RE.findall(text)
The pattern #(\w+) matches a literal '#' followed by one or more word characters; because the word characters are wrapped in a capturing group, findall returns just the captured text (without the '#') for each match.
What does this code print?
'aaa' then 'no match'. {2,4} requires between 2 and 4 repetitions inclusive; fullmatch requires the entire string to match, so 'aaa' (3 a's) fits within the range and matches fully, but 'aaaaa' (5 a's) exceeds the maximum of 4 and cannot fully match even though a prefix of it would.
Write a function that normalizes all runs of whitespace in a string to a single space and strips leading/trailing whitespace, using re.sub.
import re def normalize_whitespace(text: str) -> str: return re.sub(r"\s+", " ", text).strip() re.sub(r'\s+', ' ', text) collapses every run of one or more whitespace characters (spaces, tabs, newlines) anywhere in the string into a single space;.strip() then removes any resulting leading or trailing space at the very ends of the string.
Write a function that redacts all phone numbers in the format XXX-XXX-XXXX from a string, replacing each with 'XXX-XXX-XXXX' literally, using a substitution function to log how many were redacted.
import re PHONE_RE = re.compile(r"\d{3}-\d{3}-\d{4}") def redact_phones(text: str) -> tuple[str, int]: count = 0 def replacer(m: re.Match) -> str: nonlocal count count += 1 return "XXX-XXX-XXXX" result = PHONE_RE.sub(replacer, text) return result, count
Passing a function to re.sub calls it once per match with the Match object, letting the replacer both produce the redacted text and increment a counter (via nonlocal, since count is defined in the enclosing scope) — subn could also return the count directly, but this shows the function-based substitution pattern for cases needing custom per-match logic.
import re
text = "aaa"
m = re.fullmatch(r"a{2,4}", text)
print(m.group() if m else "no match")
text2 = "aaaaa"
m2 = re.fullmatch(r"a{2,4}", text2)
print(m2.group() if m2 else "no match")