Logging
The standard library's logging module records what a program is doing at configurable severity levels, routed through loggers, handlers, and formatters, and is the production-grade alternative to scattered print statements.
The logging module is part of the Python standard library and provides a flexible system for emitting timestamped, leveled diagnostic messages from any part of a program. Rather than scattering print statements that must be manually added and removed, logging lets you leave calls in the codebase permanently, then control what actually gets recorded — and where — through configuration rather than code changes.
Why Logging Instead of print
print() always writes to standard output, has no severity levels, and offers no easy way to route output to a file, silence noisy modules, or attach structured context. The logging module solves all of these: messages carry a severity level, can be routed to multiple destinations (console, file, network) simultaneously, and can be turned on or off per module without touching source code.
Quick Start with the Root Logger
The simplest usage calls logging.basicConfig() once to configure output, then uses the module-level convenience functions.
This quick-start pattern uses the implicit 'root' logger and is fine for small scripts, but real applications should get a named logger per module instead (next section).
import logging
logging.basicConfig(level=logging.INFO)
logging.info("application starting")
logging.warning("cache miss for key=%s", "user:42")
logging.error("failed to connect to database")Loggers and Hierarchy
A Logger is the object you actually call (.info(), .warning(), etc.). Loggers are organized in a dotted hierarchy matching module names; the standard idiom is one logger per module, named after that module.
Because __name__ is the module's dotted path, getLogger(__name__) automatically places each logger correctly in the hierarchy, so configuration applied to a parent logger (e.g. 'mypackage') affects all its children ('mypackage.db', 'mypackage.api', ...) unless overridden.
# inside mypackage/db.py
import logging
logger = logging.getLogger(__name__) # e.g. "mypackage.db"
def connect():
logger.info("connecting to database")Log Levels
Each log call has a severity level; a logger (and each handler) has a threshold level, and only messages at or above that threshold are actually processed. The standard levels, from least to most severe, are:
DEBUG 10 detailed diagnostic information, useful only when debugging
INFO 20 confirmation that things are working as expected
WARNING 30 something unexpected happened, but the program continues (this is the default threshold)
ERROR 40 a serious problem prevented a function from completing
CRITICAL 50 a very serious error; the program itself may be unable to continuelogger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG) # process everything DEBUG and aboveHandlers: Where Messages Go
A Handler determines the destination for log records that pass a logger's level filter — console, a file, a rotating set of files, a network socket, or a system log. A single logger can have multiple handlers, each with its own level and formatter, so the same log call can, for example, print WARNING+ to the console while writing DEBUG+ to a file.
import logging
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
console = logging.StreamHandler()
console.setLevel(logging.WARNING)
file_handler = logging.FileHandler("app.log")
file_handler.setLevel(logging.DEBUG)
logger.addHandler(console)
logger.addHandler(file_handler)
logger.debug("only goes to app.log")
logger.warning("goes to both console and app.log")Formatters
A Formatter controls the textual layout of each log record: timestamp, logger name, level, and message, assembled from a format string using %-style placeholders drawn from LogRecord attributes.
formatter = logging.Formatter(
"%(asctime)s %(levelname)-8s %(name)s: %(message)s"
)
console.setFormatter(formatter)
file_handler.setFormatter(formatter)2024-03-14 09:15:02,481 WARNING mypackage.db: connection retry 1/3Filters
A Filter provides finer-grained control than a level threshold, deciding per-record whether it should be processed based on arbitrary logic (e.g. only messages containing a certain field, or rate-limiting repeated messages). Filters can be attached to either a Logger or a Handler.
import logging
class RequireRequestId(logging.Filter):
def filter(self, record):
return hasattr(record, "request_id")
logger.addFilter(RequireRequestId())
logger.info("handled", extra={"request_id": "abc123"}) # passes
logger.info("no context") # droppedConfiguring Logging: dictConfig
For anything beyond a small script, logging.config.dictConfig lets you define loggers, handlers, and formatters declaratively (often loaded from a JSON or YAML file), which is easier to manage and vary between environments than imperative setup code.
import logging.config
config = {
"version": 1,
"formatters": {
"standard": {"format": "%(asctime)s %(levelname)s %(name)s: %(message)s"},
},
"handlers": {
"console": {"class": "logging.StreamHandler", "formatter": "standard", "level": "INFO"},
},
"root": {"handlers": ["console"], "level": "INFO"},
}
logging.config.dictConfig(config)Exception Logging
logger.exception() (only valid inside an except block) logs at ERROR level and automatically includes the full traceback, which is the standard way to record a caught exception without losing the diagnostic detail a plain error message would discard.
logger.error('...', exc_info=True) does the same thing as logger.exception() and can be used at any level if you want a different severity than ERROR while still attaching the traceback.
import logging
logger = logging.getLogger(__name__)
try:
result = 10 / 0
except ZeroDivisionError:
logger.exception("division failed while computing result")ERROR:__main__:division failed while computing result
Traceback (most recent call last):
File "app.py", line 6, in <module>
result = 10 / 0
ZeroDivisionError: division by zeroStructured Logging Concepts
Plain-text log lines are easy for a human to read but hard for machines to query at scale. Structured logging attaches key-value context to each record — often emitted as JSON — so log aggregation systems can filter and search on fields directly instead of parsing free text.
The standard library's logging module supports attaching arbitrary fields via the extra= parameter, but producing actual JSON-formatted output typically requires a custom Formatter or a third-party library (e.g. python-json-logger); this is a concept the standard library enables rather than fully implements out of the box.
import logging
logger = logging.getLogger(__name__)
logger.info("order placed", extra={"order_id": 42, "user_id": 7, "total": 99.5})Avoid Secrets in Logs
Log output is often persisted, shipped to third-party aggregation services, and retained far longer than the request or process that generated it. Never log passwords, API keys, access tokens, full credit card numbers, or other sensitive data — including accidentally, by logging an entire object (like a request or config dict) that happens to contain a secret field.
# risky: logs the whole payload, which may contain a password field
logger.info("login attempt: %s", request.form)
# safer: log only what's needed, explicitly
logger.info("login attempt for user=%s", request.form.get("username"))Practical Usage and Best Practices
Use logging.getLogger(__name__) per module rather than the root logger directly. Use %-style lazy formatting (logger.info('value=%s', x)) rather than an f-string in the call, so the string is only formatted if the message actually passes the level filter — an f-string argument is always evaluated eagerly even when the log call is filtered out. Configure logging once, near the application's entry point, not inside library code (libraries should just call getLogger and let the application decide handlers/levels).
# wasteful: the f-string is built even if DEBUG is filtered out
logger.debug(f"payload={expensive_serialize(payload)}")
# lazy: expensive_serialize(payload) only runs if DEBUG is enabled
logger.debug("payload=%s", expensive_serialize(payload))Complete Example
import logging
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)-8s %(name)s: %(message)s",
)
logger = logging.getLogger(__name__)
def process_order(order_id: int, amount: float) -> bool:
logger.info("processing order %s for $%.2f", order_id, amount)
try:
if amount <= 0:
raise ValueError("order amount must be positive")
# ... charge payment, update inventory ...
logger.info("order %s completed", order_id)
return True
except ValueError:
logger.exception("order %s failed validation", order_id)
return False
process_order(101, 49.99)
process_order(102, -5)Mental Model and Quick Reference
Think of a log call as flowing through a pipeline: the logger checks its level, passes the record to each attached handler, each handler checks its own level and filters, and finally formats and writes the record to its destination. Configuration at any stage can stop or reshape the record without touching the original log call in your code.
logging.getLogger(__name__) -> per-module logger, named by hierarchy
logger.debug/info/warning/error/critical(msg, *args) -> emit at a level
logger.exception(msg) -> ERROR + automatic traceback (in except block)
Handler -> where records go (console, file, ...)
Formatter -> how records are rendered as text
Filter -> fine-grained per-record accept/reject
logging.config.dictConfig -> declarative configurationKey Takeaways
The logging module is standard library and replaces ad hoc print statements with leveled, configurable, permanent diagnostic calls.
Use logging.getLogger(__name__) per module; loggers form a dotted hierarchy matching package/module structure.
Levels (DEBUG, INFO, WARNING, ERROR, CRITICAL) filter which messages are processed, at both the logger and each handler.
Handlers control destination, formatters control text layout, and filters allow fine-grained per-record logic.
logger.exception() inside an except block logs the message plus the full traceback automatically.
Use lazy %-style formatting in log calls, not eagerly-evaluated f-strings, so unneeded formatting work is skipped when a level is filtered out.
Never log secrets — passwords, tokens, full card numbers — including accidentally via logging an entire object that contains one.
print(f"user {user_id} logged in") # scattered, unleveled, hard to disable logger.info("user %s logged in", user_id) print statements have no severity, can't be selectively silenced, and must be manually removed; logging calls carry a level and are controlled entirely through configuration.
logger.debug(f"state={compute_expensive_debug_dump()}") # always evaluated, even if DEBUG is off logger.debug("state=%s", compute_expensive_debug_dump()) # only formatted if DEBUG is enabled
An f-string embedded in the call is evaluated eagerly regardless of whether the message passes the level filter; %-style lazy arguments defer the expensive work until (and unless) it's actually needed.
logger.info("received request: %s", request.__dict__) # may include an auth token or password field logger.info("received request for endpoint=%s user=%s", request.path, request.user_id)
Logging an entire object or payload risks capturing sensitive fields (tokens, passwords) that get persisted and shipped to log aggregators; log only the specific fields needed.
try: risky() except Exception as e: logger.error(str(e)) # loses the traceback, only keeps the message try: risky() except Exception: logger.exception("risky() failed") # includes the full traceback automatically logger.error(str(e)) discards the traceback entirely; logger.exception() (or error with exc_info=True) preserves it, which is essential for diagnosing where and how the failure occurred.
Why is logging.getLogger(__name__) preferred over calling logging.info() directly (the root logger) in application code?
getLogger(__name__) creates a logger named after the current module, placing it correctly in the dotted hierarchy so configuration (levels, handlers) can be applied per-module or per-package rather than globally to everything.
List the five standard logging levels in increasing order of severity and state which is the default threshold.
DEBUG, INFO, WARNING, ERROR, CRITICAL — WARNING is the default threshold, meaning INFO and DEBUG messages are suppressed unless the level is explicitly lowered.
What is the difference between a Handler and a Formatter?
A Handler decides where a log record goes (console, file, network, etc.) and applies its own level filter; a Formatter decides how that record is rendered as text (what fields appear and in what layout).
Why should log calls use %-style arguments (logger.info('x=%s', x)) instead of an f-string built inline?
The %-style arguments are only formatted into a string if the record actually passes the logger's and handler's level filters; an f-string is evaluated eagerly every time the call executes, wasting work when the message would be filtered out anyway.
What does logger.exception() do that logger.error() does not, by default?
logger.exception() logs at ERROR level and automatically attaches the current exception's traceback; logger.error() alone does not include traceback information unless you pass exc_info=True explicitly.
Why is it risky to log an entire request or config object with a single log call?
Such objects can contain sensitive fields (passwords, API keys, tokens) that would then be persisted in log files and potentially shipped to external log aggregation services, creating a security/privacy exposure.
What problem does structured logging (attaching key-value fields via extra=) solve compared to plain text log messages?
It lets log aggregation and search systems filter and query on specific fields directly (e.g. order_id=42) instead of parsing free-form text, making large volumes of logs practically searchable.
Write code that configures logging with basicConfig at INFO level, gets a module logger, and logs an INFO message when a function 'save_file' is called with a filename.
import logging logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__) def save_file(filename: str) -> None: logger.info("saving file: %s", filename) # ... actual save logic ...
basicConfig sets up a default handler and level once; the module logger then emits an INFO-level record whenever save_file runs, visible because INFO meets the configured threshold.
Which of these two log calls actually produces visible output, and why?
Only the warning() call produces output. The configured level is WARNING, so INFO-level records (which are less severe than WARNING) are filtered out and never reach a handler, while WARNING and above pass through.
Write a try/except block around a call to json.loads(text) that logs the failure with logger.exception() and re-raises a custom ParseError.
import json import logging logger = logging.getLogger(__name__) class ParseError(Exception): pass def parse_config(text: str) -> dict: try: return json.loads(text) except json.JSONDecodeError: logger.exception("failed to parse configuration text") raise ParseError("invalid configuration format") logger.exception() records the JSONDecodeError with its full traceback for later debugging before the code raises a cleaner, application-specific ParseError to the caller.
Does this warning message get processed, assuming no explicit level was set on the 'app.payments' logger itself and default propagation is in effect?
No — a logger with no explicit level set inherits its effective level from the nearest ancestor that has one, here 'app' at ERROR. Since WARNING is less severe than ERROR, the message is filtered out at the logger level before it ever reaches a handler.
import logging
logging.basicConfig(level=logging.WARNING)
logger = logging.getLogger(__name__)
logger.info("starting up")
logger.warning("low disk space")import logging
logger = logging.getLogger("app.payments")
parent = logging.getLogger("app")
parent.setLevel(logging.ERROR)
logger.warning("card declined")