os & os.path

The standard library's low-level interface to the operating system: environment variables, the working directory, filesystem operations, and string-based path manipulation.

The os module is Python's thin wrapper around operating-system services: environment variables, processes, and file/directory operations. os.path is its companion sub-module for manipulating filesystem paths as plain strings. Together they were the standard way to touch the filesystem for decades, and you will still meet them constantly in existing code, tutorials, and lower-level scripts — even though modern Python code increasingly reaches for pathlib instead.

What Is os / os.path

os exposes operating-system functionality that isn't tied to any one file: reading environment variables, getting the current process id, changing the working directory, listing directories, renaming and removing files. os.path is imported automatically as part of os (accessible as os.path) and focuses specifically on path string manipulation — joining components, splitting extensions, checking existence — without actually touching the disk for most of its functions (except things like exists, isfile, getsize, which do query the filesystem).

import os
import os.path

print(os.getcwd())          # current working directory, e.g. '/home/user/project'
print(os.name)              # 'posix' on Linux/macOS, 'nt' on Windows
print(os.path.join("data", "input.csv"))  # 'data/input.csv' (or 'data\\input.csv' on Windows)

Why It Matters

Almost every real program eventually reads configuration from the environment, needs to know where it's running from, or has to walk a directory tree. os is the foundation those operations are built on — even libraries that offer nicer interfaces (like pathlib) ultimately call into os and the C standard library underneath. Understanding os also means you can read the huge amount of existing Python code, especially older codebases and scripts, that still uses it directly.

The Environment: os.environ and Friends

os.environ is a mapping (dict-like, actually an os._Environ object) representing the process's environment variables. Reading and writing it is the standard way to access configuration passed in from the shell, container, or CI system.

Environment variable values are always strings. If you need an int, float, or bool, you must convert it yourself — os.environ.get("PORT", "8000") returns the string "8000", not the integer 8000.

import os

# Reading — prefer .get with a default over os.environ[key], which raises KeyError
db_host = os.environ.get("DB_HOST", "localhost")
port = int(os.environ.get("DB_PORT", "5432"))

# os.getenv is equivalent to os.environ.get
api_key = os.getenv("API_KEY")   # None if not set

# Writing — only affects this process and its children, never the parent shell
os.environ["DEBUG"] = "1"

# Checking presence
if "CI" in os.environ:
    print("Running in a CI environment")

Working Directory

Every process has a current working directory, which relative paths are resolved against. os.getcwd() returns it; os.chdir(path) changes it for the current process (affecting subsequent relative-path operations, including in any modules that use relative paths).

Changing the working directory is process-global state — every part of your program is affected, which makes it easy to introduce subtle bugs in larger programs. Prefer passing explicit absolute paths (e.g. built with pathlib or os.path.abspath) over relying on chdir.

import os

start = os.getcwd()
os.chdir("/tmp")
print(os.getcwd())   # /tmp
os.chdir(start)      # change back

Listing and Walking Directories

os.listdir(path) returns filenames (not full paths) in a directory. os.scandir(path) is the more efficient modern replacement — it returns os.DirEntry objects that cache type/stat information, avoiding extra system calls when you also need to know if an entry is a file or directory. os.walk(top) recursively descends a directory tree, yielding (dirpath, dirnames, filenames) for each directory visited.

import os

# Simple listing
for name in os.listdir("."):
    print(name)

# Efficient listing with type info, no extra stat() calls
with os.scandir(".") as entries:
    for entry in entries:
        kind = "dir" if entry.is_dir() else "file"
        print(f"{entry.name}: {kind}")

# Recursive walk
for dirpath, dirnames, filenames in os.walk("."):
    for fname in filenames:
        print(os.path.join(dirpath, fname))

Filesystem Operations

os provides direct wrappers for common filesystem actions: creating and removing directories, renaming, removing files, and querying metadata.

os.rmdir only removes empty directories; deleting a whole populated tree is the job of shutil.rmtree (a related concept, not covered here).

import os

os.mkdir("reports")                 # create one directory (fails if parent missing)
os.makedirs("reports/2024/q1", exist_ok=True)  # create nested dirs, ok if they exist

os.rename("old_name.txt", "new_name.txt")
os.remove("new_name.txt")           # delete a file

os.rmdir("reports/2024/q1")         # remove an EMPTY directory only
# For a non-empty tree, use shutil.rmtree (a related concept, in the shutil module)

info = os.stat("reports")
print(info.st_size, info.st_mtime)  # size in bytes, last-modified timestamp

Process-Related Basics

os also exposes basic information about the running process and lets you spawn new ones at a low level.

os.system and the older os.popen are legacy; for running external commands with proper argument handling and output capture, the subprocess module is the modern, safer standard-library tool.

import os

print(os.getpid())    # this process's id
print(os.getppid())   # parent process's id

exit_code = os.system("echo hello")  # runs a shell command; returns exit status
# For anything beyond trivial commands, prefer the subprocess module (a related
# concept) — it gives you captured output, timeouts, and no shell-injection risk.

os.path: Splitting and Joining

os.path treats paths as plain strings and provides functions to combine, split, and inspect them in an OS-appropriate way (using / on POSIX, \ on Windows automatically).

import os.path

p = os.path.join("data", "2024", "input.csv")
print(p)                              # data/2024/input.csv

print(os.path.dirname(p))             # data/2024
print(os.path.basename(p))            # input.csv
print(os.path.splitext(p))            # ('data/2024/input', '.csv')
print(os.path.split(p))               # ('data/2024', 'input.csv')

print(os.path.abspath("input.csv"))   # absolute path resolved against cwd
print(os.path.normpath("a/./b/../c")) # a/c  — collapses . and ..

os.path: Existence and Type Checks

These functions actually query the filesystem, unlike the pure string-manipulation functions above.

Checking os.path.exists() and then opening the file is inherently race-prone (the file could vanish between the check and the open, especially with concurrent processes). Prefer opening directly inside a try/except FileNotFoundError block when the outcome matters — a pattern often summarized as "easier to ask forgiveness than permission" (EAFP).

import os.path

print(os.path.exists("input.csv"))    # True/False, file or directory
print(os.path.isfile("input.csv"))    # True only if it's a regular file
print(os.path.isdir("data"))          # True only if it's a directory
print(os.path.islink("shortcut"))     # True if it's a symbolic link
print(os.path.getsize("input.csv"))   # size in bytes (raises if missing)

Cross-Platform Path Handling

os.path automatically adapts its separator (os.sep) and join behavior to the running platform, so os.path.join("data", "file.txt") produces data/file.txt on Linux/macOS and data\\file.txt on Windows. Always build paths with os.path.join (or pathlib) rather than hardcoding / or \ — hardcoded separators silently break on the other platform.

import os

print(os.sep)        # '/' on POSIX, '\\' on Windows
print(os.linesep)    # '\n' on POSIX, '\r\n' on Windows
print(os.pathsep)    # ':' on POSIX, ';' on Windows — used to split PATH-style variables

Why pathlib Is Often Preferred for New Code

pathlib (a related, newer standard-library module) represents paths as Path objects instead of plain strings, with the same operations available as methods and operators. It's generally preferred for new code because it's more readable, harder to misuse, and unifies string-manipulation with filesystem-querying under one consistent object.

os and os.path are not deprecated — they remain the right tool when you need process/environment features (os.environ, os.getpid), or when interfacing with older APIs that expect string paths. pathlib.Path objects convert to strings with str(path) or os.fspath(path), and most standard library functions accept Path objects directly.

# os.path style
import os.path
p = os.path.join("data", "2024", "input.csv")
if os.path.exists(p) and os.path.isfile(p):
    size = os.path.getsize(p)

# pathlib style — same logic, reads more directly
from pathlib import Path
p = Path("data") / "2024" / "input.csv"
if p.is_file():
    size = p.stat().st_size

Common Patterns in Practice

A few idioms show up repeatedly in real scripts that still use os/os.path.

import os

# Locate a file relative to the currently running script, not the cwd
script_dir = os.path.dirname(os.path.abspath(__file__))
config_path = os.path.join(script_dir, "config.json")

# Build a config dict from environment variables with typed defaults
config = {
    "debug": os.environ.get("DEBUG", "0") == "1",
    "workers": int(os.environ.get("WORKERS", "4")),
}

# Ensure an output directory exists before writing into it
os.makedirs("output", exist_ok=True)

Key Takeaways

os covers environment variables, working directory, and process/filesystem operations; os.path covers path string manipulation and existence/type queries.

os.environ values are always strings — convert to int/float/bool explicitly.

Use os.path.join (or pathlib) rather than hardcoding path separators, for cross-platform correctness.

os.scandir and os.walk are the efficient tools for listing and recursively traversing directories.

Checking existence then opening is race-prone; prefer try/except around the actual operation when correctness under concurrency matters.

For new code, pathlib.Path is usually preferred over os.path for readability and safety, while os itself remains necessary for environment and process features.

path = "data" + "/" + "file.txt" path = os.path.join("data", "file.txt")

Hardcoded separators break on Windows and mishandle trailing slashes; os.path.join (or pathlib) adapts to the platform correctly.

port = os.environ["PORT"] # int comparison later fails / KeyError if unset port = int(os.environ.get("PORT", "8000"))

Environment values are always strings and may be missing; use .get with a default and convert types explicitly.

if os.path.exists(path): f = open(path) # could still fail: race condition or permissions try: f = open(path) except FileNotFoundError:...

The file can disappear between the check and the open; handling the exception directly is more robust than a check-then-act pattern.

os.rmdir("build") # raises OSError if build/ is not empty import shutil shutil.rmtree("build") os.rmdir only removes empty directories; deleting a populated tree requires shutil.rmtree.

What is the difference between os.listdir() and os.scandir()?

os.listdir() returns a list of plain filename strings. os.scandir() returns an iterator of os.DirEntry objects that cache file-type and stat information, avoiding extra system calls when you need to check is_file()/is_dir() for each entry — more efficient for that common case.

Why should you avoid relying on os.environ values as anything but strings?

Environment variables are always passed as strings by the operating system; Python does not infer types. Code that treats a value like '8000' as an int without explicit conversion will fail arithmetic or comparisons.

What does os.path.join do differently from simple string concatenation with '/'?

It uses the correct separator for the current platform (os.sep) and correctly handles edge cases like an already-absolute later component (which discards earlier components) — plain concatenation hardcodes one separator and can produce malformed paths on other platforms.

Why is checking os.path.exists() before opening a file considered a weaker pattern than a try/except around the open() call?

It's a check-then-act race condition: the filesystem can change between the check and the subsequent operation (deleted, permissions changed, replaced by another process). Wrapping the actual operation in try/except handles the failure atomically at the point it matters.

What does os.makedirs(path, exist_ok=True) do that os.mkdir(path) does not?

os.makedirs creates all missing intermediate directories in the path (like 'mkdir -p'), whereas os.mkdir creates only the final directory and fails if its parent doesn't exist. exist_ok=True additionally suppresses the error if the target directory already exists.

Give one concrete reason to prefer pathlib over os.path in new code, and one reason you might still need os directly.

pathlib.Path offers a more readable, object-oriented API (path / 'sub' instead of os.path.join, path.is_file() instead of os.path.isfile(path)) that's harder to misuse. You still need os directly for process/environment features like os.environ, os.getpid(), or os.system() that pathlib doesn't provide.

Write a function that returns True if a given path exists and is a non-empty regular file, using os.path.

import os.path def is_nonempty_file(path: str) -> bool: return os.path.isfile(path) and os.path.getsize(path) > 0 os.path.isfile() confirms it's a regular file (not a directory or missing), and os.path.getsize() > 0 confirms it has content. Short-circuiting with 'and' avoids calling getsize on a path that doesn't exist as a file.

What happens when this code runs?

TypeError: can only concatenate str (not 'int') to str. os.environ.get('LEVEL', '1') returns the string '3', not the integer 3 — it must be converted with int() before arithmetic.

Write a function using os.walk that returns the total size in bytes of all files under a directory tree.

import os def total_size(root: str) -> int: total = 0 for dirpath, _dirnames, filenames in os.walk(root): for name in filenames: full_path = os.path.join(dirpath, name) total += os.path.getsize(full_path) return total os.walk yields (dirpath, dirnames, filenames) for every directory in the tree; joining dirpath with each filename gives a full path suitable for os.path.getsize, and the sizes are accumulated across the whole traversal.

import os
os.environ["LEVEL"] = "3"
print(os.environ.get("LEVEL", "1") + 1)