HTTP Client
Making HTTP requests from Python: the standard-library urllib.request/http.client tools, the essentials of HTTP itself, and why the third-party requests library became the ecosystem's de facto standard.
Fetching a web resource is one of the most common tasks in real programs. Python ships everything needed to do it — urllib.request for a simple high-level call, http.client for lower-level control — while the third-party requests library is not part of the standard library but is worth knowing because it is the tool most Python code actually uses in practice.
HTTP Basics: Requests, Responses, and Statelessness
HTTP (HyperText Transfer Protocol) is a request/response protocol: a client sends a request (a method, a URL, headers, and optionally a body) and the server sends back a response (a status code, headers, and optionally a body). Each request is independent — HTTP itself is stateless; state across requests (like login sessions) is layered on top using mechanisms such as cookies or tokens.
Common HTTP methods carry different intent: GET retrieves a resource (should have no side effects), POST submits data (often creates something), PUT replaces a resource, PATCH partially updates one, DELETE removes one. This topic focuses on issuing these requests from Python, not on server-side HTTP handling.
urllib.request: The Standard-Library HTTP Client
urllib.request is part of the standard library and handles the common case of issuing an HTTP(S) request and reading the response, with no extra dependency required.
urlopen() returns a file-like response object; read() returns bytes, which you decode yourself if you need text. HTTPError (raised for 4xx/5xx status codes) is itself a subclass of URLError, so ordering the except clauses matters.
Always pass timeout= — without it, urlopen() can block indefinitely against a slow or unresponsive server.
from urllib.request import urlopen, Request
from urllib.error import URLError, HTTPError
req = Request(
"https://api.example.com/status",
headers={"User-Agent": "sampleapp-demo/1.0"},
)
try:
with urlopen(req, timeout=5) as response:
status = response.status # e.g. 200
body = response.read() # bytes
content_type = response.headers.get("Content-Type")
print(status, content_type, len(body))
except HTTPError as e:
print("HTTP error:", e.code, e.reason) # server responded with 4xx/5xx
except URLError as e:
print("Network/connection error:", e.reason) # DNS failure, refused connection, etc.Building Requests: Method, Headers, and Body
Request() lets you customize the method, add headers, and attach a body for POST/PUT-style requests.
Passing data= implicitly makes the request a POST unless method= overrides it; the body must be bytes, so text/JSON is encoded before being handed to Request().
import json
from urllib.request import Request, urlopen
payload = json.dumps({"name": "Ada", "role": "engineer"}).encode("utf-8")
req = Request(
"https://api.example.com/users",
data=payload,
method="POST",
headers={
"Content-Type": "application/json",
"Accept": "application/json",
},
)
with urlopen(req, timeout=5) as resp:
result = json.loads(resp.read())
print(result)http.client: One Level Lower
http.client is the lower-level module urllib.request is itself built on. It gives you explicit control over the connection (useful for keep-alive across multiple requests, or protocol-level debugging) at the cost of more manual bookkeeping.
You must read the full response body (or close the connection) before issuing another request on the same HTTPConnection object, and you are responsible for closing the connection yourself.
Most application code should reach for urllib.request (or the third-party requests) rather than http.client directly; drop to http.client only when you need connection-level control that the higher-level APIs don't expose.
import http.client
conn = http.client.HTTPSConnection("api.example.com", timeout=5)
try:
conn.request("GET", "/status", headers={"User-Agent": "sampleapp-demo/1.0"})
resp = conn.getresponse()
print(resp.status, resp.reason)
body = resp.read()
finally:
conn.close()Status Codes
HTTP status codes are grouped by their first digit: 1xx informational, 2xx success, 3xx redirection, 4xx client error, 5xx server error. urlopen()/http.client follow redirects (3xx) automatically for GET/HEAD by default in urllib.request, but raise HTTPError for 4xx and 5xx responses rather than returning them silently.
200 OK - request succeeded
201 Created - resource created (common after POST)
204 No Content - success, no response body
301/302 Redirect - resource moved (permanent/temporary)
400 Bad Request - malformed request
401 Unauthorized - missing/invalid credentials
403 Forbidden - authenticated but not allowed
404 Not Found - resource doesn't exist
429 Too Many Requests - rate limited
500 Internal Server Error
503 Service Unavailablefrom urllib.error import HTTPError
from urllib.request import urlopen
try:
urlopen("https://api.example.com/missing", timeout=5)
except HTTPError as e:
print(e.code) # 404
print(e.reason) # "Not Found"
print(e.read()) # the error response body, if anyHeaders: Sending and Reading
Request headers configure the request itself (authentication, content negotiation, custom metadata); response headers describe the returned resource (content type, length, caching directives). Header names are case-insensitive.
Always send a descriptive User-Agent for scripts making outbound requests — many servers reject or rate-limit requests with a missing or generic default user agent.
from urllib.request import Request, urlopen
req = Request(
"https://api.example.com/data",
headers={
"Authorization": "Bearer <token>",
"Accept": "application/json",
"User-Agent": "sampleapp-demo/1.0",
},
)
with urlopen(req, timeout=5) as resp:
print(resp.headers.get("Content-Type"))
print(resp.headers.get("content-type")) # same value; header lookup is case-insensitive
for name, value in resp.headers.items():
print(name, ":", value)Timeouts
Every HTTP call over the network can hang: DNS can be slow, the TCP handshake can stall, or the server can accept the connection and never respond. Always bound requests with an explicit timeout.
urlopen()'s timeout bounds the whole operation (connect + read); http.client's HTTPConnection(..., timeout=...) does the same at the connection level.
from urllib.request import urlopen
from urllib.error import URLError
try:
with urlopen("https://api.example.com/slow", timeout=3) as resp:
data = resp.read()
except URLError as e:
if isinstance(e.reason, TimeoutError):
print("Request timed out")
else:
print("Other network error:", e.reason)requests: The Third-Party Ecosystem Standard
requests is a third-party package (install with pip install requests) — it is NOT part of the Python standard library, unlike urllib.request and http.client covered above.
Despite not being built in, requests is worth knowing because it is the de facto standard HTTP client across the Python ecosystem, prized for a much more ergonomic API: automatic JSON encoding/decoding, connection pooling via Session objects, simpler error handling, and less boilerplate for common tasks.
Compare this to the equivalent urllib.request code above: requests.post(json=...) handles encoding and the Content-Type header automatically, and .json() parses the response body for you.
import requests # third-party package, not stdlib
resp = requests.get(
"https://api.example.com/status",
headers={"Accept": "application/json"},
timeout=5,
)
resp.raise_for_status() # raises requests.HTTPError on 4xx/5xx
data = resp.json() # parses JSON body directly
print(resp.status_code, data)
# POST with a JSON body, no manual encoding needed
resp = requests.post(
"https://api.example.com/users",
json={"name": "Ada", "role": "engineer"},
timeout=5,
)Choosing Between urllib.request and requests
Use urllib.request when you want zero extra dependencies (e.g. a small script, a library that shouldn't force a dependency on its users, or a constrained deployment environment). Use requests for application code where ergonomics matter and adding a well-established dependency is acceptable — most real-world Python HTTP code in the wild uses it.
Feature urllib.request (stdlib) requests (third-party)
-----------------------------------------------------------------------
Install needed no yes (pip install requests)
JSON encode/decode manual built in (json=, .json())
Session/connection pooling manual requests.Session()
Error model HTTPError/URLError raise_for_status() + exceptions
Boilerplate more lessPractical Usage: A Small Reusable Fetch Helper
Wrapping the low-level exceptions into a single application-specific error keeps calling code simple while still preserving the original exception via from e.
import json
from urllib.request import Request, urlopen
from urllib.error import HTTPError, URLError
def fetch_json(url: str, timeout: float = 5.0) -> dict:
"""Fetch a URL and parse a JSON response, using only the standard library."""
req = Request(url, headers={"Accept": "application/json", "User-Agent": "sampleapp/1.0"})
try:
with urlopen(req, timeout=timeout) as resp:
return json.loads(resp.read())
except HTTPError as e:
raise RuntimeError(f"HTTP {e.code} fetching {url}: {e.reason}") from e
except URLError as e:
raise RuntimeError(f"Network error fetching {url}: {e.reason}") from eEdge Cases
Redirect loops: urllib.request follows redirects automatically but will give up after too many hops, raising an error. Encoding: response bodies are always bytes — decode using the charset declared in the Content-Type header (or a sensible default like UTF-8) rather than assuming text. Large responses: read() loads the entire body into memory; for large downloads, stream and read in chunks instead.
Reading in fixed-size chunks avoids buffering an entire large download in memory at once.
from urllib.request import urlopen
with urlopen("https://example.com/large-file.zip", timeout=10) as resp:
with open("large-file.zip", "wb") as f:
while chunk := resp.read(65536):
f.write(chunk)Key Takeaways
HTTP is a stateless request/response protocol; methods (GET, POST, ...) express intent and status codes report the outcome.
urllib.request (and the lower-level http.client) are standard-library tools requiring no extra install.
requests is a third-party package, not part of the standard library, but is the ecosystem's practical default for its ergonomics.
Always pass an explicit timeout; an unbounded HTTP call can hang your program indefinitely.
4xx/5xx responses raise HTTPError in urllib (or trigger raise_for_status() in requests) rather than silently returning as success.
Response bodies come back as bytes — decode explicitly based on the declared content type/charset.
from urllib.request import urlopen resp = urlopen(url) # no timeout from urllib.request import urlopen resp = urlopen(url, timeout=5)
Without a timeout, a hung or slow server can block your program indefinitely.
resp = urlopen(url) data = resp.read() text = data # treated as str resp = urlopen(url) data = resp.read() text = data.decode(resp.headers.get_content_charset() or "utf-8") urlopen().read() returns bytes, not str; decode explicitly using the response's declared charset.
import requests resp = requests.get(url) data = resp.json() # assumes success import requests resp = requests.get(url, timeout=5) resp.raise_for_status() data = resp.json()
A 4xx/5xx response can still return a body that .json() will happily (mis)parse; raise_for_status() ensures errors are caught before you trust the payload.
requests is third-party; code and documentation should make the dependency explicit rather than implying it ships with Python.
Is requests part of the Python standard library? What should you use instead if you cannot add a dependency?
No, requests is a third-party package that must be installed separately. Without it, use the standard-library urllib.request (or the lower-level http.client) instead.
What is the practical difference between urlopen() raising HTTPError versus returning a response object with a status attribute?
urllib.request raises HTTPError for 4xx/5xx status codes rather than returning them as a normal response, so error handling for bad HTTP status codes must use a try/except rather than checking a status field on a returned object (in contrast to requests, which returns a response object and requires calling raise_for_status() to get the same exception-based behavior).
Why should you always pass an explicit timeout to an HTTP request?
Without one, the call can block indefinitely if DNS is slow, the TCP handshake stalls, or the server accepts the connection but never sends a response, effectively hanging the program.
Why is response.read() bytes rather than str, and how do you get text?
HTTP bodies are raw bytes with encoding described (if at all) by headers like Content-Type's charset parameter; you must decode explicitly using that declared (or a sensible default) encoding rather than assuming str.
What HTTP status code range indicates a problem on the client's side of the request, and which range indicates a server-side problem?
4xx codes indicate a client error (e.g. 400 Bad Request, 404 Not Found); 5xx codes indicate a server-side error (e.g. 500 Internal Server Error, 503 Service Unavailable).
When would you reach for http.client instead of urllib.request?
When you need lower-level control over the connection itself — such as reusing one connection across multiple requests explicitly, or inspecting/controlling protocol details that the higher-level urlopen() API hides.
Write a standard-library-only function get_status_code(url) that returns the HTTP status code for a GET request, returning the code even for 4xx/5xx responses (not raising).
from urllib.request import urlopen from urllib.error import HTTPError def get_status_code(url: str, timeout: float = 5.0) -> int: try: with urlopen(url, timeout=timeout) as resp: return resp.status except HTTPError as e: return e.code
Success responses come back through urlopen() normally, giving resp.status; error responses (4xx/5xx) raise HTTPError instead, which itself carries the status code as e.code, so both paths must be handled to always get a code.
What type does this print, and why does it matter for further processing?
It prints <class 'bytes'>. HTTP response bodies are raw bytes; you must explicitly .decode() them (using the appropriate charset) before treating them as text, and JSON bodies must be parsed explicitly (e.g. with json.loads).
Using the third-party requests library, write a function post_json(url, payload) that POSTs a dict as JSON, raises on HTTP errors, and returns the parsed JSON response.
import requests # third-party, not stdlib def post_json(url: str, payload: dict, timeout: float = 5.0) -> dict: resp = requests.post(url, json=payload, timeout=timeout) resp.raise_for_status() return resp.json() requests.post(json=payload) serializes the dict and sets the Content-Type header automatically; raise_for_status() converts a 4xx/5xx response into a raised exception;.json() parses the response body back into Python data.
from urllib.request import urlopen
resp = urlopen("https://example.com", timeout=5)
print(type(resp.read()))