Robot Perception, Localisation & Learning

How robots see, know where they are and learn. Learn robot sensors (cameras, depth cameras, lidar, IMU, force/torque, tactile), the pinhole camera model and intrinsics/extrinsics, camera calibration, stereo and depth, point clouds and pose estimation (ICP, 6-D pose), object detection and grasp detection with deep learning, visual servoing, localisation and SLAM (odometry, particle filters, EKF, graph SLAM), sensor fusion, and robot learning — imitation learning, reinforcement learning (Q-learning, policy gradients), sim-to-real and robot foundation models — with Python projecting a 3-D point to pixels and training Q-learning on a simple task.

Robots in unstructured environments must perceive objects, localise themselves and adapt. Perception turns raw sensor data into useful estimates — the pose of a part, a map of a warehouse, the location of a grasp.

Learning lets robots acquire skills from demonstrations and experience instead of explicit programming — the frontier of modern robotics.

An analogy

A camera is like a pinhole in a dark box: every 3-D point projects to one spot on the back wall. Calibration measures the box's exact size and pinhole position so the robot can turn spots back into 3-D directions — and with depth, into 3-D positions.

Robot sensors

Sensing options:

Vision for manipulation

Calibrate intrinsics (checkerboard, Zhang's method) and extrinsics (hand–eye). Pipeline: detect/segment objects (classical features or deep learning — YOLO, Mask R-CNN), estimate 6-D pose (template matching, point-cloud registration with ICP, learned pose estimators), compute grasps (analytic or learned grasp networks), then plan motion. Visual servoing closes the loop directly on image features for precise alignment.

Localisation and SLAM

Dead reckoning (odometry, IMU) drifts; correcting it needs landmarks or maps. Localisation in a known map: particle filters (AMCL) or EKF. SLAM builds the map and localises simultaneously: EKF-SLAM, graph-based SLAM (pose graphs with loop closures, e.g. Cartographer, ORB-SLAM, RTAB-Map). Sensor fusion (Kalman filters) combines IMU, wheel odometry, GPS, lidar and vision.

Robot learning

Imitation learning / behaviour cloning learns policies from demonstrations (teleoperation, kinesthetic teaching); diffusion policies and action-chunking transformers produce smooth multimodal actions. Reinforcement learning optimises reward via trial and error (PPO, SAC), mostly in simulation (Isaac, MuJoCo) with domain randomisation for sim-to-real transfer. Vision-language-action foundation models (RT-2, π0, OpenVLA) generalise across tasks and robots. Safety, sample efficiency and evaluation remain key challenges.

Formulas

Worked example: Projection

Given: f 800 px, c (320, 240), point (0.2, 0.1, 2.0) m

Find: (u, v).

Answer: (400, 280)

Worked example: Stereo depth

Given: f 700 px, baseline 0.12 m, disparity 42 px

Find: Z.

Answer: 2.0 m

Worked example: Q-learning value

Given: γ 0.9, reward 1 at goal, 3 steps away

Find: Optimal Q.

Answer: 0.81

Camera projection and Q-learning

import numpy as np
# Robot perception: pinhole camera projection, and tabular Q-learning on a 1-D corridor
K = np.array([[800, 0, 320], [0, 800, 240], [0, 0, 1]], float)   # intrinsics (pixels)
P_cam = np.array([0.10, -0.05, 1.0])                              # point 1 m in front of camera
uvw = K @ P_cam
print(f"pixel coordinates u = {uvw[0]/uvw[2]:.1f}, v = {uvw[1]/uvw[2]:.1f}")
print(f"one pixel at 1 m spans {1/800*1000:.2f} mm")
# Q-learning: 6 cells, reward +1 at the right end, actions 0 = left, 1 = right
rng = np.random.default_rng(0)
Q = np.zeros((6, 2)); alpha, gamma, eps = 0.5, 0.9, 0.2
for episode in range(200):
    s = 0
    while s != 5:
        a = rng.integers(2) if rng.random() < eps else int(np.argmax(Q[s]))
        s2 = max(0, s - 1) if a == 0 else s + 1
        r = 1.0 if s2 == 5 else 0.0
        Q[s, a] += alpha * (r + gamma * np.max(Q[s2]) - Q[s, a])
        s = s2
print("learned values of moving right:", np.round(Q[:5, 1], 3).tolist())
print("greedy policy:", ["R" if np.argmax(Q[s]) == 1 else "L" for s in range(5)])
# pixel coordinates u = 400.0, v = 200.0
# one pixel at 1 m spans 1.25 mm
# learned values of moving right: [0.656, 0.729, 0.81, 0.9, 1.0]
# greedy policy: ['R', 'R', 'R', 'R', 'R']

A point 10 cm right and 5 cm up at 1 m lands 80 px right and 40 px above the image centre; at this distance each pixel covers 1.25 mm, limiting how precisely vision can locate parts. Q-learning discovers the optimal policy (always move right) purely from rewards; the learned values follow γ^distance (0.9⁴ = 0.656), exactly the discounted return. Real robot RL uses neural networks and simulation because physical trials are slow and risky.

Common mistakes

Applications

Key takeaways

Going further