Course Overview

What this CUDA course covers, how to use it, prerequisites.

A self-contained, hands-on course covering GPU programming with NVIDIA CUDA C/C++: GPU architecture, the thread hierarchy, memory management, the memory hierarchy, shared memory and tiling, synchronization and atomics, warp-level programming, parallel patterns (reduction, scan, histogram, sort, stencil), streams and concurrency, unified memory, cooperative groups, CUDA graphs, performance profiling and optimization, Tensor Cores, the CUDA libraries (cuBLAS, cuFFT, Thrust, CUB, cuDNN, NCCL, CUTLASS), multi-GPU programming, modern Hopper/Blackwell features, PTX, debugging, and Python GPU programming. Every chapter has runnable examples and exercises.

Course Map

Part I — Foundations Introduction to GPU Computing & CUDA Installation, Toolkit & nvcc Your First Kernel

The Thread Hierarchy: Grids, Blocks, Threads Memory Management Basics Device Query & Compute Capability Part II — Core CUDA Programming The Memory Hierarchy Global Memory & Coalescing

Shared Memory & Tiling (Matrix Multiplication) Synchronization & Atomic Operations Parallel Patterns I: Reduction & Scan

Parallel Patterns II: Histogram, Stencil, Convolution, Sort, Compaction Constant, Texture & Surface Memory

Streams, Events & Asynchronous Execution Pinned Memory & Data Transfer Optimization Occupancy & Launch Configuration Part III — Advanced CUDA

Performance Analysis: Nsight Systems, Nsight Compute, Roofline

Stream-Ordered Memory Allocation & Memory Pools Mixed Precision & Tensor Cores

Modern Architecture Features: Async Copy, TMA, Clusters, Distributed Shared Memory CUDA C++: Templates, Lambdas, libcu++ Multi-GPU Programming: P2P, NCCL, MPI

Debugging & Correctness: cuda-gdb, compute-sanitizer PTX, SASS, Inline Assembly & Runtime Compilation (NVRTC)

Interoperability: Graphics, Driver API, IPC, MPS/MIG Python & Other Languages: CuPy, Numba, CUDA Python, PyTorch Extensions, Triton

Portability: HIP, SYCL, OpenACC, Standard Parallelism Part IV — Applications & Real-World CUDA Application Case Studies

Appendices: Cheat Sheet, API Reference, Architecture Table, Resources, Study Plan