GPU Programming

Opens in a new tab.

GPU programming is the practice of shaping work so many lightweight execution lanes can run related operations in parallel while data moves through a constrained memory hierarchy. In modern AI systems, it is where linear algebra, training neural networks, compiler behavior, memory layout, profiling, and hardware scheduling meet.

Most practitioners do not start by writing assembly or custom kernels. They begin with high-level frameworks, measure where the bottlenecks are, then move downward only when the model, data movement, deployment target, or cost envelope makes the abstraction leak.

This page treats GPU programming as a knowledge-graph bridge between mathematical operations, hardware constraints, software libraries, source data, model artifacts, and performance evidence.

GPU programming covers the languages, libraries, runtimes, kernels, compilers, tools, and mental models used to run general-purpose computation on graphics processors and related accelerators.

The central question is not "can this run on a GPU?" but "does the work have enough parallelism, regularity, arithmetic intensity, and data locality to justify the transfer, launch, synchronization, and debugging costs?"

A GPU is built to keep many operations in flight. It works best when thousands or millions of similar elements can be processed with predictable memory access and limited branch divergence. The programmer's job is to expose parallelism without feeding the hardware one tiny task at a time.

The practical model has four layers:

• host code prepares data, launches kernels, schedules copies, and orchestrates libraries;
• device code runs the parallel work;
• memory hierarchy determines what data is cheap or expensive to reach;
• profiling evidence decides whether the program is compute-bound, memory-bound, launch-bound, or synchronization-bound.

That model connects GPU programming to rules of thumb: intuition is useful only when it points to measurement.

Important GPU concepts include:

• kernel: a function launched across many elements or threads;
• grid and block: CUDA's execution grouping vocabulary;
• workgroup: a common vocabulary in portable GPU APIs;
• warp or wavefront: a group of lanes scheduled together;
• occupancy: how many warps or wavefronts can be resident relative to hardware limits;
• coalescing: arranging memory access so adjacent lanes read adjacent memory;
• shared or local memory: programmer-managed memory close to the execution units;
• register pressure: per-thread storage that can limit occupancy;
• divergence: lanes in a group following different control paths;
• arithmetic intensity: work done per byte moved.

These terms should be stored as graph anchors because they explain why a mathematically simple operation can be fast, slow, portable, or fragile depending on layout and hardware.

GPU work usually moves down this ladder only when necessary:

• use framework operations first;
• use vendor libraries for standard dense, sparse, convolution, communication, and transform primitives;
• use compiler-backed kernel DSLs when an operation is custom but regular;
• write C++/CUDA, HIP, OpenCL, Metal, Vulkan compute, or WebGPU when control matters;
• inspect PTX, LLVM IR, SPIR-V, ISA, or generated assembly when the compiler's choices matter;
• write assembly-like code only for rare, high-value kernels.

The lower layers are powerful, but they are expensive to maintain. A compendium record should preserve why the lower layer was needed, what measurement justified it, and what higher-level fallback exists.

CUDA is NVIDIA's parallel programming platform for NVIDIA GPUs. It supplies the programming model, compiler path, runtime APIs, libraries, and tools used by much of the AI and high-performance-computing ecosystem.

CUDA code typically launches kernels from host code. The hardware schedules blocks onto streaming multiprocessors, and the program's performance depends on launch shape, occupancy, memory access, synchronization, and library use.

CUDA is important in this compendium because it is the practical implementation layer beneath many transformers, training neural networks, and Python workflows.

Domain libraries are usually the first serious optimization:

• cuBLAS for dense linear algebra;
• cuSPARSE for sparse matrix work;
• cuDNN for neural-network primitives;
• NCCL for collective communication across GPUs;
• vendor FFT, random-number, graph, and solver libraries where appropriate.

The rule is simple: use the mature library when the operation is standard and well-supported. Write a kernel when the operation is custom, fusion matters, data movement dominates, or library boundaries force avoidable materialization.

ROCm is AMD's open software stack for GPU computing. HIP provides a C++ runtime and portability layer that can target AMD GPUs and can help port CUDA-style code.

ROCm belongs in the graph because it changes the portability question. A GPU page should distinguish CUDA-specific source, HIP-portable source, OpenCL source, WebGPU source, and framework-level code. Those are not interchangeable implementation records.

OpenCL is a cross-platform framework for heterogeneous computing. Vulkan includes compute capabilities used in graphics-adjacent and cross-vendor systems. Metal is Apple's GPU API. WebGPU and WGSL define the emerging web-facing GPU programming surface.

These APIs matter when deployment target, portability, browser execution, graphics integration, or vendor neutrality matters more than maximum CUDA ecosystem depth. For WASM, WebGPU is especially relevant because it lets browser and web-app code access GPU compute through a standardized web surface.

Many GPU workflows live in Python because the surrounding model, experiment, data loading, plotting, and training loop already live there. Useful paths include:

• framework operations in PyTorch, JAX, TensorFlow, or similar libraries;
• PyTorch C++/CUDA extensions when a framework needs a custom operator;
• Triton for writing fused GPU kernels in a Python-facing DSL;
• Numba CUDA for Python-authored kernels;
• CuPy custom kernels for NumPy-like GPU workflows.

The graph should record which layer is being used. "Python GPU code" can mean a framework call, a generated kernel, a compiled extension, a DSL kernel, or a host orchestration script.

Rust is useful around GPU systems when host-side correctness, memory ownership, async orchestration, FFI boundaries, or service integration matter. It does not remove the need to understand the GPU execution model, but it can make the surrounding system more explicit.

Rust GPU work should record the backend, unsafe boundaries, memory ownership rules, device-buffer lifetimes, and build toolchain. A graph edge from a Rust crate to a GPU backend is more useful than a vague "accelerated" label.

PTX is NVIDIA's virtual instruction set for GPU programs. SASS is NVIDIA machine code for a particular GPU architecture. Other ecosystems have their own intermediate and machine representations, such as LLVM IR or SPIR-V.

Most work does not require writing PTX. Reading generated lower-level code becomes useful when profiling shows a kernel is limited by instruction mix, memory operations, register pressure, instruction scheduling, or compiler-generated behavior.

Lower-level records should preserve architecture target, compiler version, flags, generated representation, and the benchmark that justified looking below source code.

GPU performance is often a memory problem wearing a compute costume. Global memory, caches, shared/local memory, registers, constant memory, unified memory, pinned host memory, peer-to-peer transfers, and interconnects all shape performance.

Good GPU code tries to:

• move less data;
• move data in larger, regular batches;
• reuse data close to the execution units;
• avoid unnecessary host-device round trips;
• fuse operations when intermediate materialization dominates;
• preserve layouts that libraries and kernels can consume efficiently.

This links GPU programming to data storage: tensors, checkpoints, datasets, memory-mapped files, precision formats, and cache layouts all decide how much work reaches the accelerator.

Common GPU patterns include maps, reductions, scans, stencils, matrix multiplication, convolution, sorting, histograms, graph traversal, sampling, attention, normalization, and fused elementwise operations.

Some patterns are naturally regular. Others are hard because they branch, scatter, gather, use irregular graph topology, or depend on dynamic shapes. That is where graphs, data visualization, and performance traces become useful: they reveal where the workload's shape fights the hardware.

GPU performance claims should come with evidence. Profilers such as NVIDIA Nsight Compute help connect runtime behavior to occupancy, memory throughput, instruction mix, and bottlenecks.

Useful measurements include wall time, kernel time, launch count, transfer time, memory bandwidth, achieved occupancy, arithmetic intensity, utilization, power, numerical error, and end-to-end throughput. A fast isolated kernel can still slow the application if it adds copies, synchronization, compilation overhead, or maintenance risk.

Benchmark records should preserve hardware, driver, toolkit, library versions, tensor shapes, batch sizes, precision, input distribution, warmup, variance, and the baseline being compared. AI inference optimization under real constraints extends that record through queue time, tail latency, task quality, retries, placement, and cost per successful task.

GPU programs are shape-sensitive. A kernel that is excellent for one tensor shape, batch size, sequence length, graph degree distribution, or image resolution can be mediocre for another. Benchmarks should therefore record the shape contract rather than report a single speedup. For transformers, this includes batch size, context length, hidden size, attention heads, key-value cache layout, precision, and whether prefill or decode is being measured.

The same rule applies outside AI. Matrix dimensions, stride, sparsity, histogram bucket count, graph topology, stencil radius, and input skew can all change the bottleneck. A useful benchmark compares against a strong baseline: vendor library, framework primitive, CPU implementation, previous kernel, or simpler algorithm. Otherwise a custom kernel may only prove that the first baseline was weak.

For graph utility, the benchmark should become its own evidence node connected to kernel source, hardware, dataset, command, profiler trace, and result. That lets a future reader ask whether a claim is still valid after a driver upgrade, library change, new model shape, or deployment target shift.

GPU debugging is difficult because failures can involve asynchronous launches, undefined behavior, data races, numerical instability, out-of-bounds memory, missing synchronization, driver/toolkit mismatch, or a hidden framework fallback.

Good practice is to isolate a minimal kernel, add correctness tests against a CPU or library baseline, make synchronization explicit during debugging, run sanitizers or vendor tools when available, and keep a small set of representative shapes. The debugging artifact belongs near data sources: it is evidence about a computation, not just a developer note.

GPU programs often trade precision, throughput, and memory. FP32, FP16, BF16, TF32, INT8, quantized formats, tensor cores, reduction order, accumulation precision, and deterministic settings can change results.

For AI workloads, numerical behavior should be recorded with the model and benchmark. A faster kernel is not equivalent if it changes loss, evaluation score, stability, or reproducibility in a way the project cannot explain.

Modern AI uses GPUs for matrix multiplication, convolutions, attention, normalization, sampling, embedding lookup, communication, and data preprocessing. The most valuable kernel work often reduces memory traffic or fuses operations around a known bottleneck rather than inventing a new primitive from scratch.

Transformers make this visible. Attention, MLP layers, normalization, KV cache layout, batch scheduling, quantization, and interconnect all interact. A model-speed record should therefore link architecture, tensor shapes, precision, hardware, library versions, runtime, and profiling evidence.

GPU code can run in notebooks, training clusters, inference servers, local workstations, browsers, native apps, and embedded systems. Each surface changes the constraints: driver availability, cold-start time, compilation cache, memory limits, sandboxing, observability, and portability.

For web surfaces, WASM and WebGPU matter. For production inference, scheduling and memory fragmentation may matter more than a single-kernel microbenchmark. For research, iteration speed and debuggability can dominate peak throughput.

A custom kernel is a maintenance object. Before writing one, record why existing libraries are insufficient: unsupported operation, bad shape coverage, unnecessary memory traffic, fusion opportunity, portability requirement, or deployment constraint. After writing one, preserve correctness tests, benchmark scripts, representative shapes, numerical tolerance, profiler output, and fallback path.

The lifecycle does not end when the kernel is fast once. Hardware generations change, compiler heuristics change, frameworks add new primitives, and model shapes drift. A good GPU record therefore includes a retirement rule: when the vendor library catches up, when portability matters more than speed, when the shape disappears, or when the maintenance cost exceeds the gain.

This connects GPU programming to software libraries and standards. The fastest path may be a custom CUDA kernel; the durable path may be a portable API, library call, generated kernel, or WebGPU implementation. The article should make that tradeoff visible.

GPU programming should not be an isolated vendor-tool page. It is a practical neighbor of transformers, training neural networks, linear algebra, Python, Rust, WASM, data storage, software libraries, and standards.

Useful graph nodes include kernel, library, runtime, compiler, IR, hardware architecture, memory space, tensor shape, benchmark, profiler trace, model layer, precision mode, dataset, and deployment target.

Useful edges include implements, accelerates, profiles, compiles_to, targets, transfers_to, reads_from, writes_to, fuses, benchmarks_against, depends_on, limited_by, and valid_for_shape. Those edges let a reader move from mathematical intent to implementation evidence.

Fields worth preserving include backend, vendor, device architecture, driver version, toolkit version, compiler flags, library versions, kernel source, generated IR, tensor shapes, precision, memory layout, benchmark command, profiler output, baseline, numerical tolerance, source dataset, and deployment target.

Those fields keep GPU claims honest. "Faster" is only useful when the graph also knows faster than what, on which hardware, for which shapes, with which precision, and at what maintenance cost.

GPU claims should carry evidence status labels. Useful states include hypothesis, microbenchmark, profiled bottleneck, numerically validated kernel, production measurement, portability claim, and retired optimization. A hypothesis says where speed might come from. A microbenchmark says a narrow shape improved. A profiled bottleneck ties the claim to a trace. A numerically validated kernel says the output stayed within an accepted tolerance. A production measurement says the whole workflow improved under real data, scheduling, and deployment constraints.

These labels help readers avoid a common trap: treating one fast kernel as proof that the application is faster, cheaper, or more reliable. They also help connect GPU pages to data visualization, training neural networks, and data storage, because performance evidence needs charts, artifacts, representative inputs, and preserved profiler records.

GPU work should prove correctness before celebrating speed. Parallel reductions can change floating-point order. Lower precision can alter model behavior. Fused kernels can hide intermediate values that were previously checked. Asynchronous launches can make timing misleading. A memory bug can look like random numerical noise until it corrupts a later stage.

A useful validation record names the baseline, input distribution, output tolerance, random seeds, precision, device, library versions, and the failure cases that were tested. For linear algebra, this often means comparing against a trusted CPU or vendor-library implementation across representative shapes. For training neural networks, it may mean checking loss curves, gradient behavior, evaluation metrics, and downstream task quality rather than only kernel output.

Correctness evidence should also distinguish bitwise equality, numerical tolerance, statistical equivalence, and task-level equivalence. These are not interchangeable. A renderer, simulation, ranking model, and language model may each tolerate different numerical drift. The compendium graph should preserve that tolerance as part of the claim, so a speedup remains attached to its validity boundary.

A practical GPU optimization pass usually starts at the application boundary, not the kernel source. First measure end-to-end time and cost. Then split host work, device kernels, data transfer, synchronization, compilation, and I/O. Only after that does it make sense to inspect memory coalescing, occupancy, fusion, tiling, precision, batching, streams, graph capture, or custom kernels.

The strongest optimizations are often boring: use a better library call, batch work, keep data on device, choose a layout the library expects, avoid unnecessary format conversion, or remove synchronization. Custom kernels are valuable when the workload shape is stable, the baseline is known, numerical tolerance is documented, and profiling shows a specific bottleneck. They are a liability when they merely encode undocumented assumptions.

This playbook gives the knowledge graph a useful sequence: workload -> baseline -> profile -> bottleneck -> change -> validation -> production measurement. It links GPU programming to rules of thumb, software libraries, data sources, and data visualization, because performance claims become durable only when measurements are preserved and readable.

GPU work fails in recurring ways:

• copying data to the GPU costs more than the kernel saves;
• a custom kernel beats a weak baseline but loses to a vendor library;
• benchmarks use tiny shapes that hide launch overhead or huge shapes that hide deployment reality;
• memory layout fights coalescing or library expectations;
• numerical shortcuts change model behavior;
• a framework silently falls back to CPU;
• portability claims ignore vendor-specific extensions;
• profiler evidence is not preserved with the claim.

The remedy is measurement, source-backed records, and clear boundaries between mathematical operation, kernel implementation, library call, benchmark, and deployment context.

• NVIDIA CUDA C++ Programming Guide
• NVIDIA CUDA C++ Best Practices Guide
• NVIDIA PTX ISA documentation
• NVIDIA cuBLAS documentation
• NVIDIA cuDNN documentation
• NVIDIA Nsight Compute documentation
• AMD ROCm documentation
• Khronos OpenCL
• Khronos Vulkan
• W3C WebGPU specification
• W3C WGSL specification
• Triton documentation
• Numba CUDA documentation
• CuPy custom kernels
• PyTorch C++ and CUDA extensions
• Apple Metal

• Linear Algebra for matrix operations, tensors, and numerical structure.
• Training Neural Networks and Transformers for the AI workloads that make GPU programming visible.
• Python, Rust, and WASM for host-language and deployment surfaces.
• Data Storage for tensors, checkpoints, datasets, caches, and profiler artifacts.
• Data Sources for benchmark inputs and reproducible experiment records.
• Software Libraries for wrappers, bindings, kernels, and vendor libraries.
• Standards for OpenCL, Vulkan, WebGPU, WGSL, and portable interface contracts.
• Rules of Thumb for deciding when to measure, when to use a library, and when to write a custom kernel.