PyTorch Development Patterns

Problem

PyTorch projects often fail in training details rather than model architecture: hard-coded devices, missing random seeds, unchecked tensor shapes, forgotten train()/eval() transitions, growing GPU memory, and slow data loading. This skill turns those recurring issues into reviewable patterns for new models, training scripts, code review, training-loop debugging, and data pipelines.

How It Works

It organizes guidance around device independence, reproducibility, and explicit shape management. Common practices include using .to(device) instead of hard-coding GPUs, fixing random sources with torch.manual_seed, documenting output shapes in nn.Module, and using torch.no_grad(), optimizer.zero_grad(set_to_none=True), and model.train()/model.eval() for standard training and validation loops. Data coverage includes Dataset, DataLoader, pin_memory=True, and variable-length collate_fn logic. Experiment handling covers checkpoint save/load, with a reminder to prefer weights_only=True for safer loading. Performance guidance covers torch.amp.autocast mixed precision, gradient_checkpointing, and torch.compile, with validation through torch.profiler and torch.cuda.memory_summary().

Boundaries

It fits engineering checks for PyTorch training code and inference services, not full distributed training, data-science experimentation, or model compression. Behavior of torch.compile, mixed precision, and memory strategies depends on version, operators, and input shapes, so results should be profiled on real data.