# PyTorch/XLA

> Run PyTorch models on Cloud TPUs through the XLA compiler

- Category: Training & Fine-Tuning / Efficient Training
- Language: C++
- Licence: BSD-3-Clause
- Repository: https://github.com/pytorch/xla
- Website: https://pytorch.org/xla
- Canonical page: https://ai-tldr.dev/tools/pytorch-xla/
- Updated: 2026-04-07

## In short

PyTorch/XLA is the package that lets PyTorch code run on Google Cloud TPUs. It sits in the Efficient Training category of the Training & Fine-Tuning tools and is open-source, written in C++ and licensed BSD-3-Clause. To get started: install.

## Overview

PyTorch/XLA is the package that lets PyTorch code run on Google Cloud TPUs. It uses the XLA deep-learning compiler as the bridge: tensors are moved to an `xla` device, the training step is traced into an XLA graph, and the compiler turns that graph into TPU execution. In day-to-day use the change to an existing script is small — move the model and inputs to `'xla'`, and wrap the step in `torch_xla.step()`.

The distributed story is what most users come for. Single-process and multi-process training are both supported, along with FSDP for sharded parameters, `DistributedDataParallel`, and SPMD, where you describe how tensors are sharded and the compiler works out the collectives. Dynamic shapes and eager mode are available for the cases where full tracing is awkward, and the project ships profiling and performance-analysis tooling because compiled execution makes "why is this slow?" a different question than it is on GPU.

Google has since shipped [TorchTPU](/learn/landscape/torchtpu), a more native PyTorch-on-TPU path that registers a `tpu` device with eager execution rather than bridging through a traced compiler, and has said it will replace PyTorch/XLA. TorchTPU's public repository and its 0.2.0 release landed in October 2026, so new PyTorch-on-TPU work increasingly starts there, while PyTorch/XLA remains the codebase much of the existing TPU ecosystem is built on.

## What it does

- Runs PyTorch models on Cloud TPUs by moving tensors to an `xla` device
- Single-process and multi-process training modes
- SPMD sharding, FSDP and DistributedDataParallel for large-model training
- Dynamic shape and eager mode support alongside traced execution
- Profiling and performance-analysis tooling for compiled graphs
- Versioned in step with PyTorch releases (`torch` and `torch_xla` share a version)

## Getting started

Install `torch_xla` matched to your `torch` version, then adapt the training loop. The TPU extra pulls in the runtime.

1. Install

   The torch and torch_xla versions must match; the [tpu] extra installs the TPU runtime. This is the stable Python 3.11 line from the README.

```bash
pip install torch==2.8.0 'torch_xla[tpu]==2.8.0'
```

2. Move the model to the XLA device

   Same as `.to('cuda')` on GPU — the device string is `xla`.

```python
import torch_xla

model.to('xla')
```

3. Wrap the training step

   torch_xla.step() marks the graph boundary so the compiler knows what to compile and execute.

```python
with torch_xla.step():
    inputs, labels = inputs.to('xla'), labels.to('xla')
    # forward, loss, backward, optimizer step
```

4. Scale out

   From there, the docs at https://pytorch.org/xla cover multi-process launch, SPMD sharding annotations, FSDP and the profiler.

## When to use it

- Train an existing PyTorch model on Cloud TPU pods without rewriting it for JAX
- Shard a large model across TPU cores with SPMD or FSDP
- Fine-tune on TPU capacity when GPUs are unavailable or more expensive
- Profile and debug TPU execution of a PyTorch training loop

## Version history

- 2026-04-07: Google announced TorchTPU, which integrates PyTorch with TPUs natively through PrivateUse1 instead of a compiler bridge, and said it will replace PyTorch/XLA once its public GitHub repository ships.
