DTensor 101: Mesh, Layout, and SPMD in TensorFlow

DTensor adds a global, sharded-tensor model to TensorFlow. You define a device Mesh and a per-axis Layout; DTensor expands your code via SPMD, inserting collectives so ops like tf.matmul run across CPUs/GPUs/TPUs—or even multi-client setups—with the same Python. You can replicate or shard tensors, pack/unpack components, request sharded outputs with dtensor.call_with_layout, and use dtensor.DVariable (fixed layout) instead of tf.Variable. The notebook walks through creating meshes, applying layouts, matmul sharding strategies, generating DTensor outputs, and variable semantics—setting you up for DTensor-based distributed training.