How Datadog Turned Noisy Observability Metrics Into AI Gold

Datadog’s Toto model was trained on roughly one trillion time series data points—75% from curated observability metrics and 25% from the LOTSA dataset. Through padding, masking, and data augmentation (including random offsets and Gaussian sampling), Datadog ensured data diversity and quality. Synthetic data (about 5%) simulated additional real-world variability via ARMA processes, seasonal trends, and noise. Together, these methods improved Toto’s robustness and ability to generalize across domains.