DreamSim and the Future of Embedding Models in Radiology AI

This article explores how different embedding approaches perform in medical image retrieval tasks. Self-supervised models slightly edge out supervised ones, though the performance gap across architectures is narrow. Surprisingly, pretraining on natural images (ImageNet) outperforms domain-specific sets (RadImageNet), while fractal-based embeddings achieve unexpectedly strong results given their synthetic origins. DreamSim, an ensemble of ViT embeddings fine-tuned with synthetic data, delivers the best recall overall, making it the current leader in embedding generation. Isolated anomalies—like poor recall for certain anatomies—remain unexplained, pointing to fertile ground for future research.