How Pre-Trained Vision Models Are Revolutionizing Anatomical Structure Retrieval

This study introduces a new benchmark for 3D medical image retrieval using the TotalSegmentator dataset, showcasing how pre-trained vision embeddings—originally trained on natural images—can be repurposed for anatomical structure localization. By integrating a ColBERT-inspired re-ranking approach, the method boosts recall across diverse anatomical regions, though challenges remain in retrieving certain structures like the brain and face. The findings suggest that general image pre-training (e.g., ImageNet) can be as effective, if not slightly better, than domain-specific medical datasets. This benchmark lays the groundwork for future innovations in content-based medical image search and targeted organ retrieval.