Our work on Segment Any Tumour 3D, or SAT3D, grew from a limitation we repeatedly encountered in medical image segmentation. A model trained to identify one tumour type in one imaging modality could perform remarkably well, yet often could not be readily applied to another cancer, anatomical region, or scan type. Developing a separate model for every tumour, organ, and modality was not scalable.
This led us to ask: could we build a single three-dimensional model that learns from tumours across the body and multiple medical imaging modalities?
Foundation models had begun transforming computer vision by learning general representations from large and diverse datasets. The Segment Anything Model showed that users could guide a model with simple prompts, such as points or boxes, to identify objects in natural images. Medical scans, however, present distinct challenges. They are commonly three-dimensional, tumours may extend across many slices, and their boundaries can be irregular, poorly defined, or difficult to distinguish from surrounding tissue. Their appearance also varies across cancer types, anatomical regions, scanners, protocols, and modalities.
We therefore developed SAT3D as a prompt-driven foundation model for volumetric tumour segmentation. One of our first challenges was assembling a diverse training collection. We brought together 17,075 three-dimensional image and tumour-mask pairs from 11 public datasets, covering computed tomography, computed tomography angiography, magnetic resonance imaging, positron emission tomography, and ultrasound. Combining these datasets was far more complicated than simply collecting files. They differed in image resolution, anatomical coverage, annotation conventions, tumour definitions, and physical dimensions. Some scans contained a single tumour, while others included multiple lesions. The data therefore had to be carefully harmonised while preserving clinically meaningful information. This reinforced an important lesson: the generality of a foundation model depends not only on its architecture, but also on the diversity and quality of its training data.
From the beginning, we did not want SAT3D to operate as a fully automatic system without user involvement. Tumour boundaries can be inherently ambiguous, and even experts may interpret difficult regions differently. Instead, we designed SAT3D as an interactive model. A user can provide a positive point to indicate tumour tissue or a negative point to identify an area that should be excluded. The model then updates the three-dimensional segmentation using this guidance. This allows users to refine a result rather than simply accept or reject an automatic prediction. It also reflects how we believe artificial intelligence should be used in clinical environments: as a tool that supports medical expertise rather than replaces it.
As the project progressed, we encountered another problem. During interactive refinement, the model could use its previous segmentation when generating the next prediction, but it did not necessarily know which parts were reliable. This created a risk of reinforcing its own errors. To address this, we introduced a discriminator network, which we call a critic network. The critic learns to estimate where the segmentation is likely to be incorrect and produces a confidence map highlighting less reliable regions. Rather than presenting every part of the segmentation with equal confidence, SAT3D can indicate areas requiring closer inspection or further interaction. Uncertainty often appears around tumour boundaries, where distinguishing tumour from surrounding tissue is particularly difficult. Training the critic was challenging because it needed to learn a different task from the segmentation network. The segmentation model aimed to outline the tumour, while the critic aimed to recognise where that outline might be wrong. We trained both components together so the critic could learn from the segmentation model’s evolving errors.
Developing a foundation model for volumetric imaging also presented computational challenges. Three-dimensional scans require substantially more memory and processing than two-dimensional images, particularly when training across thousands of cases from different modalities and anatomical regions.
Rather than assuming that increasing model size would automatically improve performance, we focused on a compact architecture. Computational limitations encouraged us to reconsider components, simplify the design, and determine where additional complexity was genuinely beneficial. This helped balance general tumour-segmentation capability, interactive refinement, uncertainty estimation, and computational efficiency. Compactness was not SAT3D’s central purpose, but it became an important design principle because research laboratories and hospitals have varying access to computing infrastructure.
We evaluated SAT3D across diverse tumour-segmentation tasks and compared it with medical foundation models, interactive approaches, lesion-focused methods, and task-specific models. We also tested it on datasets not included during training to examine whether it had learned general tumour characteristics rather than memorised familiar examples. The results showed that SAT3D could support segmentation across different tumour types, anatomical locations, and imaging modalities while retaining a relatively compact design. However, no foundation model performs equally well in every situation. Small lesions, poorly defined boundaries, unusual disease presentations, and previously unseen imaging characteristics remain challenging. Differences in scanners, image quality, clinical populations, and annotation practices can also affect performance.
SAT3D should therefore be viewed as a general foundation that must be carefully evaluated and adapted for specific clinical applications, rather than as a system ready for deployment without validation. We also wanted to move beyond numerical metrics. We developed an interactive interface that allows users to load scans, provide prompts, inspect tumour segmentations, and review confidence information. Building this interface shifted our focus towards practical questions: how quickly can users correct errors, is uncertainty information understandable, and how might the system fit into existing medical-imaging workflows?
Our next step is to explore these questions through clinical collaboration. Through the Smart Annotation and Interpretation Interface for Multi-Modal Cancer Imaging project, or SCAN-AI, we are investigating whether interactive tumour segmentation can reduce annotation time while preserving expert oversight. Looking back, SAT3D emerged from several connected ideas: moving beyond task-specific models, keeping clinicians involved, enabling AI systems to recognise uncertainty, and designing a computationally practical system.
We hope SAT3D contributes to a broader shift in medical artificial intelligence from systems that provide a single prediction towards systems that communicate uncertainty, accept expert guidance, and work collaboratively with the people responsible for patient care.