Evaluating AI in scientific imaging: start with the measurement
Published in Chemistry, Protocols & Methods, and Cell & Molecular Biology
Suppose a three-dimensional microscopy study asks whether a treatment changes the volume of an organelle. A machine-learning pipeline reconstructs the images and identifies the structures to be measured. Its outputs look clearer, and its average image-quality score improves.
Would that establish that the estimated change in organelle volume is reliable?
This hypothetical example separates two questions that deserve their own evidence: how well a method produces an image, and how well that image supports a measurement.
In my first-author work on electron tomography, my colleagues and I investigated adaptively learned dictionaries for reconstructing three-dimensional morphology from incomplete projection data [1]. The underlying challenge remains relevant: an algorithm helps determine which structures become visible and measurable.
My starting point for evaluating imaging AI is therefore the quantity the experiment needs to measure.
Define the scientific claim
For our organelle example, the intended result is an estimate of volume, and possibly a difference between experimental conditions. I would specify the structures, specimens and acquisition conditions covered by the claim, alongside the measurement errors that could change its interpretation.
An average image-similarity score can remain useful. However, a pipeline intended for quantitative morphology also needs evaluation of volume errors, missed structures and differences between conditions. A constant additive bias could cancel when comparing mean volumes; a bias that varies with structure size or treatment could distort the estimated effect. Appropriate controls and biological replication remain essential to interpreting the treatment comparison.
This distinction also helps set reasonable expectations. A method tested on one defined experimental system can make a valuable, carefully bounded contribution.
Test both stability and sensitivity
I find it useful to organise evaluation around two complementary tests.
Stability: when a change should leave the target quantity unchanged, does the measurement remain acceptably consistent? In our example, we might vary realistic acquisition noise while keeping the specimen structure fixed.
Sensitivity: when the target quantity really changes, does the pipeline recover the change? Here, we might assess whether known differences in organelle volume remain detectable after reconstruction and segmentation.
These tests address different failures. A method that suppresses small structures could appear stable because it removes variation, including variation the experiment needs to detect.
Antun and colleagues demonstrated relevant failure modes in the learned reconstruction systems they tested: small input perturbations could produce artefacts, and small structural changes could disappear from reconstructions [2]. Their findings motivate targeted testing; they do not establish that every learned reconstruction method behaves this way.
The perturbations must also make physical sense. Altering dose, focus or sampling can remove information. I would define the acquisition range within which reliable measurement is claimed, then examine how errors grow near its limits.
Make independence match the intended use
For a claim about new specimens, randomly separating image patches may leave training and test sets sharing information from the same specimen. For a claim about new experiments or instruments, the relevant separation may need to occur at those levels.
Yagis and colleagues provide a striking illustration from brain MRI classification. When slices from the same subjects could appear in both training and test data, slice-level test accuracy exceeded 90% even after the subject labels had been randomised [3]. The evaluation allowed subject-related information to inflate an apparent disease-classification result.
In microscopy, I would document how patches, images, specimens and experimental batches are related, and choose the split accordingly. Model selection and any preprocessing learned across the dataset should respect that separation.
The same reasoning applies to uncertainty: many patches from one specimen do not provide the same evidence as many independent specimens.
Use references and comparators that answer the question
Known structures in simulations or physical reference samples can help test measurement errors. Their value depends on how well they represent the acquisition process and specimens of interest. I would combine them with independent experimental data where the claim requires it.
Culley and colleagues’ NanoJ-SQUIRREL offers a useful methodological example: comparing super-resolution images with corresponding diffraction-limited images can reveal local inconsistencies [4]. That comparison does not uniquely validate every finer detail. A reference is informative within its own resolution and uncertainty.
For the organelle example, the practical question is whether an appropriate reference can reveal volume bias or missed structures that an aggregate image score obscures.
Comparisons should also include well-configured, task-appropriate alternatives. The nnU-Net study demonstrated the strength of a systematically configured segmentation pipeline across diverse datasets [5]. It is a reminder to examine the whole pipeline when interpreting gains attributed to a new model.
Follow uncertainty through to the measurement
If the conclusion concerns organelle volume, an uncertainty map over voxels does not automatically provide a calibrated uncertainty interval for that volume: errors can be spatially correlated. The evaluation should examine how reconstruction and segmentation uncertainty affect volume estimates and differences between conditions, while accounting for variation between independent specimens.
Liu and colleagues’ 2025 study of Bayesian structured illumination microscopy illustrates a constructive direction. Their method estimated reconstruction uncertainty and flagged unreliable generalisation in tested cases where training and imaging conditions did not match [6]. The authors also recognised an important limit: identifying such cases does not itself make the model generalise successfully to them.
For quantitative use, uncertainty estimates should be checked against observed errors in the conditions being claimed. A warning should also lead to an action, such as reviewing the raw data, acquiring a better measurement or restricting the conclusion.
Make the evidence easy to inspect
A useful report would connect the target measurement to its validation data, relevant error estimates, consequential failure cases and tested acquisition range. It should also describe the split definitions, preprocessing, model selection and model versions clearly enough for others to reproduce the analysis, with access to code and data where sharing is possible.
Together, these established concerns suggest a proportionate approach: identify the central claim, ask which error could overturn it, and prioritise the test that resolves that uncertainty.
For our organelle example, that decisive test might be whether the estimated treatment difference survives a credible assessment of measurement bias. Another benchmark score might add less information.
Which measurement error would most change the scientific conclusion in your imaging workflow? I would welcome published examples, discussion and potential collaborations with researchers evaluating reconstruction, microscopy and quantitative image-analysis methods.
References
-
Al-Afeef, A. et al. (2016). Electron tomography image reconstruction using data-driven adaptive compressed sensing. Scanning.
-
Antun, V. et al. (2020). On instabilities of deep learning in image reconstruction and the potential costs of AI. PNAS.
-
Yagis, E. et al. (2021). Effect of data leakage in brain MRI classification using 2D convolutional neural networks. Scientific Reports.
-
Culley, S. et al. (2018). Quantitative mapping and minimization of super-resolution optical imaging artifacts. Nature Methods.
-
Isensee, F. et al. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods.
-
Liu, T. et al. (2025). Bayesian deep-learning structured illumination microscopy enables reliable super-resolution imaging with uncertainty quantification. Nature Communications.
About the author: Dr Ala Al-Afeef is a computing scientist whose research includes machine learning, image reconstruction and scientific imaging. He holds a PhD in Computing Science from the University of Glasgow. ORCID.