Medical images like radiographs (“x-rays”), CT and MRI scans, and retinal photographs are routinely used to evaluate disease severity and change over time. These assessments help to inform clinical decision making. For many diseases, the evaluating physician classifies disease severity into categories (such as mild, moderate, and severe) and whether this category has changed on follow-up imaging. However, experts are not always consistent and do not agree with each other, resulting in variable interpretations. This variability is not surprising, as diseases do not always fall cleanly into these human-constructed categories.

To address these issues, we developed an automatic algorithm for continuous disease severity evaluation and change detection in medical imaging. The basis of our algorithm is the Siamese neural network, a type of deep learning architecture that takes two images as inputs. This network architecture was originally used for verification of credit card signatures in the 1990s. In our study, we modified and trained Siamese neural networks to output a quantitative measure of difference in disease severity between input images. We tested this approach for two medical imaging tasks: 1) evaluating the severity of retinopathy of prematurity from retinal photographs and 2) evaluating the severity of knee osteoarthritis from radiographs.

We show that the output of the Siamese neural network can be used to provide a continuous measure of disease severity and change over time. For example, knee osteoarthritis can be classified into ordinal disease severity categories, like normal, minimal, mild, moderate, and severe disease. A standardized grading system for osteoarthritis like the Kellgren-Lawrence grading system can be used, with ordinal categories that loosely correspond to such descriptors of severity. Instead of reporting that the knee osteoarthritis increased from minimal to moderate for example (which struggles with inter-rater variability issues), our algorithm can report a quantitative measure of severity relative to normal and how that changes over time. Furthermore, in cases where there is a slight change in disease severity that is still within the same ordinal disease category (e.g. mild to mild change), our approach allows for that difference to be quantified. Thus, this Siamese neural network approach can allow for a more reproducible and granular evaluation of disease severity and its change over time.

Siamese neural network outputs can be used to represent a continuous spectrum of longitudinal change in disease severity, shown here for retinopathy of prematurity and knee osteoarthritis. See manuscript Figures 3 and 4 for details. 

We also show that our Siamese neural network can be trained with relatively weak labels; image pairs fed into the algorithm for training are labeled only for change versus no change in the ordinal disease severity category, instead of a continuous score. Additionally, we can localize longitudinal changes in the images with our algorithm, without training using localization data.

Taken together, our results demonstrate that Siamese neural networks are potentially useful for evaluating the continuous spectrum of disease severity and change in medical imaging. This architecture could be incorporated in any algorithm developed for a workflow where the clinician may ask: "How bad is the disease in this image, and has it changed?"

Please see the full manuscript entitled "Siamese neural networks for continuous disease severity evaluation and change detection in medical imaging" at the link.