Learning from Longitudinal Mammography Studies

When reading imaging studies, radiologists often compare the acquired images to one or more prior studies of the patient. Machine learning algorithms that assist in identifying abnormalities in medical images usually do not analyze prior images. This work describes a deep-learning classification framework for mammography studies, which incorporates prior image information using four approaches: (1) late fusion of prediction scores; (2) early fusion of input layers; (3) feature fusion combining a convolutional neural network (CNN) and gradient boosting trees; and (4) feature fusion using CNN and long-short term memory (LSTM) architecture. We demonstrate the advantages and limitations of each approach and compare their performance in identifying biopsy-proven malignancies in mammography screening studies. On an evaluation cohort of 439 patients, adding prior studies to the analysis improved the diagnostic performance of the classification framework. The CNN-LSTM architecture achieved the highest area under the ROC curve of 0.88, with sensitivity and specificity of 0.87 and 0.78, respectively. The methods that were trained using information from prior studies achieved better results than the baseline classifier, with up to 45% reduction in false-positive rate at the same sensitivity. The major advantage of the CNN-LSTM approach is in its flexibility and scalability; it allows to use the same network to classify sequences of multiple priors with variable length. The study demonstrates that longitudinal analysis of images can potentially improve the ability of machine learning algorithms to accurately and reliably interpret imaging studies, thus providing value to the radiology community.