Automatic Breast Lesion Classification by Joint Neural Analysis of Mammography and Ultrasound

Gavriel Habib, Nahum Kiryati, Miri Sklair-Levy, Anat Shalmon, Osnat Halshtok Neiman, Renata Faermann Weidenfeld, Yael Yagil, Eli Konen, Arnaldo Mayer

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Mammography and ultrasound are extensively used by radiologists as complementary modalities to achieve better performance in breast cancer diagnosis. However, existing computer-aided diagnosis (CAD) systems for the breast are generally based on a single modality. In this work, we propose a deep-learning based method for classifying breast cancer lesions from their respective mammography and ultrasound images. We present various approaches and show a consistent improvement in performance when utilizing both modalities. The proposed approach is based on a GoogleNet architecture, fine-tuned for our data in two training steps. First, a distinct neural network is trained separately for each modality, generating high-level features. Then, the aggregated features originating from each modality are used to train a multimodal network to provide the final classification. In quantitative experiments, the proposed approach achieves an AUC of 0.94, outperforming state-of-the-art models trained over a single modality. Moreover, it performs similarly to an average radiologist, surpassing two out of four radiologists participating in a reader study. The promising results suggest that the proposed method may become a valuable decision support tool for breast radiologists.

Original languageEnglish
Title of host publicationMultimodal Learning for Clinical Decision Support and Clinical Image-Based Procedures - 10th International Workshop, ML-CDS 2020, and 9th International Workshop, CLIP 2020, Held in Conjunction with MICCAI 2020, Proceedings
EditorsTanveer Syeda-Mahmood, Klaus Drechsler, Hayit Greenspan, Anant Madabhushi, Alexandros Karargyris, Cristina Oyarzun Laura, Stefan Wesarg, Marius George Linguraru, Raj Shekhar, Marius Erdt, Miguel Ángel González Ballester
PublisherSpringer Science and Business Media Deutschland GmbH
Pages125-135
Number of pages11
ISBN (Print)9783030609450
DOIs
StatePublished - 2020
Event10th International Workshop on Multimodal Learning for Clinical Decision Support, ML-CDS 2020, and the 9th International Workshop on Clinical Image-Based Procedures, CLIP 2020, held in conjunction with the 23rd International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2020 - Lima, Peru
Duration: 4 Oct 20208 Oct 2020

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume12445 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference10th International Workshop on Multimodal Learning for Clinical Decision Support, ML-CDS 2020, and the 9th International Workshop on Clinical Image-Based Procedures, CLIP 2020, held in conjunction with the 23rd International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2020
Country/TerritoryPeru
CityLima
Period4/10/208/10/20

Keywords

  • Deep learning
  • Mammography
  • Ultrasound

Fingerprint

Dive into the research topics of 'Automatic Breast Lesion Classification by Joint Neural Analysis of Mammography and Ultrasound'. Together they form a unique fingerprint.

Cite this