Visualizing and enhancing a deep learning framework using patients age and gender for chest x-ray image retrieval

Yaron Anavi, Ilya Kogan, Elad Gelbart, Ofer Geva, Hayit Greenspan

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review


We explore the combination of text metadata, such as patients' age and gender, with image-based features, for X-ray chest pathology image retrieval. We focus on a feature set extracted from a pre-trained deep convolutional network shown in earlier work to achieve state-of-the-art results. Two distance measures are explored: a descriptor-based measure, which computes the distance between image descriptors, and a classification-based measure, which performed by a comparison of the corresponding SVM classification probabilities. We show that retrieval results increase once the age and gender information combined with the features extracted from the last layers of the network, with best results using the classification-based scheme. Visualization of the X-ray data is presented by embedding the high dimensional deep learning features in a 2-D dimensional space while preserving the pairwise distances using the t-SNE algorithm. The 2-D visualization gives the unique ability to find groups of X-ray images that are similar to the query image and among themselves, which is a characteristic we do not see in a 1-D traditional ranking.

Original languageEnglish
Title of host publicationMedical Imaging 2016
Subtitle of host publicationComputer-Aided Diagnosis
EditorsGeorgia D. Tourassi, Samuel G. Armato
ISBN (Electronic)9781510600201
StatePublished - 2016
EventMedical Imaging 2016: Computer-Aided Diagnosis - San Diego, United States
Duration: 28 Feb 20162 Mar 2016

Publication series

NameProgress in Biomedical Optics and Imaging - Proceedings of SPIE
ISSN (Print)1605-7422


ConferenceMedical Imaging 2016: Computer-Aided Diagnosis
Country/TerritoryUnited States
CitySan Diego


Dive into the research topics of 'Visualizing and enhancing a deep learning framework using patients age and gender for chest x-ray image retrieval'. Together they form a unique fingerprint.

Cite this