Gaze-assisted automatic captioning of fetal ultrasound videos using three-way multi-modal deep neural networks

Mohammad Alsharid*, Yifan Cai, Harshita Sharma, Lior Drukker, Aris T. Papageorghiou, J. Alison Noble

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

8 Scopus citations


In this work, we present a novel gaze-assisted natural language processing (NLP)-based video captioning model to describe routine second-trimester fetal ultrasound scan videos in a vocabulary of spoken sonography. The primary novelty of our multi-modal approach is that the learned video captioning model is built using a combination of ultrasound video, tracked gaze and textual transcriptions from speech recordings. The textual captions that describe the spatio-temporal scan video content are learnt from sonographer speech recordings. The generation of captions is assisted by sonographer gaze-tracking information reflecting their visual attention while performing live-imaging and interpreting a frozen image. To evaluate the effect of adding, or withholding, different forms of gaze on the video model, we compare spatio-temporal deep networks trained using three multi-modal configurations, namely: (1) a gaze-less neural network with only text and video as input, (2) a neural network additionally using real sonographer gaze in the form of attention maps, and (3) a neural network using automatically-predicted gaze in the form of saliency maps instead. We assess algorithm performance through established general text-based metrics (BLEU, ROUGE-L, F1 score), a domain-specific metric (ARS), and metrics that consider the richness and efficiency of the generated captions with respect to the scan video. Results show that the proposed gaze-assisted models can generate richer and more diverse captions for clinical fetal ultrasound scan videos than those without gaze at the expense of the perceived sentence structure. The results also show that the generated captions are similar to sonographer speech in terms of discussing the visual content and the scanning actions performed.

Original languageEnglish
Article number102630
JournalMedical Image Analysis
StatePublished - Nov 2022


FundersFunder number
Clare Teng and Mourad Gridach
NIHR Maudsley Biomedical Research Centre
Engineering and Physical Sciences Research CouncilEP/MO13774/1
Rhodes Scholarships


    • Audio–visual
    • Fetal ultrasound
    • Gaze tracking
    • Multi-modal
    • Video captioning


    Dive into the research topics of 'Gaze-assisted automatic captioning of fetal ultrasound videos using three-way multi-modal deep neural networks'. Together they form a unique fingerprint.

    Cite this