First Trimester Video Saliency Prediction Using Clstmu-Net with Stochastic Augmentation

Elizaveta Savochkina, Lok Hin Lee, He Zhao, Lior Drukker, Aris T. Papageorghiou, J. Alison Noble

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

1 Scopus citations

Abstract

In this paper we develop a multi-modal video analysis algorithm to predict where a sonographer should look next. Our approach uses video and expert knowledge, defined by gaze tracking data, which is acquired during routine first-trimester fetal ultrasound scanning. Specifically, we propose a spatio-temporal convolutional LSTMU-Net neural network (cLSTMU-Net) for video saliency prediction with stochastic augmentation. The architecture design consists of a U-Net based encoder-decoder network and a cLSTM to take into account temporal information. We compare the performance of the cLSTMU-Net alongside spatial-only architectures for the task of predicting gaze in first trimester ultrasound videos. Our study dataset consists of 115 clinically acquired first trimester US videos and a total of 45, 666 video frames. We adopt a Random Augmentation strategy (RA) from a stochastic augmentation policy search to improve model performance and reduce over-fitting. The proposed cLSTMU-Net using a video clip of 6 frames outperforms the baseline approach on all saliency metrics: KLD, SIM, NSS and CC (2.08, 0.28, 4.53 and 0.42 versus 2.16, 0.27, 4.34 and 0.39).

Original languageEnglish
Title of host publicationISBI 2022 - Proceedings
Subtitle of host publication2022 IEEE International Symposium on Biomedical Imaging
PublisherIEEE Computer Society
ISBN (Electronic)9781665429238
DOIs
StatePublished - 2022
Event19th IEEE International Symposium on Biomedical Imaging, ISBI 2022 - Kolkata, India
Duration: 28 Mar 202231 Mar 2022

Publication series

NameProceedings - International Symposium on Biomedical Imaging
Volume2022-March
ISSN (Print)1945-7928
ISSN (Electronic)1945-8452

Conference

Conference19th IEEE International Symposium on Biomedical Imaging, ISBI 2022
Country/TerritoryIndia
CityKolkata
Period28/03/2231/03/22

Funding

FundersFunder number
Engineering and Physical Sciences Research CouncilEP/T028572/1, EP/R013853/1
Engineering and Physical Sciences Research Council
NIHR Oxford Biomedical Research Centre

    Keywords

    • Fetal ultrasound
    • U-Net
    • convolutional LSTM
    • first trimester
    • gaze tracking
    • stochastic augmentation
    • video saliency prediction

    Fingerprint

    Dive into the research topics of 'First Trimester Video Saliency Prediction Using Clstmu-Net with Stochastic Augmentation'. Together they form a unique fingerprint.

    Cite this