Mapping Images to Scene Graphs with Permutation-Invariant Structured Prediction

Roei Herzig, Moshiko Raboh, Gal Chechik, Jonathan Berant, Amir Globerson

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Machine understanding of complex images is a key goal of artificial intelligence. One challenge underlying this task is that visual scenes contain multiple interrelated objects, and that global context plays an important role in interpreting the scene. A natural modeling framework for capturing such effects is structured prediction, which optimizes over complex labels, while modeling within-label interactions. However, it is unclear what principles should guide the design of a structured prediction model that utilizes the power of deep learning components. Here we propose a design principle for such architectures that follows from a natural requirement of permutation invariance. We prove a necessary and sufficient characterization for architectures that follow this invariance, and discuss its implication on model design. Finally, we show that the resulting model achieves new state-of-the-art results on the Visual Genome scene-graph labeling benchmark, outperforming all recent approaches.
Original languageEnglish
Title of host publicationADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 31 (NIPS 2018)
EditorsS. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi , R. Garnett
PublisherCurran Associates, Inc.
Number of pages11
StatePublished - 2018
Event32nd Conference on Neural Information Processing Systems, NeurIPS 2018 - Montreal, Canada
Duration: 2 Dec 20188 Dec 2018

Publication series

NameAdvances in Neural Information Processing Systems
Volume31
ISSN (Print)1049-5258

Conference

Conference32nd Conference on Neural Information Processing Systems, NeurIPS 2018
Country/TerritoryCanada
CityMontreal
Period2/12/188/12/18

Fingerprint

Dive into the research topics of 'Mapping Images to Scene Graphs with Permutation-Invariant Structured Prediction'. Together they form a unique fingerprint.

Cite this