GrabAR: Occlusion-aware grabbing virtual objects in AR

Xiao Tang, Xiaowei Hu, Chi Wing Fu, Daniel Cohen-Or

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Existing augmented reality (AR) applications often ignore the occlusion between real hands and virtual objects when incorporating virtual objects in user's views. The challenges come from the lack of accurate depth and mismatch between real and virtual depth. This paper presents GrabAR1, a new approach that directly predicts the real-and-virtual occlusion and bypasses the depth acquisition and inference. Our goal is to enhance AR applications with interactions between hand (real) and grabbable objects (virtual). With paired images of hand and object as inputs, we formulate a compact deep neural network that learns to generate the occlusion mask. To train the network, we compile a large dataset, including synthetic data and real data. We then embed the trained network in a prototyping AR system to support real-time grabbing of virtual objects. Further, we demonstrate the performance of our method on various virtual objects, compare our method with others through two user studies, and showcase a rich variety of interaction scenarios, in which we can use bare hand to grab virtual objects and directly manipulate them.

Original languageEnglish
Title of host publicationUIST 2020 - Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology
PublisherAssociation for Computing Machinery, Inc
Pages697-708
Number of pages12
ISBN (Electronic)9781450375146
DOIs
StatePublished - 20 Oct 2020
Event33rd Annual ACM Symposium on User Interface Software and Technology, UIST 2020 - Virtual, Online, United States
Duration: 20 Oct 202023 Oct 2020

Publication series

NameUIST 2020 - Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology

Conference

Conference33rd Annual ACM Symposium on User Interface Software and Technology, UIST 2020
Country/TerritoryUnited States
CityVirtual, Online
Period20/10/2023/10/20

Keywords

  • Augmented reality
  • Interaction
  • Neural network
  • Occlusion

Fingerprint

Dive into the research topics of 'GrabAR: Occlusion-aware grabbing virtual objects in AR'. Together they form a unique fingerprint.

Cite this