Semantic object reconstruction via casual handheld scanning

Ruizhen Hu, Cheng Wen, Oliver Van Kaick, Luanmin Chen, Di Lin, Daniel Cohen-Or, Hui Huang*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review


We introduce a learning-based method to reconstruct objects acquired in a casual handheld scanning setting with a depth camera. Our method is based on two core components. First, a deep network that provides a semantic segmentation and labeling of the frames of an input RGBD sequence. Second, an alignment and reconstruction method that employs the semantic labeling to reconstruct the acquired object from the frames. We demonstrate that the use of a semantic labeling improves the reconstructions of the objects, when compared to methods that use only the depth information of the frames. Moreover, since training a deep network requires a large amount of labeled data, a key contribution of our work is an active self-learning framework to simplify the creation of the training data. Speciically, we iteratively predict the labeling of frames with the neural network, reconstruct the object from the labeled frames, and evaluate the conidence of the labeling, to incrementally train the neural network while requiring only a small amount of user-provided annotations. We show that this method enables the creation of data for training a neural network with high accuracy, while requiring only little manual efort.

Original languageEnglish
Article number219
JournalACM Transactions on Graphics
Issue number6
StatePublished - Nov 2018
Externally publishedYes


  • 3D scanning
  • Active learning
  • Semantic reconstruction
  • object registration


Dive into the research topics of 'Semantic object reconstruction via casual handheld scanning'. Together they form a unique fingerprint.

Cite this