Specular-to-Diffuse Translation for Multi-view Reconstruction

Shihao Wu, Hui Huang*, Tiziano Portenier, Matan Sela, Daniel Cohen-Or, Ron Kimmel, Matthias Zwicker

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

12 Scopus citations

Abstract

Most multi-view 3D reconstruction algorithms, especially when shape-from-shading cues are used, assume that object appearance is predominantly diffuse. To alleviate this restriction, we introduce S2Dnet, a generative adversarial network for transferring multiple views of objects with specular reflection into diffuse ones, so that multi-view reconstruction methods can be applied more effectively. Our network extends unsupervised image-to-image translation to multi-view “specular to diffuse” translation. To preserve object appearance across multiple views, we introduce a Multi-View Coherence loss (MVC) that evaluates the similarity and faithfulness of local patches after the view-transformation. In addition, we carefully design and generate a large synthetic training data set using physically-based rendering. During testing, our network takes only the raw glossy images as input, without extra information such as segmentation masks or lighting estimation. Results demonstrate that multi-view reconstruction can be significantly improved using the images filtered by our network.

Original languageEnglish
Title of host publicationComputer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings
EditorsVittorio Ferrari, Cristian Sminchisescu, Yair Weiss, Martial Hebert
PublisherSpringer Verlag
Pages193-211
Number of pages19
ISBN (Print)9783030012243
DOIs
StatePublished - 2018
Event15th European Conference on Computer Vision, ECCV 2018 - Munich, Germany
Duration: 8 Sep 201814 Sep 2018

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11208 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference15th European Conference on Computer Vision, ECCV 2018
Country/TerritoryGermany
CityMunich
Period8/09/1814/09/18

Funding

FundersFunder number
ISF-NSFC2472/17, KQJSCX20170727101233642
Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung169151
National Natural Science Foundation of China61761146002, 61522213, 61861130365
National Key Research and Development Program of China2015CB352501
Guangdong Provincial Applied Science and Technology Research and Development Program2015A030312015

    Keywords

    • Generative adversarial network
    • Image translation
    • Multi-view coherence
    • Multi-view reconstruction
    • Specular-to-diffuse

    Fingerprint

    Dive into the research topics of 'Specular-to-Diffuse Translation for Multi-view Reconstruction'. Together they form a unique fingerprint.

    Cite this