Towards a Neural Graphics Pipeline for Controllable Image Generation

Xuelin Chen, Daniel Cohen-Or, Baoquan Chen, Niloy J. Mitra

Research output: Contribution to journalArticlepeer-review

Abstract

In this paper, we leverage advances in neural networks towards forming a neural rendering for controllable image generation, and thereby bypassing the need for detailed modeling in conventional graphics pipeline. To this end, we present Neural Graphics Pipeline (NGP), a hybrid generative model that brings together neural and traditional image formation models. NGP decomposes the image into a set of interpretable appearance feature maps, uncovering direct control handles for controllable image generation. To form an image, NGP generates coarse 3D models that are fed into neural rendering modules to produce view-specific interpretable 2D maps, which are then composited into the final output image using a traditional image formation model. Our approach offers control over image generation by providing direct handles controlling illumination and camera parameters, in addition to control over shape and appearance variations. The key challenge is to learn these controls through unsupervised training that links generated coarse 3D models with unpaired real images via neural and traditional (e.g., Blinn-Phong) rendering functions, without establishing an explicit correspondence between them. We demonstrate the effectiveness of our approach on controllable image generation of single-object scenes. We evaluate our hybrid modeling framework, compare with neural-only generation methods (namely, DCGAN, LSGAN, WGAN-GP, VON, and SRNs), report improvement in FID scores against real images, and demonstrate that NGP supports direct controls common in traditional forward rendering. Code is available at http://geometry.cs.ucl.ac.uk/projects/2021/ngp.

Original languageEnglish
Pages (from-to)127-140
Number of pages14
JournalComputer Graphics Forum
Volume40
Issue number2
DOIs
StatePublished - May 2021

Keywords

  • CCS Concepts
  • Computing methodologies → Rendering
  • Shape modeling

Fingerprint

Dive into the research topics of 'Towards a Neural Graphics Pipeline for Controllable Image Generation'. Together they form a unique fingerprint.

Cite this