TextDeformer: Geometry Manipulation using Text Guidance

William Gao, Noam Aigerman, Thibault Groueix, Vova Kim, Rana Hanocka

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review


We present a technique for automatically producing a deformation of an input triangle mesh, guided solely by a text prompt. Our framework is capable of deformations that produce both large, low-frequency shape changes, and small high-frequency details. Our framework relies on differentiable rendering to connect geometry to powerful pre-trained image encoders, such as CLIP and DINO. Notably, updating mesh geometry by taking gradient steps through differentiable rendering is notoriously challenging, commonly resulting in deformed meshes with significant artifacts. These difficulties are amplified by noisy and inconsistent gradients from CLIP. To overcome this limitation, we opt to represent our mesh deformation through Jacobians, which updates deformations in a global, smooth manner (rather than locally-sub-optimal steps). Our key observation is that Jacobians are a representation that favors smoother, large deformations, leading to a global relation between vertices and pixels, and avoiding localized noisy gradients. Additionally, to ensure the resulting shape is coherent from all 3D viewpoints, we encourage the deep features computed on the 2D encoding of the rendering to be consistent for a given vertex from all viewpoints. We demonstrate that our method is capable of smoothly-deforming a wide variety of source mesh and target text prompts, achieving both large modifications to, e.g., body proportions of animals, as well as adding fine semantic details, such as shoe laces on an army boot and fine details of a face.

Original languageEnglish
Title of host publicationProceedings - SIGGRAPH 2023 Conference Papers
EditorsStephen N. Spencer
PublisherAssociation for Computing Machinery, Inc
ISBN (Electronic)9798400701597
StatePublished - 23 Jul 2023
Externally publishedYes
Event2023 Special Interest Group on Computer Graphics and Interactive Techniques Conference, SIGGRAPH 2023 - Los Angeles, United States
Duration: 6 Aug 202310 Aug 2023

Publication series

NameProceedings - SIGGRAPH 2023 Conference Papers


Conference2023 Special Interest Group on Computer Graphics and Interactive Techniques Conference, SIGGRAPH 2023
Country/TerritoryUnited States
CityLos Angeles


  • CLIP
  • deformation
  • mesh
  • text guidance


Dive into the research topics of 'TextDeformer: Geometry Manipulation using Text Guidance'. Together they form a unique fingerprint.

Cite this