TY - JOUR
T1 - CLIPasso
T2 - Semantically-Aware Object Sketching
AU - Vinker, Yael
AU - Pajouheshgar, Ehsan
AU - Bo, Jessica Y.
AU - Bachmann, Roman Christian
AU - Bermano, Amit Haim
AU - Cohen-Or, Daniel
AU - Zamir, Amir
AU - Shamir, Ariel
N1 - Publisher Copyright:
© 2022 ACM.
PY - 2022/7/22
Y1 - 2022/7/22
N2 - Abstraction is at the heart of sketching due to the simple and minimal nature of line drawings. Abstraction entails identifying the essential visual properties of an object or scene, which requires semantic understanding and prior knowledge of high-level concepts. Abstract depictions are therefore challenging for artists, and even more so for machines. We present CLIPasso, an object sketching method that can achieve different levels of abstraction, guided by geometric and semantic simplifications. While sketch generation methods often rely on explicit sketch datasets for training, we utilize the remarkable ability of CLIP (Contrastive-Language-Image-Pretraining) to distill semantic concepts from sketches and images alike. We define a sketch as a set of Bézier curves and use a differentiable rasterizer to optimize the parameters of the curves directly with respect to a CLIP-based perceptual loss. The abstraction degree is controlled by varying the number of strokes. The generated sketches demonstrate multiple levels of abstraction while maintaining recognizability, underlying structure, and essential visual components of the subject drawn.
AB - Abstraction is at the heart of sketching due to the simple and minimal nature of line drawings. Abstraction entails identifying the essential visual properties of an object or scene, which requires semantic understanding and prior knowledge of high-level concepts. Abstract depictions are therefore challenging for artists, and even more so for machines. We present CLIPasso, an object sketching method that can achieve different levels of abstraction, guided by geometric and semantic simplifications. While sketch generation methods often rely on explicit sketch datasets for training, we utilize the remarkable ability of CLIP (Contrastive-Language-Image-Pretraining) to distill semantic concepts from sketches and images alike. We define a sketch as a set of Bézier curves and use a differentiable rasterizer to optimize the parameters of the curves directly with respect to a CLIP-based perceptual loss. The abstraction degree is controlled by varying the number of strokes. The generated sketches demonstrate multiple levels of abstraction while maintaining recognizability, underlying structure, and essential visual components of the subject drawn.
KW - Image-based rendering
KW - Sketch synthesis
KW - Vector line art generation
UR - http://www.scopus.com/inward/record.url?scp=85135167908&partnerID=8YFLogxK
U2 - 10.1145/3528223.3530068
DO - 10.1145/3528223.3530068
M3 - ???researchoutput.researchoutputtypes.contributiontojournal.article???
AN - SCOPUS:85135167908
SN - 0730-0301
VL - 41
JO - ACM Transactions on Graphics
JF - ACM Transactions on Graphics
IS - 4
M1 - 3530068
ER -