StylePart: Image-based Shape Part Manipulation

I-Chao Shen1     Li-Wen Su2     Yu-Ting Wu3     Bing-Yu Chen2
The University of Tokyo3     National Taiwan University1     National Taipei University2    

Overview of StylePart. (a) We first project the input image into the GAN latent space, and (b) map the projected GAN latent code w_input to its corresponding shape attributes and viewing angle using a forward shape-consistent latent mapping function. (c) A user can directly manipulate the image shape at the part-level. Then, we obtain the manipulated GAN latent code w_manipulate by (d) mapping the manipulated attributes to the GAN latent space with a backward mapping function. Finally, we (e) synthesize the final edited image without the need of any 3D workflow.

Abstract
Direct part-level manipulation of man-made shapes in an image is desired given its simplicity. However, it is not intuitive given the existing manually created cuboid and cylinder controllers. To tackle this problem, we present StylePart, a framework that enables direct shape manipulation of an image by leveraging generative models of both images and 3D shapes. Our key contribution is a shape-consistent latent mapping function that connects the image generative latent space and the 3D man-made shape attribute latent space. Our method “forwardly maps” the image content to its corresponding 3D shape attributes, where the shape part can be easily manipulated. The attribute codes of the manipulated 3D shape are then “backwardly mapped” to the image latent code to obtain the final manipulated image. By using both forward and backward mapping, an user can edit the image directly without resorting to any 3D workflow. We demonstrate our approach through various manipulation tasks, including part replacement, part resizing, and shape orientation manipulation, and evaluate its effectiveness through extensive ablation studies.

Publication
I-Chao Shen, Li-Wen Su, Yu-Ting Wu, Bing-Yu Chen
StylePart: Image-based Shape Part Manipulation.
The Visual Computer, April 2024.
The Visual Computer 2024 Paper (2.5MB PDF)
Digital library