ICCV 2023

Leveraging Intrinsic Properties for Non-Rigid Garment Alignment

 

Siyou Lin, Boyao Zhou, Zerong Zheng, Hongwen Zhang, Yebin Liu

Tsinghua University

 

 

CVF page | Paper | Supp | Code | Poster

 

One-sentence Overview

A novel method that leverages intrinsic manifold properties for wrinkle-level and texture-level non-rigid garment alignment.

 

 

 

Abstract

We address the problem of aligning real-world 3D data of garments, which benefits many applications such as texture learning, physical parameter estimation, generative modeling of garments, etc. Existing extrinsic methods typically perform non-rigid iterative closest point and struggle to align details due to incorrect closest matches and rigidity constraints. While intrinsic methods based on functional maps can produce high-quality correspondences, they work under isometric assumptions and become unreliable for garment deformations which are highly non-isometric. To achieve wrinkle-level as well as texture-level alignment, we present a novel coarse-to-fine two-stage method that leverages intrinsic manifold properties with two neural deformation fields, in the 3D space and the intrinsic space, respectively. The coarse stage performs a 3D fitting, where we leverage intrinsic manifold properties to define a manifold deformation field. The coarse fitting then induces a functional map that produces an alignment of intrinsic embeddings. We further refine the intrinsic alignment with a second neural deformation field for higher accuracy. We evaluate our method with our captured garment dataset, GarmCap. The method achieves accurate wrinkle-level and texture-level alignment and works for difficult garment types such as long coats.

 

Movitation & Method

 

Accurately aligning garment geometries benefits many applications such as texture learning, physical parameter estimation, generative modeling of garments, etc. However, existing methods cannot achieve wrinkle-level and texture-level alignment, because:

  • existing extrinsic (3D) methods mostly use non-rigid iterative closest point (ICP), facing a tradeoff between rigidity and flexibility;
  • existing intrinsic methods use intrinsic features, which suffer from ambiguities (eigenfunction sign ambiguity, eigenvalue switching and multiplicity of eigenvalues).
Our observations are:
  • Garment deformations being highly non-rigid (e.g., detailed wrinkles) is the major difficulty.
  • Intrinsic shapes, compared to their extrinsic counterparts, are more "flat". In other words, they do not have "wrinkles". However, intrinsic shapes suffer from ambiguities, so they cannot be used for fitting directly.
  • A coarse 3D fitting is enough for resolving said ambiguities using the Functional Maps framework. After this, we can perform another fitting stage in the intrinsic shape, obtaining wrinkle-level alignment.
Our whole pipeline is structured as shown below.

 

 

 

The overall pipeline of our intrinsic garment alignment method consists the following steps.

  1. The coarse stage computes an intrinsic neural deformation field from a linear blend skinning (LBS) initialization. Since coarse alignment do not align fine details, we use it to induce a functional map that resolves the intrinsic embedding ambiguity.
  2. In the refining stage we refine the intrinsic shape matching using another neural deformation field in the intrinsic space.
  3. The final alignment comes from shape transfer through the point-to-point matching given by the refined intrinsic shape correspondences.

Results

 

 

Demo Video