US 11,961,266 B2 · Multiview neural human prediction using implicit differentiable renderer for facial expression, body pose shape and clothes performance capture

Abstract

A neural human performance capture framework (MVS-PERF) captures the skeleton, body shape and clothes displacement, and appearance of a person from a set of calibrated multiview images. It addresses the ambiguity of predicting the absolute position in monocular human mesh recovery, and bridges the volumetric representation from NeRF to animation-friendly performance capture. MVS-PERF includes three modules to extract feature maps from multiview images and fuse them to a feature volume, regress the feature volume to a naked human parameters vector, generating an SMPL-X skin-tight body mesh with a skeletal pose, body shape, and expression, and leverage a neural radiance field and a deformation field to infer the clothes as the displacement on the naked body using differentiable rendering. Clothed body mesh is obtained by adding the interpolated displacement vectors to the SMPL-X skin-tight body mesh vertices. The obtained radiance field is used for free-view volumetric rendering of the input subject.

Claims26

Dependent claims

A method programmed in a non-transitory of a device comprising:

acquiring a set of images as input;
and
processing, with a neural network, the set of images, wherein processing includes:
encoding the set of images to one or more features;
regressing the features to human parameters;
fine-tuning the neural network;
and
decoding a query 3D ray to an RGB color and a clothes-to-body displacement, wherein the RGB color is based on the set of images.

5 more elements

Dependencies are inferred from the claim text.

Facts

Application

17/701,991Filed 23 Mar 2022 · us provisional application

Priority

31 Mar 2021

Published

US 2022/0319055 A16 Oct 2022

Granted

16 Apr 2024

Term ends

Not estimatedThe source records don’t support an estimate.

Classification16 codes

G06T 9/002
Image coding: using neural networks
G06T 15/20
Geometric effects: perspective computation
G06T 7/251
Using feature-based methods, e.g. the tracking of corners or segments: involving models
G06T 13/40
Three-dimensional [3D] animation: of characters, e.g. humans, animals or virtual beings

Multiview neural human prediction using implicit differentiable renderer for facial expression, body pose shape and clothes performance capture

Life of this patent

Multiview neural human prediction using implicit differentiable renderer for facial expression, body pose shape and clothes performance capture

Life of this patent

Abstract

Claims26