# US 2024/0203078 A1 · Methods and systems for refining a 3d representation of a subject

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0203078 A1
- Document ID: US20240203078A1
- Status: Published application
- Kind: A1
- Application number: 18/083,940
- Filed: 2022-12-19
- Published: 2024-06-20
- Earliest claimed: 2022-12-19
- Assignees: Verizon Patent and Licensing Inc.
- Applicants: Verizon Patent and Licensing Inc.
- Inventors: Solmaz Hajmohammadi; Richard A. Paris; Daniel M. Michler; Jie Yu
- Web page: https://patentgrep.com/patent/US20240203078A1

## CPC

- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06T 7/55: from multiple images
- G06T 7/90: Determination of colour characteristics
- G06T 15/04: Texture mapping
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation}
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/30168: Image quality inspection
- G06T 2207/30196: Human being; Person
- G06T 2207/30221: Sports video; Sports image
- G06T 2210/36: Level of detail
- G06T 2219/2004: Aligning objects, relative positioning of parts
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

An illustrative volumetric modeling system may obtain a provisional 3D representation of a subject present at a scene and identify a deficiency in the provisional 3D representation. The provisional 3D representation may be based on image data captured by a set of cameras configured to have different fields of view at the scene. The volumetric modeling system may determine a set of parameters for a parameterizable body model associated with a body type of the subject such that an application of the determined set of parameters to the body model may result in a parameterized body model that imitates a pose of the provisional 3D representation of the subject. Based on the provisional 3D representation and the parameterized body model, the volumetric modeling system may then generate a refined 3D representation of the subject in which the deficiency is mitigated. Corresponding methods and systems are also disclosed.

## Claims

### Claim 1

A method comprising:
obtaining, by a volumetric modeling system, a provisional 3D representation of a subject present at a scene, the provisional 3D representation based on image data captured by a set of cameras configured to have different fields of view according to different vantage points the cameras have at the scene;
identifying, by the volumetric modeling system, a deficiency in the provisional 3D representation of the subject;
determining, by the volumetric modeling system, a set of parameters for a body model that is parameterizable to adaptively model a body type of the subject, the determining performed such that an application of the set of parameters to the body model produces a parameterized body model that imitates a pose of the provisional 3D representation of the subject; and
generating, by the volumetric modeling system and based on the provisional 3D representation and the parameterized body model, a refined 3D representation of the subject in which the deficiency is mitigated.

### Claim 2 (depends on 1)

The method of claim 1, wherein:
the subject is a human subject present at the scene;
the provisional 3D representation is a point cloud representation of the human subject; and
the refined 3D representation is a textured mesh representation of the human subject.

### Claim 3 (depends on 1)

The method of claim 1, wherein the deficiency in the provisional 3D representation of the subject is identified at a region of the provisional 3D representation that corresponds to a region of the subject that is not located within any of the different fields of view of the set of cameras.

### Claim 4 (depends on 1)

The method of claim 1, wherein the deficiency in the provisional 3D representation of the subject is identified at a region of the provisional 3D representation that corresponds to a region of the subject that is occluded, within a particular one of the different fields of view in which the region of the subject is located, by an object present at the scene and further located in the particular one of the different fields of view.

### Claim 5 (depends on 1)

The method of claim 1, wherein the deficiency in the provisional 3D representation of the subject is identified at a region of the provisional 3D representation that corresponds to a region of the subject that:
is located within one or more of the different fields of view, and
is represented by the image data at a level of detail less than a threshold level of detail.

### Claim 6 (depends on 1)

The method of claim 1, wherein:
the determining of the set of parameters for the body model is performed using a model-fitting technique associated with an error equation that quantifies a fit of the body model to the pose of the provisional 3D representation; and
the model-fitting technique involves iteratively adjusting and reassessing the fit of the body model to the pose of the provisional 3D representation with an objective of minimizing the error equation until the error equation satisfies a predetermined error threshold that represents an acceptable error in the fit of the body model to the pose of the provisional 3D representation.

### Claim 7 (depends on 6)

The method of claim 6, wherein:
the error equation includes a plurality of terms representing different aspects of the fit of the body model to the pose of the provisional 3D representation; and
the error equation is configured to quantify the fit of the body model to the pose of the provisional 3D representation based on a combination of different assessments of the fit corresponding to the plurality of terms.

### Claim 8 (depends on 6)

The method of claim 6, wherein the error equation includes a joint-based term configured to account for a position similarity between joints of the body model and corresponding joints of the subject as represented in the provisional 3D representation.

### Claim 9 (depends on 6)

The method of claim 6, wherein the error equation includes a vector-based term configured to account for a pose similarity between vectors extending between particular joints of the body model and corresponding vectors extending between particular joints of the subject as represented in the provisional 3D representation.

### Claim 10 (depends on 1)

The method of claim 1, wherein:
the determining of the set of parameters for the body model is performed prior to the identifying of the deficiency in the provisional 3D representation of the subject;
the method further comprises applying, by the volumetric modeling system subsequent to the application of the set of parameters to the body model, texture content to the parameterized body model, the texture content based on the image data captured by the set of cameras; and
the identifying of the deficiency in the provisional 3D representation is performed based on an analysis of the parameterized body model to which the texture content has been applied.

### Claim 11 (depends on 1)

The method of claim 1, wherein:
the image data captured by the set of cameras includes both color data and depth data representative of objects present at the scene; and
the obtaining of the provisional 3D representation of the subject is performed by generating the provisional 3D representation of the subject based on color data and depth data representative of the subject.

### Claim 12 (depends on 1)

The method of claim 1, further comprising providing, by the volumetric modeling system, the refined 3D representation of the subject to an extended reality (XR) presentation device that is configured to provide an extended reality experience to a user and to present the refined 3D representation to the user as part of the extended reality experience.

### Claim 13

A system comprising:
a memory storing instructions; and
one or more processors communicatively coupled to the memory and configured to execute the instructions to perform a process comprising:
obtaining a provisional 3D representation of a subject present at a scene, the provisional 3D representation based on image data captured by a set of cameras configured to have different fields of view according to different vantage points the cameras have at the scene;
identifying a deficiency in the provisional 3D representation of the subject;
determining a set of parameters for a body model that is parameterizable to adaptively model a body type of the subject, the determining performed such that an application of the set of parameters to the body model produces a parameterized body model that imitates a pose of the provisional 3D representation of the subject; and
generating, based on the provisional 3D representation and the parameterized body model, a refined 3D representation of the subject in which the deficiency is mitigated.

### Claim 14 (depends on 13)

The system of claim 13, wherein:
the subject is a human subject present at the scene;
the provisional 3D representation is a point cloud representation of the human subject; and
the refined 3D representation is a textured mesh representation of the human subject.

### Claim 15 (depends on 13)

The system of claim 13, wherein the deficiency in the provisional 3D representation of the subject is identified at a region of the provisional 3D representation that corresponds to a region of the subject that is not located within any of the different fields of view of the set of cameras.

### Claim 16 (depends on 13)

The system of claim 13, wherein the deficiency in the provisional 3D representation of the subject is identified at a region of the provisional 3D representation that corresponds to a region of the subject that is occluded, within a particular one of the different fields of view in which the region of the subject is located, by an object present at the scene and further located in the particular one of the different fields of view.

### Claim 17 (depends on 13)

The system of claim 13, wherein the deficiency in the provisional 3D representation of the subject is identified at a region of the provisional 3D representation that corresponds to a region of the subject that:
is located within one or more of the different fields of view, and
is represented by the image data at a level of detail less than a threshold level of detail.

### Claim 18 (depends on 13)

The system of claim 13, wherein:
the determining of the set of parameters for the body model is performed using a model-fitting technique associated with an error equation that quantifies a fit of the body model to the pose of the provisional 3D representation;
the model-fitting technique involves iteratively adjusting and reassessing the fit of the body model to the pose of the provisional 3D representation with an objective of minimizing the error equation until the error equation satisfies a predetermined error threshold that represents an acceptable error in the fit of the body model to the pose of the provisional 3D representation;
the error equation includes a plurality of terms representing different aspects of the fit of the body model to the pose of the provisional 3D representation; and
the error equation is configured to quantify the fit of the body model to the pose of the provisional 3D representation based on a combination of different assessments of the fit corresponding to the plurality of terms.

### Claim 19 (depends on 13)

The system of claim 13, wherein:
the determining of the set of parameters for the body model is performed prior to the identifying of the deficiency in the provisional 3D representation of the subject;
the process further comprises applying, subsequent to the application of the set of parameters to the body model, texture content to the parameterized body model, the texture content based on the image data captured by the set of cameras; and
the identifying of the deficiency in the provisional 3D representation is performed based on an analysis of the parameterized body model to which the texture content has been applied.

### Claim 20

A non-transitory computer-readable medium storing instructions that, when executed, direct a processor of a computing device to perform a process comprising:
obtaining a provisional 3D representation of a subject present at a scene, the provisional 3D representation based on image data captured by a set of cameras configured to have different fields of view according to different vantage points the cameras have at the scene;
identifying a deficiency in the provisional 3D representation of the subject;
determining a set of parameters for a body model that is parameterizable to adaptively model a body type of the subject, the determining performed such that an application of the set of parameters to the body model produces a parameterized body model that imitates a pose of the provisional 3D representation of the subject; and
generating, based on the provisional 3D representation and the parameterized body model, a refined 3D representation of the subject in which the deficiency is mitigated.
