# US 2023/0267671 A1 · Apparatus and method for synchronization with virtual avatar, and system for synchronization with virtual avatar

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0267671 A1
- Document ID: US20230267671A1
- Status: Published application
- Kind: A1
- Application number: 18/102,160
- Filed: 2023-01-27
- Published: 2023-08-24
- Earliest claimed: 2022-02-18
- Assignees: Electronics and Telecommunications Research Institute
- Applicants: Electronics and Telecommunications Research Institute
- Inventors: Dae-Hwan KIM; Ki-Hong KIM; Yong-Wan KIM; Jin-Sung CHOI
- Web page: https://patentgrep.com/patent/US20230267671A1

## CPC

- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06T 7/11: Region-based segmentation
- G06T 7/50: Depth or shape recovery
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/73: using feature-based methods
- G06T 19/003: {Navigation within 3D models or images}
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/764: using classification, e.g. of video objects
- G06T 2207/20044: Skeletonization; Medial axis transform
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30196: Human being; Person
- G06T 2215/16: Using real world measurements to influence rendering
- G06V 2201/07: Target detection
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Disclosed herein is an apparatus for synchronization with a virtual avatar. The apparatus may include a body part detection unit for detecting a human body part in an input 2D image, a visible body part estimation unit for estimating the shape and pose of a visible body part based on the detected body part, an invisible body part generation unit for generating a shape and pose of an invisible body part based on the body part and the shape and pose thereof, a body estimation unit for estimating a full-body shape and pose based on the shape and pose of the visible body part and the shape and pose of the invisible body part, and a virtual avatar synchronization unit for encoding and transmitting the estimated full-body shape and pose information for synchronization with a virtual avatar modeled in advance in a server.

## Claims

### Claim 1

An apparatus for synchronization with a virtual avatar, comprising:
a body part detection unit for detecting a body part of a human in an input 2D image;
a visible body part estimation unit for estimating a shape and pose of a visible body part based on the detected body part;
an invisible body part generation unit for generating a shape and pose of an invisible body part based on the body part and a shape and pose of the body part;
a body estimation unit for estimating a full-body shape and pose based on the shape and pose of the visible body part and the shape and pose of the invisible body part; and
a virtual avatar synchronization unit for encoding and transmitting the estimated full-body shape and pose for synchronization with a virtual avatar that is modeled in advance in a server.

### Claim 2 (depends on 1)

The apparatus of claim 1, further comprising:
an image collection unit for collecting the 2D image.

### Claim 3 (depends on 1)

The apparatus of claim 1, wherein the body part detection unit includes:
a body part detection module for detecting a region in which a body is located in the 2D image as a boxed region; and
a body part classification module for classifying a body part in the boxed region.

### Claim 4 (depends on 3)

The apparatus of claim 3, wherein the body part detection module predicts joint key points of a main part using a deep neural network and detects the boxed region based on the joint key points.

### Claim 5 (depends on 3)

The apparatus of claim 3, wherein the body part classification module divides the boxed region into multiple regions and classifies each of the multiple regions as a body part using a convolutional neural network.

### Claim 6 (depends on 1)

The apparatus of claim 1, wherein the visible body part estimation unit includes:
a joint position detection module for detecting a joint position of the detected body part; and
a visible body part estimation module for estimating a shape and pose of a 3D body part based on the detected joint position.

### Claim 7 (depends on 1)

The apparatus of claim 1, wherein the invisible body part generation unit includes:
an invisible body part estimation module for estimating the shape and pose of the invisible body part based on the shape and pose of the visible body part; and
a body part generation module for generating a shape and pose of an invisible 3D body part based on the estimated shape and pose of the invisible body part.

### Claim 8 (depends on 1)

The apparatus of claim 1, wherein the body estimation unit includes:
a normalization module for normalizing size and orientation information pertaining to the shape and pose of the visible body part and size and orientation information pertaining to the shape and pose of the invisible body part; and
a full-body estimation module for estimating a human full-body shape and pose by fusing the normalized shape and pose of the visible body part and the normalized shape and pose of the invisible body part.

### Claim 9 (depends on 1)

The apparatus of claim 1, wherein the server includes:
a decoding module for decoding the encoded full-body shape and pose; and
a mapping module for mapping the decoded full-body shape and pose to the virtual avatar.

### Claim 10

A method for synchronization with a virtual avatar, comprising:
detecting a body part of a human in an input 2D image;
estimating a shape and pose of a visible body part based on the detected body part;
generating a shape and pose of an invisible body part based on the body part and a shape and pose of the body part;
estimating a full-body shape and pose based on the shape and pose of the visible body part and the shape and pose of the invisible body part; and
encoding and transmitting the estimated full-body shape and pose for synchronization with a virtual avatar that is modeled in advance in a server.

### Claim 11 (depends on 10)

The method of claim 10, further comprising:
collecting the 2D image.

### Claim 12 (depends on 10)

The method of claim 10, wherein detecting the body part includes:
detecting a region in which a body is located in the 2D image as a boxed region; and
classifying a body part in the boxed region.

### Claim 13 (depends on 12)

The method of claim 12, wherein detecting the region as the boxed region comprises predicting joint key points of a main part using a deep neural network and detecting the boxed region based on the joint key points.

### Claim 14 (depends on 12)

The method of claim 12, wherein classifying the body part comprises dividing the boxed region into multiple regions and classifying each of the multiple regions as a body part using a convolutional neural network.

### Claim 15 (depends on 10)

The method of claim 10, wherein estimating the shape and pose of the visible body part includes:
detecting a joint position of the detected body part of the human; and
estimating a shape and pose of a 3D body part based on the detected joint position.

### Claim 16 (depends on 10)

The method of claim 10, wherein generating the shape and pose of the invisible body part includes:
estimating the shape and pose of the invisible body part based on the shape and pose of the body part; and
generating a shape and pose of an invisible 3D body part based on the estimated shape and pose of the invisible body part.

### Claim 17 (depends on 10)

The method of claim 10, wherein estimating the full-body shape and pose includes:
normalizing size and orientation information pertaining to the shape and pose of the visible body part and size and orientation information pertaining to the shape and pose of the invisible body part; and
estimating a human full-body shape and pose by fusing the normalized shape and pose of the visible body part and the normalized shape and pose of the invisible body part.

### Claim 18 (depends on 10)

The method of claim 10, wherein the server decodes the encoded full-body shape and pose and maps the decoded full-body shape and pose to the virtual avatar.

### Claim 19

A system for synchronization with a virtual avatar, comprising:
a body part detection unit for detecting a body part of a human in an input 2D image;
a visible body part estimation unit for estimating a shape and pose of a visible body part based on the detected body part;
an invisible body part generation unit for generating a shape and pose of an invisible body part based on the body part and a shape and pose of the body part;
a body estimation unit for estimating a full-body shape and pose based on the shape and pose of the visible body part and the shape and pose of the invisible body part;
a server for storing a virtual avatar modeled in advance; and
an avatar synchronization unit for encoding and transmitting the estimated full-body shape and pose for synchronization with the avatar modeled in advance in the server.

### Claim 20 (depends on 19)

The system of claim 19, wherein the server includes:
a decoding module for decoding the encoded full-body shape and pose; and
a mapping module for mapping the decoded full-body shape and pose to the virtual avatar.
