# US 12,118,817 B2 · Pose data generation device, CG data generation system, pose data generation method, and non-transitory computer readable storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,817 B2
- Document ID: US12118817B2
- Status: Granted
- Kind: B2
- Application number: 17/690,015
- Filed: 2022-03-09
- Published: 2022-06-30
- Granted: 2024-10-15
- Earliest claimed: 2020-03-11
- Assignees: MegaChips Corporation
- Applicants: MegaChips Corporation
- Inventors: Mahito Matsumoto
- Web page: https://patentgrep.com/patent/US12118817B2

## CPC

- G06T 7/251: {involving models}
- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- A61B 5/107: Measuring physical dimensions, e.g. size of the entire body or parts thereof
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06V 10/34: Smoothing or thinning of the pattern; Morphological operations; Skeletonisation
- G06V 10/96: Management of image or video recognition tasks
- G06V 20/64: Three-dimensional [3D] objects
- G06V 40/23: {Recognition of whole body movements, e.g. for sport training}
- G06T 2200/04: involving 3D image data
- G06T 2207/30196: Human being; Person
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)

## Abstract

Provided is a system that estimates pose data of a person with high accuracy at low cost. At a time step at which image data is to be captured and obtained, a pose data generation system obtains pose data based on the image data. At a time step at which image data is not to be obtained, the pose data generation system predicts pose data at a current time step from a previous time step using IMU data and performs interpolation processing to obtain pose data. Thus, even when a rate of obtaining the image data is low, the pose data generation system performs the above interpolation processing using IMU data to obtain pose data with a high frame rate.

## Claims

### Claim 1

A pose data generation device comprising:
image-based pose data obtaining circuitry that obtains image-based pose data, which is data including posture information of a subject, from image data obtained by imaging the subject in a three-dimensional space at a time step t (t is an integer);
inertial measurement unit (IMU) data obtaining processing circuitry that obtains IMU data outputting from an inertial measurement unit attached to the subject, the IMU data including at least one of position data, displacement data, velocity data, acceleration data, and angular velocity data in the three-dimensional space of the inertial measurement unit attached to the subject; and
interpolation circuitry that, at a timing within a period between a time step t and a time step t+k (k is a natural number) wherein the time step t+k is a timing at which image data is to be obtained next to the time step t at which image data has been obtained, performs interpolation processing both on the image-based pose data and the IMU data to obtain interpolation pose data that is interpolated data for pose data of the subject, the interpolation pose data corresponding to pose data at a timing when the IMU data obtaining processing circuitry has obtained the IMU data, and
wherein a frame rate that the image data is to be obtained is less than a frame rate of the interpolated pose data including the image-based pose data and the interpolation pose data.

### Claim 2 (depends on 1)

The pose data generation device according to claim 1, wherein
the image-based pose data obtaining circuitry obtains the image-based pose data, which is three-dimensional data, from the image data.

### Claim 3 (depends on 1)

The pose data generation device according to claim 1, wherein
the image-based pose data obtaining circuitry obtains the image-based pose data, which is two-dimensional data, from the image data,
the IMU data is three-dimensional data, and
the IMU data obtaining processing circuitry performs a three-dimension to two-dimension conversion processing on the IMU data, which is three-dimensional data, to obtain IMU data that is two-dimensional data.

### Claim 4 (depends on 1)

The pose data generation device according to claim 1, wherein
the image-based pose data obtaining circuitry obtains the image-based pose data, which is two-dimensional data, from the image data,
the IMU data is three-dimensional data,
the interpolation pose data is three-dimensional data, and
the pose data generation device further comprising:
error obtaining circuitry that obtains two-dimensional error data, which is data including an error between the image-based pose data and the two-dimensional interpolation pose data, which is data obtained by converting the interpolation pose data, which is three-dimensional data, into two-dimensional data, and
2D/3D conversion circuitry that obtains three-dimensional error data by converting the two-dimensional error data into three-dimensional data,
wherein the interpolation circuitry performs processing for correcting the error based on the three-dimensional error data on the interpolation pose data to obtain interpolation pose data after error correction processing.

### Claim 5 (depends on 1)

The pose data generation device according to claim 1, wherein assuming that a frame rate of the pose data generated by the pose data generation device is R1 fps (R1 is a real number, fps: frames per second), the image-based pose data obtaining circuitry obtains the image-based pose from the image data obtained at a frame rate of 0.5×Ri fps or less.

### Claim 6 (depends on 1)

The pose data generation device according to claim 1, wherein
at the time step at which the image data is to be obtained, the interpolation circuitry compares the image-based pose data obtained from the image data with the interpolation pose data obtained by the interpolation processing,
(1) when a difference between the image-based pose data obtained from the image data and the interpolation pose data obtained by the interpolation processing is larger than a predetermined threshold value, the interpolation circuitry sets the image-based image obtained from the image data as output data from the pose data generation device,
(2) when a difference between the image-based pose data obtained from the image data and the interpolation pose data obtained by the interpolation processing is equal to or less than the predetermined threshold value, the interpolation circuitry sets the interpolation pose data obtained by the interpolation processing as output data from the pose data generation device.

### Claim 7 (depends on 1)

The pose data generation device according to claim 1, wherein the interpolation circuitry performs the interpolation processing by performing processing using a time series filter.

### Claim 8 (depends on 3)

A computer graphics (CG) data generation system comprising:
the pose data generation device according to claim 3; and
CG data generation circuitry that generates two-dimensional CG data from three-dimensional CG data and generates two-dimensional CG data from two-dimensional pose data,
wherein the CG data generation circuitry performs processing for converting three-dimensional CG data to two-dimensional CG data based on conversion parameters, which the IMU data obtaining processing circuitry has used in 3D-to-2D conversion processing for obtaining IMU data which is 2D data, to obtain CG data for two-dimensional synthesis, and synthesize the obtained CG data for two-dimensional synthesis and two-dimensional CG data generated from the two-dimensional pose data to obtain CG data for outputting.

### Claim 9 (depends on 4)

A computer graphics (CG) data generation system comprising:
the pose data generation device according to claim 4;
three-dimensional CG data generation circuitry that performs CG processing based on three-dimensional pose data to generate three-dimensional CG data; and
two-dimensional CG data generation circuitry that performs 3D/2D conversion processing for converting three-dimensional CG data generated by the three-dimensional CG data generation circuitry to two-dimensional data to obtain two-dimensional CG data;
wherein the error obtaining circuitry uses two-dimensional pose data corresponding to the two-dimensional CG data obtained by the two-dimensional CG data generation circuitry as the two-dimensional interpolation pose data, and obtains two-dimensional error data that is data including the two-dimensional interpolation pose data and the image-based pose data.

### Claim 10 (depends on 1)

The pose data generation device according to claim 1, wherein the image-based pose data obtaining circuitry further obtains camera parameters of a camera that captured the image data.

### Claim 11 (depends on 10)

The pose data generation device according to claim 10, wherein the camera parameters include at least one of image angle, aperture, focal length, and shutter speed.

### Claim 12

A pose data generation method comprising:
an image-based pose data obtaining step of obtaining image-based pose data, which is data including posture information of a subject, from image data obtained by imaging the subject in a three-dimensional space at a time step t (t is an integer);
an inertial measurement unit (IMU) data obtaining processing step of obtaining IMU data outputting from an inertial measurement unit attached to the subject, the IMU data including at least one of position data, displacement data, velocity data, acceleration data, and angular velocity data in the three-dimensional space of the inertial measurement unit attached to the subject; and
an interpolation step of, at a timing within a period between a time step t and a time step t+k (k is a natural number) wherein the time step t+k is a timing at which image data is to be obtained next to the time step t at which image data has been obtained, performing interpolation processing both on the image-based pose data and the IMU data to obtain interpolation pose data that is interpolated data for pose data of the subject, the interpolation pose data corresponding to pose data at a timing when the IMU data obtaining processing circuitry has obtained the IMU data, and
wherein a frame rate that the image data is to be obtained is less than a frame rate of the interpolated pose data including the image-based pose data and the interpolation pose data.

### Claim 13 (depends on 12)

A non-transitory computer readable storage medium storing a program for causing a computer to execute the pose data generation method according to claim 12.