US 12,762,720 B2Granted

Data conversion device, moving image conversion system, data conversion method, and recording medium

AssigneeNEC Corporation

InventorsYoshitaka Nozaki, Kenichiro Fukushi, Kosuke Nishihara, Kentaro Nakahara

Life of this patent

Estimating the term…

Loading patent…

Abstract

A data conversion device including a feature amount calculation unit that normalizes posture data estimated in each frame constituting moving image data including a synchronization target motion into an angular representation, and calculates a feature amount in an embedded space by inputting the posture data normalized into the angular representation to an encoder, a distance calculation unit that calculates a distance between a feature amount calculated in each frame constituting reference moving image data and a feature amount calculated in each frame constituting synchronization target moving image data, a synchronization processing unit that calculates an optimal path for each frame based on the calculated distance and synchronizes the synchronization target moving image data with the reference moving image data by aligning timings of frames connected by the optimal path, and an output unit that outputs the synchronization target moving image data synchronized with the reference moving image data.

Claims7

Dependent claims

A data conversion device comprising:

a memory storing instructions;
and
a processor connected to the memory and configured to execute the instructions to:
normalize posture data estimated in each frame constituting moving image data including a synchronization target motion into an angular representation by calculating Euler angles formed by connection lines connecting joints of a person extracted from each frame;
calculate a feature amount in an embedded space by inputting the posture data normalized into the angular representation to an encoder including a graph convolutional network that performs graph convolution regarding adjacent joints represented in a skeleton form as a graph structure;
calculate a distance between a feature amount calculated in each frame constituting reference moving image data and a feature amount calculated in each frame constituting synchronization target moving image data using dynamic time warping;
calculate an optimal path for each frame based on the calculated distance;
synchronize the synchronization target moving image data with the reference moving image data by aligning timings of frames connected by the optimal path to align timing of the synchronization target motion included in the synchronization target moving image data with timing of the synchronization target motion included in the reference moving image data, wherein the processor is further configured to execute the instructions to synchronize the synchronization target moving image data with the reference moving image data by:
storing a conversion array used for synchronization of the synchronization target moving image data,
performing inverse conversion of the synchronization target moving image data used as the reference moving image data, by using the conversion array,
setting one of the plurality of pieces of synchronization target moving image data as the reference moving image data,
calculating the optimal path for each frame based on the calculated distance,
synchronizing the synchronization target moving image data with the reference moving image data by aligning timings of frames connected by the optimal path,
storing the conversion array used for synchronization of the synchronization target moving image data in the conversion array storage means, and
synchronizing one piece of the synchronization target moving image data used as the reference moving image data with another piece of the synchronization target moving image data not used as the reference moving image data, with reference to the another piece of the synchronization target moving image data;
and
output the synchronization target moving image data synchronized with the reference moving image data.

14 more elements

A data conversion method executed by a computer, the method comprising—:

normalizing posture data estimated in each frame constituting moving image data including a synchronization target motion into an angular representation by calculating Euler angles formed by connection lines connecting joints of a person extracted from each frame;
inputting the posture data normalized into the angular representation to an encoder including a graph convolutional network that performs graph convolution regarding adjacent joints represented in a skeleton form as a graph structure to calculate a feature amount in an embedded space;
calculating a distance between a feature amount calculated in each frame constituting the reference moving image data and a feature amount calculated in each frame constituting the synchronization target moving image data using dynamic time warping;
calculating an optimal path for each frame based on the calculated distance;
synchronizing the synchronization target moving image data with the reference moving image data by aligning timings of frames connected by the optimal path to align timing of the synchronization target motion included in the synchronization target moving image data with timing of the synchronization target motion included in the reference moving image data, wherein the synchronizing the synchronization target moving image data with the reference moving image data further comprises:
storing a conversion array used for synchronization of the synchronization target moving image data;
performing inverse conversion of the synchronization target moving image data used as the reference moving image data by using the conversion array;
setting one of the plurality of pieces of synchronization target moving image data as the reference moving image data;
calculating the optimal path for each frame based on the calculated distance;
synchronizing the synchronization target moving image data with the reference moving image data by aligning timings of frames connected by the optimal path;
storing the conversion array used for synchronization of the synchronization target moving image data in the conversion array storage means;
and
synchronizing one piece of the synchronization target moving image data used as the reference moving image data with another piece of the synchronization target moving image data not used as the reference moving image data, with reference to the another piece of the synchronization target moving image data;
and
outputting the synchronization target moving image data synchronized with the reference moving image data.

12 more elements

A non-transitory recording medium recording a program for causing a computer to execute:

normalizing posture data estimated in each frame constituting moving image data including a synchronization target motion into an angular representation by calculating Euler angles formed by connection lines connecting joints of a person extracted from each frame;
inputting the posture data normalized into the angular representation to an encoder including a graph convolutional network that performs graph convolution regarding adjacent joints represented in a skeleton form as a graph structure to calculate a feature amount in an embedded space;
calculating a distance between a feature amount calculated in each frame constituting the reference moving image data and a feature amount calculated in each frame constituting the synchronization target moving image data using dynamic time warping;
calculating an optimal path for each frame based on the calculated distance;
synchronizing the synchronization target moving image data with the reference moving image data by aligning timings of frames connected by the optimal path to align timing of the synchronization target motion included in the synchronization target moving image data with timing of the synchronization target motion included in the reference moving image data, wherein the synchronizing the synchronization target moving image data with the reference moving image data further comprises:
storing a conversion array used for synchronization of the synchronization target moving image data;
performing inverse conversion of the synchronization target moving image data used as the reference moving image data by using the conversion array;
setting one of the plurality of pieces of synchronization target moving image data as the reference moving image data;
calculating the optimal path for each frame based on the calculated distance;
synchronizing the synchronization target moving image data with the reference moving image data by aligning timings of frames connected by the optimal path;
storing the conversion array used for synchronization of the synchronization target moving image data in the conversion array storage means;
and
synchronizing one piece of the synchronization target moving image data used as the reference moving image data with another piece of the synchronization target moving image data not used as the reference moving image data, with reference to the another piece of the synchronization target moving image data;
and
outputting the synchronization target moving image data synchronized with the reference moving image data.

12 more elements

Dependencies are inferred from the claim text.

Facts

Application: 18/380,384Filed 16 Oct 2023
Priority: JP 2022-1759592 Nov 2022
Published: US 2024/0144500 A12 May 2024
Granted: 13 Oct 2026
Term ends: Not estimatedThe source records don’t support an estimate.

Classification8 codes

G06T 7/337
Using feature-based methods: involving reference images or patches
G06T 7/248
Using feature-based methods, e.g. the tracking of corners or segments: involving reference images or patches
G06T 7/74
Using feature-based methods: involving reference images or patches
G06T 2207/10016
Image acquisition modality: video; Image sequence