# US 2025/0148677 A1 · Program, image processing apparatus, and image processing method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0148677 A1
- Document ID: US20250148677A1
- Status: Published application
- Kind: A1
- Application number: 18/837,241
- Filed: 2023-02-02
- Published: 2025-05-08
- Earliest claimed: 2022-02-17
- Assignees: Sony Group Corporation
- Applicants: Sony Group Corporation
- Inventors: LEO HOSHIKAWA; ATSUSHI IRIE; TAKESHI OHASHI
- Web page: https://patentgrep.com/patent/US20250148677A1

## CPC

- G06T 13/205: {driven by audio data}
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 7/251: {involving models}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 10/82: using neural networks
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 40/23: {Recognition of whole body movements, e.g. for sport training}
- G06V 40/25: {Recognition of walking or running movements, e.g. gait recognition}
- G10H 1/0008: {Associated control or indicating means}
- G10H 1/368: {displaying animated or moving pictures synchronized with the music or audio part}
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30196: Human being; Person
- G06T 2219/2004: Aligning objects, relative positioning of parts
- G10H 2210/076: for extraction of timing, tempo; Beat detection
- G10H 2220/401: 3D sensing, i.e. three-dimensional (x, y, z) position or movement sensing
- G10H 2220/455: Camera input, e.g. analyzing pictures from a video camera and using the analysis results as control data
- G10H 2250/311: Neural networks for electrophonic musical instruments or musical processing, e.g. for musical recognition or control, automatic composition or improvisation
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10H: ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE

## Abstract

The present disclosure relates to a program, an image processing apparatus, and an image processing method capable of appropriately synchronizing movements of a plurality of dancers with music in a video in which the plurality of dancers dances. In a video in which a plurality of dancers dances, an action of a main dancer is synchronized with music, and the action of the main dancer synchronized with the music is synchronized with an operation of a sub-dancer, so that an image in which the sub-dancer is synchronized with the action of the main dancer is generated. This can be applied to video editing software.

## Claims

### Claim 1

A program causing a computer to function as:
an image synchronization unit that generates, on a basis of an image in which a first person makes an action in accordance with predetermined music, an image in which a second person different from the first person makes an action in synchronization with the action of the first person.

### Claim 2 (depends on 1)

The program according to claim 1, causing a computer to further function as:
a music synchronization unit that synchronizes the action of the first person with the music in the image in which the first person makes an action,
wherein the image synchronization unit synchronizes the action of the first person synchronized with the music, with the action of the second person to generate an image in which the second person makes an action synchronized with the action of the first person.

### Claim 3 (depends on 2)

The program according to claim 2, causing a computer to further function as:
a first skeleton information acquisition unit that acquires first skeleton information expressing a movement of a skeleton of the first person based on an image obtained by imaging the action of the first person;
a music feature amount acquisition unit that acquires a feature amount of the music as a music feature amount; and
a second skeleton information acquisition unit that acquires second skeleton information expressing a movement of a skeleton of the second person based on an image obtained by imaging the action of the second person,
wherein the music synchronization unit synchronizes the first skeleton information with the music on a basis of the first skeleton information and the music feature amount to synchronize the action of the first person with the music, and
the image synchronization unit synchronizes the second skeleton information with the first skeleton information that has been synchronized with the music to generate an image in which the second person makes an action in synchronization with the action of the first person.

### Claim 4 (depends on 2)

The program according to claim 2,
wherein the music synchronization unit and the image synchronization unit include a convolutional neural network (CNN).

### Claim 5 (depends on 3)

The program according to claim 3, causing a computer to further function as:
a first feature amount acquisition unit that acquires a feature amount of the first person based on the image in which the first person makes an action, as a first feature amount,
wherein the music synchronization unit adjusts a degree of synchronization between the first skeleton information and the music based on the first feature amount.

### Claim 6 (depends on 3)

The program according to claim 3,
wherein the music synchronization unit searches for a key point of the movement of the first person expressed by the first skeleton information, and synchronizes the first skeleton information with the music on a basis of the searched key point.

### Claim 7 (depends on 6)

The program according to claim 6,
wherein the key point includes a timing at which a pattern of the movement of the first person expressed by the first skeleton information is changed.

### Claim 8 (depends on 7)

The program according to claim 7,
wherein the timing at which the pattern of the movement of the first person expressed by the first skeleton information is changed includes a timing at which the movement of the first person jumps, a timing at which the first person raises and lowers an arm, a timing at which the first person changes a moving direction or turns by changing a body direction, or a timing at which the first person moves at a speed higher or lower than a predetermined speed or moves.

### Claim 9 (depends on 5)

The program according to claim 5,
wherein the image synchronization unit adjusts a degree of synchronization between the second skeleton information and the first skeleton information that has been synchronized with the music, on a basis of the first feature amount.

### Claim 10 (depends on 9)

The program according to claim 9, causing a computer to further function as:
a second feature amount acquisition unit that acquires a feature amount of the second person based on the image in which the second person makes an action, as a second feature amount,
wherein the image synchronization unit adjusts a degree of synchronization between the second skeleton information and the first skeleton information that has been synchronized with the music, on a basis of the first feature amount and the second feature amount.

### Claim 11 (depends on 10)

The program according to claim 10, causing a computer to further function as:
a synthesis unit that synthesizes the first feature amount and the second feature amount at a predetermined synthesis ratio to generate a synthesis feature amount,
wherein the image synchronization unit adjusts a degree of synchronization between the second skeleton information and the first skeleton information that has been synchronized with the music, on a basis of the synthesis feature amount.

### Claim 12 (depends on 10)

The program according to claim 10,
wherein the first feature amount includes information expressing a feature of a moving pattern of the first person and the second feature amount includes information expressing a feature of a moving pattern of the second person.

### Claim 13 (depends on 12)

The program according to claim 12,
wherein the information expressing the features of the moving patterns of the first person and the second person includes information on a movement speed of a body, a leg, or an arm, a jump height, rising/falling, an estimated weight, a type of facial expression, and a direction and a speed of each of a minute body movement, a head movement, and a foot movement of the first person and the second person, a feature amount used in gait recognition, and a latent variable of variational auto encoders (VAE).

### Claim 14 (depends on 1)

The program according to claim 1,
wherein the actions made by the first person and the second person include actions of dancing in accordance with the music.

### Claim 15 (depends on 9)

The program according to claim 9, causing a computer to further function as:
a storage unit that stores the first skeleton information and the first feature amount of a plurality of first persons, as library data,
wherein the image synchronization unit adjusts a degree of synchronization between the second skeleton information and the first skeleton information that has been synchronized with the music, on a basis of the library data of any of the plurality of first persons.

### Claim 16 (depends on 10)

The program according to claim 10,
wherein the second person includes an object defined by a three-dimensional model in a three-dimensional virtual space, and
the second feature amount and the second skeleton information include information set on a basis of the three-dimensional model.

### Claim 17 (depends on 16)

The program according to claim 16,
wherein the second feature amount acquisition unit stores the second feature amount set on the basis of the three-dimensional model, and
the second skeleton information acquisition unit stores the second skeleton information set on the basis of the three-dimensional model in advance.

### Claim 18 (depends on 3)

The program according to claim 3,
wherein the music feature amount acquisition unit acquires the music feature amount based on a tempo of the music.

### Claim 19

An image processing apparatus comprising:
an image synchronization unit that generates, on a basis of an image in which a first person makes an action in accordance with predetermined music, an image in which a second person different from the first person makes an action in synchronization with the action of the first person.

### Claim 20

An image processing method comprising:
generating, on a basis of an image in which a first person makes an action in accordance with predetermined music, an image in which a second person different from the first person makes an action in synchronization with the action of the first person.
