# US 2022/0020348 A1 · Video control device and video control method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0020348 A1
- Document ID: US20220020348A1
- Status: Published application
- Kind: A1
- Application number: 17/295,853
- Filed: 2018-11-22
- Published: 2022-01-20
- Earliest claimed: 2018-11-22
- Assignees: Roland Corporation
- Applicants: Roland Corporation
- Inventors: Satoshi Kusakabe; Yasuto Saito
- Web page: https://patentgrep.com/patent/US20220020348A1

## CPC

- G10H 1/40: Rhythm
- G10H 1/0008: {Associated control or indicating means}
- G10H 1/368: {displaying animated or moving pictures synchronized with the music or audio part}
- H04N 5/93: Regeneration of the television signal or of selected parts thereof
- G10H 2210/076: for extraction of timing, tempo; Beat detection
- G10H 2240/325: Synchronizing two or more audio tracks or files according to musical features or musical timings
- G10H 2250/235: Fourier transform; Discrete Fourier Transform [DFT]; Fast Fourier Transform [FFT]
- G10H 2250/251: Wavelet transform, i.e. transform with both frequency and temporal resolution, e.g. for compression of percussion sounds; Discrete Wavelet Transform [DWT]
- G10H 2250/285: Hann or Hanning window
- G10H: ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

This video control device includes: a detection unit that detects a beat timing of audio; and a control unit that updates a display mode of a video on the basis of the beat timing and change information indicating a change in a display mode of a video displayed on a display device.

## Claims

### Claim 1

A video control device comprising:
a detection unit configured to detect a beat timing of an audio; and
a control unit configured to change a display mode of a video displayed on a display device based on the beat timing and change information indicating change content of the display mode.

### Claim 2

(canceled)

### Claim 3 (depends on 1)

The video control device according to claim 1, wherein, the switching from the video to another video is performed as the change in the display mode of the video,
the video includes a first video to which the audio is added and a second video different from the first video, and
the control unit performs a parallel playback process of the first and second videos and performs repeated playback of one of the first and second videos which ends during playback of the other one of the first and second videos.

### Claim 4 (depends on 1)

The video control device according to claim 1, wherein, the switching from the video to another video is performed as the change in the display mode of the video,
the video includes a first video and a second video different from the first video, and
the control unit performs a parallel playback process of the audio, the first video, and the second video and performs repeated playback of the first video or the second video ended during playback of the audio.

### Claim 5 (depends on 1)

The video control device according to claim 1, wherein the control unit changes a parameter of the video related to the effect and performs the addition of the effect to the video as the change in the display mode.

### Claim 6 (depends on 5)

The video control device according to claim 5, wherein the control unit changes intensity of the parameter in accordance with a waveform of a temporally changing predetermined signal.

### Claim 7 (depends on 6)

The video control device according to claim 6, wherein the predetermined signal is a signal of the audio or a signal in which a signal different from a signal of the audio is mixed with the signal of the audio.

### Claim 8 (depends on 1)

The video control device according to claim 1, wherein the control unit changes the display mode using a plurality of continuous detections of the beat timing as a chance.

### Claim 9 (depends on 1)

The video control device according to claim 1, wherein the control unit changes the display mode a plurality of times within an interval of the beat timing.

### Claim 10 (depends on 1)

The video control device according to claim 1, wherein the control unit changes the display mode at an interval of the finally detected beat timing in a non-detection section in which the beat timing is not detected.

### Claim 11 (depends on 1)

The video control device according to claim 1, wherein the control unit determines a timing at which the display mode is changed in a non-detection section in which the beat timing is not detected, based on an interval of the beat timing before and after the non-detection section.

### Claim 12 (depends on 1)

The video control device according to claim 1, wherein the control unit changes the display mode whenever a musical piece which is an audio goes back from a first beat timing by an interval between the first beat timing and a subsequent beat timing in a section from start of the musical piece to the first beat timing.

### Claim 13

A video control device comprising:
a detection unit configured to detect a beat timing of an audio; and
a control unit configured to control a video displayed on a display device based on the beat timing.

### Claim 14 (depends on 13)

The video control device according to claim 13, wherein the control unit switches a video displayed on the display device to another video as controlling of the video.

### Claim 15 (depends on 13)

The video control device according to claim 13,
wherein the video includes a first video having the audio and a second video different from the first video, and
wherein the control unit performs a playback process in which the first video and the second video are parallel and performs repeated playback of one of the first video or the second video ended during playback of the other one of the first video and the second video.

### Claim 16 (depends on 13)

The video control device according to claim 13,
wherein the video includes a first video and a second video different from the first video, and
the control unit performs a parallel playback process of the audio, the first video, and the second video and performs repeated playback of the first video or the second video ended during playback of the audio.

### Claim 17 (depends on 16)

The video control device according to claim 16, wherein the control unit performs at least one of addition of an effect, a change in the effect, and cancellation of the addition on the video displayed on the display device as the controlling of the video.

### Claim 18 (depends on 17)

The video control device according to claim 17, wherein the control unit changes a parameter of the video related to the effect as the change in the video.

### Claim 19 (depends on 17)

The video control device according to claim 17, wherein the control unit changes intensity of the parameter in accordance with a waveform of a temporally changing predetermined signal.

### Claim 20

A video control method comprising:
detecting a beat timing of an audio; and
controlling a video displayed on a display device based on the beat timing.

### Claim 21 (depends on 20)

The video control method according to claim 20, wherein switching a video displayed on the display device to another video is as controlling of the video.

### Claim 22 (depends on 20)

The video control method according to claim 20, wherein performing at least one of addition of an effect, a change in the effect, and cancellation of the addition on the video displayed on the display device is as the controlling of the video.

### Claim 23 (depends on 13)

The video control device according to claim 13:
wherein the audio is attached with the video which is being reproduced, and
wherein the beat timing of the audio is detected by the detection unit from the audio attached with the video which is being reproduced.

### Claim 24 (depends on 23)

The video control device according to claim 23, wherein the detection unit performs a process of generating timing information for governing a beat of an input audio and intensity data indicating power at a timing from data of the input audio, a process of calculating a period and a phase of the beat of the audio using the intensity data, and a process of detecting the beat timing based on the period and the phase of the beat of the audio.

### Claim 25 (depends on 24)

The video control device according to claim 24, wherein the detection unit performs Fourier transform that has an attenuation term on the intensity data with regard to a plurality of beats per minute (BPMs) and calculates, as the period of the beat of the audio, the BPM when an absolute value of a value of the Fourier transform is maximum.

### Claim 26 (depends on 24)

The video control device according to claim 24, wherein the detection unit obtains a plurality of wavelet-transformed values by performing the Fourier transform on a plurality of values obtained by multiplying window functions shifted by a 1/n period of the BPM corresponding to the period of the beat of the audio by the intensity data, and calculates, as the phase of the beat of the audio, a phase when absolute values of the plurality of wavelet-transformed values are maximum, and n is an integer equal to or greater than 2.

### Claim 27 (depends on 24)

The video control device according to claim 24, wherein the detection unit obtains a count value indicating the period of the beat and the phase of the beat, measures the count value using a counter incremented for each sample of a sampling rate, and detects a timing at which a value of the counter reaches the count value as the beat timing.

### Claim 28 (depends on 1)

The video control device according to claim 1, wherein the control unit performs, as the change in the display mode, at least one of:
(i) switching from a video to another video;
(ii) addition of an effect to a video;
(iii) a change in an effect added to a video; and
(iv) cancellation of an effect added to a video.
