# US 2021/0241324 A1 · Device and method for detecting display of provided credit, and program

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241324 A1
- Document ID: US20210241324A1
- Status: Published application
- Kind: A1
- Application number: 16/972,267
- Filed: 2019-06-03
- Published: 2021-08-05
- Earliest claimed: 2018-06-05
- Assignees: Nippon Telegraph and Telephone Corporation
- Applicants: Nippon Telegraph and Telephone Corporation
- Inventors: Yasunori OISHI; Takahito KAWANISHI; Kaoru HIRAMATSU; Kunio KASHINO
- Web page: https://patentgrep.com/patent/US20210241324A1

## CPC

- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06F 16/00: Information retrieval; Database structures therefor; File system structures therefor
- G06F 16/70: of video data
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G10L 15/00: Speech recognition (G10L17/00 takes precedence)
- H04H 20/14: for monitoring programmes
- H04H 60/37: for identifying segments of broadcast information, e.g. scenes or extracting programme ID
- H04N 17/00: Diagnosis, testing or measuring for television systems or their details
- H04N 21/2187: Live feed
- H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/251: {Learning process for intelligent management, e.g. learning user preferences for recommending movies (details of learning user preferences for the retrieval of video data in a video database G06F16/739; computer systems using learning methods G06N3/08)}
- H04N 21/25435: {involving characteristics of content or additional data, e.g. video resolution or the amount of advertising}
- H04N 21/4394: {involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams (arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58)}
- H04N 21/812: {involving advertisement data (advertising per se G06Q30/02)}
- G06F 3/165: {Management of the audio stream, e.g. setting of volume, audio stream path}
- G06N 3/045: Combinations of networks
- G06N 20/00: Machine learning
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06Q 30/0272: Period of advertisement exposure
- H04H 60/56: Arrangements characterised by components specially adapted for monitoring, identification or recognition covered by groups H04H60/29-H04H60/54
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04H: BROADCAST COMMUNICATION (multiplex communication H04J; pictorial communication aspects of broadcast systems H04N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

The present invention enables detection of a sponsorship credit display in a broadcast program with higher precision. A sponsorship credit display detection device 100 according to the present invention includes: a CM segment detection unit 120 that extracts a cut point, which is a time point where a frame in which the volume of an audio signal of a broadcast program is less than a volume threshold value and the amount of change from a previous frame is at least a pixel change threshold value is played, and detects a CM segment by comparing an interval of the extracted cut point with a CM defined length; a sponsorship credit display segment estimation unit 130 that estimates, as a sponsorship credit display segment, a predetermined time period before or after at least one continuous CM segment detected by the CM segment detection unit 120; and an output unit 140 that outputs information indicating the sponsorship credit display segment.

## Claims

### Claim 1

A sponsorship credit display detection device for detecting, in a broadcast program, a sponsorship credit display that displays a sponsor of that broadcast program, the device comprising:
a commercial message (CM), segment detector configured to:
extract cut points, the cut points include time points where a frame in which the volume of the broadcast program is less than a volume threshold value and an amount of change from a previous frame being at least a pixel change threshold value played, and
detect a CM segment by comparing an interval between the extracted cut points with a CM defined length;
a first sponsorship credit display segment estimator configured to estimate, as a first sponsorship credit display segment, a predetermined time period before or after at least one continuous CM segment detected by the CM segment detection unit; and
a provider configured to provides information indicating the first sponsorship credit display segment.

### Claim 2 (depends on 1)

The sponsorship credit display detection device according to claim 1, wherein the CM segment detector is configured to include:
a CM boundary candidate segment decider configured to:
detect a plurality of low volume segments where the volume is less than the volume threshold value, and when a difference between an interval between the low volume segments and the CM defined length is less than an error threshold value,
decide whether a segment obtained by adding a predetermined time period before and after a low volume segment is a CM boundary candidate segment; and
a cut point extractor configured to extract a cut point within the CM boundary candidate segment; and
a CM segment decider configured to:
select the cut point for each CM boundary candidate segment such that the difference from the CM defined length is small, and
decide that a segment between the selected cut points is a CM segment.

### Claim 3 (depends on 2)

The sponsorship credit display detection device according to claim 2, wherein the CM segment decider is configured to:
generate a plurality of cut point series including a combination of the cut points selected one by one from each of the CM boundary candidate segments,
decide a cut point series that, in adjacent CM boundary candidate segments, minimizes a total difference of adjacent cut point times, which is the time between the cut points selected one by one, and
decide that a segment between the cut points of the cut point series is the CM segment.

### Claim 4 (depends on 1)

The sponsorship credit display detection device according to claim 1, the device further comprising:
an image recognizer configured to:
using a detection model, estimate a second sponsorship credit display segment in the broadcast program for which the sponsorship credit display has not been detected, wherein a parameter to be applied to the detection model has been learned in advance using, in a learning broadcast program in which the sponsorship credit display has been detected, learning data including a still image in which the sponsorship credit display is displayed and a still image in which the sponsorship credit display is not displayed, and
the provider configured to provide information indicating a common portion or a union of the first sponsorship credit display segment and the second sponsorship credit display segment.

### Claim 5 (depends on 1)

The sponsorship credit display detection device according to claim 1, the device further comprising:
a voice recognizer configured to:
from results of voice recognition of an audio signal of the broadcast program, detects a related phrase related to the sponsorship credit display included in an announcement of the sponsorship credit display, and
using an appearance time of the related phrase as a start point, estimate a predetermined period as a third sponsorship credit display segment,
wherein the provider is configured to provide information indicating a common portion or a union of the first sponsorship credit display segment and the third sponsorship credit display segment.

### Claim 6 (depends on 1)

The sponsorship credit display detection device according to claim 1, the device further comprising:
an image recognizer configured to:
using a detection model, estimate a second sponsorship credit display segment in the broadcast program for which the sponsorship credit display has not been detected;
a voice recognizer configured to:
from results of voice recognition of an audio signal of the broadcast program, detect a related phrase related to the sponsorship credit display included in an announcement of the sponsorship credit display, and
using an appearance time of the related phrase as a start point, estimate a predetermined period as a third sponsorship credit display segment,
wherein a parameter to be applied to the detection model has been learned in advance using, in a learning broadcast program in which the sponsorship credit display has been detected, learning data including a still image in which the sponsorship credit display is displayed and a still image in which the sponsorship credit display is not displayed; and
the provider configured to:
provide information indicating a common portion or a union of the first sponsorship credit display segment, the second sponsorship credit display segment, and the third sponsorship credit display segment.

### Claim 7

A sponsorship credit display detection method in a sponsorship credit display detection device for detecting, in a broadcast program, a sponsorship credit display that displays a sponsor of that broadcast program, the method comprising:
extracting cut points, the cut points including time points where a frame in which the volume of an audio signal of the broadcast program is less than a volume threshold value and the amount of change from a previous frame is at least a pixel change threshold value is played;
detecting a CM segment by comparing an interval between the extracted cut points with a CM defined length;
estimating, as a sponsorship credit display segment, a predetermined time period before or after at least one continuous CM segment detected in the CM segment detection step; and
providing information indicating the sponsorship credit display segment.

### Claim 8

A computer program causing a computer to function as the sponsorship credit display detection device, the device comprising:
a commercial message (CM) segment detector configured to:
extract cut points, the cut points include time points where a frame in which the volume of the broadcast program is less than a volume threshold value and an amount of change from a previous frame being at least a pixel change threshold value played, and
detect a CM segment by comparing an interval between the extracted cut points with a CM defined length;
a first sponsorship credit display segment estimator configured to estimate, as a first sponsorship credit display segment, a predetermined time period before or after at least one continuous CM segment detected by the CM segment detector; and
a provider configured to provides information indicating the first sponsorship credit display segment.

### Claim 9 (depends on 7)

The sponsorship credit display detection method according to claim 7, the method further comprising:
detecting a plurality of low volume segments where the volume is less than the volume threshold value, and when a difference between an interval between the low volume segments and the CM defined length is less than an error threshold value;
deciding whether a segment obtained by adding a predetermined time period before and after a low volume segment is a CM boundary candidate segment; extracting a cut point within the CM boundary candidate segment;
selecting the cut point for each CM boundary candidate segment such that the difference from the CM defined length is small, and
deciding that a segment between the selected cut points is a CM segment.

### Claim 10 (depends on 9)

The sponsorship credit display detection method according to claim 9, the method further comprising:
generating a plurality of cut point series including a combination of the cut points selected one by one from each of the CM boundary candidate segments;
deciding a cut point series that, in adjacent CM boundary candidate segments, minimizes a total difference of adjacent cut point times, which is the time between the cut points selected one by one; and
deciding that a segment between the cut points of the cut point series is the CM segment.

### Claim 11 (depends on 10)

The sponsorship credit display detection method according to claim 10, the method further comprising:
using a detection model, estimating a second sponsorship credit display segment in the broadcast program for which the sponsorship credit display has not been detected, wherein a parameter to be applied to the detection model has been learned in advance using, in a learning broadcast program in which the sponsorship credit display has been detected, learning data including a still image in which the sponsorship credit display is displayed and a still image in which the sponsorship credit display is not displayed; and
providing information indicating a common portion or a union of the first sponsorship credit display segment and the second sponsorship credit display segment.

### Claim 12 (depends on 10)

The sponsorship credit display detection method according to claim 10, the method further comprising:
from results of voice recognition of an audio signal of the broadcast program, detecting a related phrase related to the sponsorship credit display included in an announcement of the sponsorship credit display; and
using an appearance time of the related phrase as a start point, estimating a predetermined period as a third sponsorship credit display segment,
wherein the provider is configured to provide information indicating a common portion or a union of the first sponsorship credit display segment and the third sponsorship credit display segment.

### Claim 13 (depends on 10)

The sponsorship credit display detection method according to claim 10, the method further comprising:
estimating, by using a detection model, a second sponsorship credit display segment in the broadcast program for which the sponsorship credit display has not been detected;
from results of voice recognition of an audio signal of the broadcast program, detecting a related phrase related to the sponsorship credit display included in an announcement of the sponsorship credit display;
estimating, by using an appearance time of the related phrase as a start point, a predetermined period as a third sponsorship credit display segment,
wherein a parameter to be applied to the detection model has been learned in advance using, in a learning broadcast program in which the sponsorship credit display has been detected, learning data including a still image in which the sponsorship credit display is displayed and a still image in which the sponsorship credit display is not displayed; and providing information indicating:
a common portion or a union of the first sponsorship credit display segment,
the second sponsorship credit display segment, and
the third sponsorship credit display segment.

### Claim 14 (depends on 8)

The computer program according to claim 8 causing the computer to function as the sponsorship credit display detection device, wherein the CM segment detector is configured to include:
a CM boundary candidate segment decider configured to:
detect a plurality of low volume segments where the volume is less than the volume threshold value, and when a difference between an interval between the low volume segments and the CM defined length is less than an error threshold value,
decide whether a segment obtained by adding a predetermined time period before and after a low volume segment is a CM boundary candidate segment; and
a cut point extractor configured to extract a cut point within the CM boundary candidate segment; and
a CM segment decider configured to:
select the cut point for each CM boundary candidate segment such that the difference from the CM defined length is small, and
decide that a segment between the selected cut points is a CM segment.

### Claim 15 (depends on 8)

The computer program according to claim 8 causing the computer to function as the sponsorship credit display detection device, wherein the CM segment decider is configured to:
generate a plurality of cut point series including a combination of the cut points selected one by one from each of the CM boundary candidate segments,
decide a cut point series that, in adjacent CM boundary candidate segments, minimizes a total difference of adjacent cut point times, which is the time between the cut points selected one by one, and
decide that a segment between the cut points of the cut point series is the CM segment.

### Claim 16 (depends on 8)

The computer program according to claim 8 causing the computer to function as the sponsorship credit display detection device, the device further comprising:
an image recognizer configured to:
using a detection model, estimate a second sponsorship credit display segment in the broadcast program for which the sponsorship credit display has not been detected,
wherein a parameter to be applied to the detection model has been learned in advance using, in a learning broadcast program in which the sponsorship credit display has been detected, learning data including a still image in which the sponsorship credit display is displayed and a still image in which the sponsorship credit display is not displayed, and
the provider configured to provide information indicating a common portion or a union of the first sponsorship credit display segment and the second sponsorship credit display segment.

### Claim 17 (depends on 8)

The computer program according to claim 8 causing the computer to function as the sponsorship credit display detection device, the device further comprising:
a voice recognizer configured to:
from results of voice recognition of an audio signal of the broadcast program, detects a related phrase related to the sponsorship credit display included in an announcement of the sponsorship credit display; and
using an appearance time of the related phrase as a start point, estimate a predetermined period as a third sponsorship credit display segment,
wherein the provider is configured to provide information indicating a common portion or a union of the first sponsorship credit display segment and the third sponsorship credit display segment.

### Claim 18 (depends on 8)

The computer program according to claim 8 causing the computer to function as the sponsorship credit display detection device, the device further comprising:
an image recognizer configured to:
using a detection model, estimate a second sponsorship credit display segment in the broadcast program for which the sponsorship credit display has not been detected;
a voice recognizer configured to:
from results of voice recognition of an audio signal of the broadcast program, detect a related phrase related to the sponsorship credit display included in an announcement of the sponsorship credit display, and
using an appearance time of the related phrase as a start point, estimate a predetermined period as a third sponsorship credit display segment,
wherein a parameter to be applied to the detection model has been learned in advance using, in a learning broadcast program in which the sponsorship credit display has been detected, learning data including a still image in which the sponsorship credit display is displayed and a still image in which the sponsorship credit display is not displayed; and
the provider configured to:
provide information indicating a common portion or a union of the first sponsorship credit display segment, the second sponsorship credit display segment, and the third sponsorship credit display segment.

### Claim 19 (depends on 2)

The sponsorship credit display detection device according to claim 2, the device further comprising:
a voice recognizer configured to:
from results of voice recognition of an audio signal of the broadcast program, detects a related phrase related to the sponsorship credit display included in an announcement of the sponsorship credit display, and
using an appearance time of the related phrase as a start point, estimate a predetermined period as a third sponsorship credit display segment,
wherein the provider is configured to provide information indicating a common portion or a union of the first sponsorship credit display segment and the third sponsorship credit display segment.

### Claim 20 (depends on 2)

The sponsorship credit display detection device according to claim 2, the device further comprising:
an image recognizer configured to:
using a detection model, estimate a second sponsorship credit display segment in the broadcast program for which the sponsorship credit display has not been detected;
a voice recognizer configured to:
from results of voice recognition of an audio signal of the broadcast program, detect a related phrase related to the sponsorship credit display included in an announcement of the sponsorship credit display, and
using an appearance time of the related phrase as a start point, estimate a predetermined period as a third sponsorship credit display segment,
wherein a parameter to be applied to the detection model has been learned in advance using, in a learning broadcast program in which the sponsorship credit display has been detected, learning data including a still image in which the sponsorship credit display is displayed and a still image in which the sponsorship credit display is not displayed; and
the provider configured to:
provide information indicating a common portion or a union of the first sponsorship credit display segment, the second sponsorship credit display segment, and the third sponsorship credit display segment.