# US 2021/0082469 A1 · Automatic detection and remediation of video irregularities

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0082469 A1
- Document ID: US20210082469A1
- Status: Published application
- Kind: A1
- Application number: 16/569,335
- Filed: 2019-09-12
- Published: 2021-03-18
- Earliest claimed: 2019-09-12
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: SARBAJIT K. RAKSHIT; JOHN M. GANCI, JR.; JAMES E. BOSTICK; MARTIN G. KEEN
- Web page: https://patentgrep.com/patent/US20210082469A1

## CPC

- G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals
- G06T 7/0002: {Inspection of images, e.g. flaw detection}
- G06T 7/194: involving foreground-background segmentation
- G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44)
- G06V 20/48: {Matching video sequences}
- G11B 27/28: by using information signals recorded by the same method as the main recording {(G11B27/22 takes precedence)}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/30168: Image quality inspection
- G06V 20/44: {Event detection}
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An embodiment includes receiving, by one or more processors, a scene reference that corresponds to a reference state of the scene. The embodiment also includes comparing, by one or more processors, the candidate portion and the scene reference such that the comparing provides an indication of an irregularity in the candidate portion. The embodiment also includes responsive to the indication of the irregularity in the candidate portion, automatically evaluating the irregularity, using one or more processors, such that the evaluating generates a severity value associated with the irregularity. The embodiment also includes, responsive to the generating of the severity value, automatically initiating, by one or more processors, a selected remedial action from among a plurality of remedial action options based at least in part on the severity value, wherein the automatic initiation of the selected remedial action includes automatic actuation of an element of video production equipment.

## Claims

### Claim 1

A computer-implemented method comprising:
receiving, by one or more processors, a candidate portion of a video segment that depicts a scene;
receiving, by one or more processors, a scene reference that corresponds to a reference state of the scene;
comparing, by one or more processors, the candidate portion and the scene reference such that the comparing provides an indication of an irregularity in the candidate portion;
responsive to the indication of the irregularity in the candidate portion, automatically evaluating the irregularity, using one or more processors, such that the evaluating generates a severity value associated with the irregularity; and
responsive to the generating of the severity value, automatically initiating, by one or more processors, a selected remedial action from among a plurality of remedial action options based at least in part on the severity value, wherein the automatic initiation of the selected remedial action includes automatic actuation of an element of video production equipment.

### Claim 2 (depends on 1)

The method of claim 1, wherein the candidate portion comprises an image captured from a segment in the video segment, wherein the scene reference comprises a reference image, and wherein the comparing compares the image captured from the segment and the reference image.

### Claim 3 (depends on 1)

The method of claim 1, wherein the candidate portion comprises a text overlay on a segment in the video segment, wherein the scene reference comprises a reference text, and wherein the comparing compares the text overlay and the reference text.

### Claim 4 (depends on 3)

The method of claim 3, wherein the automatic actuation comprises automatically issuing a command to the video production equipment in order to revise the text overlay of the candidate portion.

### Claim 5 (depends on 1)

The method of claim 1, wherein the candidate portion comprises an audio track associated with the video segment, wherein the scene reference comprises a reference audio track, and wherein the comparing compares the audio track associated with the video segment and the reference audio track.

### Claim 6 (depends on 5)

The method of claim 5, wherein the automatic actuation comprises automatically issuing a command to the video production equipment in order to revise the audio track of the candidate portion.

### Claim 7 (depends on 1)

The method of claim 1, wherein the receiving of the candidate portion includes receiving a video stream that includes a plurality of video segments, the candidate portion being one of the plurality of video segments.

### Claim 8 (depends on 7)

The method of claim 7, wherein the receiving of the scene reference includes receiving a reference video stream that includes a plurality of reference video segments, the scene reference being one of the plurality of reference video segments.

### Claim 9 (depends on 1)

The method of claim 1, wherein the scene reference includes at least a portion of video storyboard.

### Claim 10 (depends on 1)

The method of claim 1, wherein the irregularity in the candidate portion relates to a visual appearance of an element depicted by the candidate portion.

### Claim 11 (depends on 1)

The method of claim 1, wherein the irregularity in the candidate portion relates to a location of an element in the scene depicted by the candidate portion.

### Claim 12 (depends on 1)

The method of claim 1, wherein the evaluating of the of the irregularity includes:
estimating a foreground region and a background region of the candidate portion; and
determining whether the irregularity is in the background region or in the foreground region of the candidate portion,
wherein the severity value depends at least in part on whether the irregularity is in the foreground region or in the background region of the candidate portion.

### Claim 13 (depends on 12)

The method of claim 12, wherein the irregularity relates to an element depicted by the candidate portion that is outside an intended scope of an image topic.

### Claim 14 (depends on 1)

The method of claim 1, wherein the irregularity relates to a shadow depicted by the candidate portion, and wherein the severity value is dependent upon a pixel value of the shadow.

### Claim 15 (depends on 1)

The method of claim 1, wherein the evaluating of the of the irregularity includes using a classifier model to generate the severity value for the irregularity.

### Claim 16 (depends on 1)

The method of claim 1, wherein the automated actuation comprises presenting a notification on a display component of the video production equipment.

### Claim 17 (depends on 1)

The method of claim 1, wherein the automated actuation comprises automatically changing a setting on a component of the video production equipment in order to correct the irregularity.

### Claim 18

A computer usable program product comprising a computer-readable storage device, and program instructions stored on the storage device, the stored program instructions comprising:
program instructions to receive, by one or more processors, a candidate portion of a video segment that depicts a scene;
program instructions to receive, by one or more processors, a scene reference that corresponds to a reference state of the scene;
program instructions to compare, by one or more processors, the candidate portion and the scene reference such that the comparing provides an indication of an irregularity in the candidate portion;
responsive to the indication of the irregularity in the candidate portion, program instructions to automatically evaluate the irregularity, using one or more processors, such that the evaluating generates a severity value associated with the irregularity; and
responsive to the generating of the severity value, program instructions to automatically initiate, by one or more processors, a selected remedial action from among a plurality of remedial action options based at least in part on the severity value, wherein the automatic initiation of the selected remedial action includes automatic actuation of an element of video production equipment.

### Claim 19 (depends on 18)

The computer usable program product of claim 18, wherein the candidate portion comprises an image captured from a segment in the video segment, wherein the scene reference comprises a reference image, and wherein the comparing compares the image captured from the segment and the reference image.

### Claim 20

A computer system comprising a processor, a computer-readable memory, and a computer-readable storage device, and program instructions stored on the storage device for execution by the processor via the memory, the stored program instructions comprising:
program instructions to receive, by one or more processors, a candidate portion of a video segment that depicts a scene;
program instructions to receive, by one or more processors, a scene reference that corresponds to a reference state of the scene;
program instructions to compare, by one or more processors, the candidate portion and the scene reference such that the comparing provides an indication of an irregularity in the candidate portion;
responsive to the indication of the irregularity in the candidate portion, program instructions to automatically evaluate the irregularity, using one or more processors, such that the evaluating generates a severity value associated with the irregularity; and
responsive to the generating of the severity value, program instructions to automatically initiate, by one or more processors, a selected remedial action from among a plurality of remedial action options based at least in part on the severity value, wherein the automatic initiation of the selected remedial action includes automatic actuation of an element of video production equipment.
