# US 2021/0241488 A1 · Monitoring Icon Status in a Display from an External Device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241488 A1
- Document ID: US20210241488A1
- Status: Published application
- Kind: A1
- Application number: 16/837,467
- Filed: 2020-04-01
- Published: 2021-08-05
- Earliest claimed: 2020-01-31
- Assignees: Gracenote, Inc.
- Applicants: Gracenote, Inc.
- Inventors: Konstantinos Antonio Dimitriou
- Web page: https://patentgrep.com/patent/US20210241488A1

## CPC

- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06F 3/04817: using icons (graphical or visual programming using iconic symbols G06F8/34)
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06T 7/11: Region-based segmentation
- G06T 7/74: {involving reference images or patches}
- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- H04N 23/60: Control of cameras or camera modules
- H04N 23/667: Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
- G06N 3/048: Activation functions
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10141: Special mode during image acquisition
- G06V 2201/02: Recognising information on displays, dials, clocks
- G06V 2201/09: Recognition of logos
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Systems and methods for monitoring of icon in an external display device are disclosed. Images of an icon displayed in a display device may be continually captured as video frames by a video camera of an icon monitoring system. While operating in a first mode, video frames may be continually analyzed to determine if the captured image matches an active template icon known to match the captured image of the icon. While the captured image matches the active template icon, operating in the first mode continues. Upon detecting a failed match to the active template icon, the system starts operating in a second to search among known template icons for a new match. Upon finding a new match, the active template icon may be updated to the new match, and operation switches back to the first mode. Times of transitions between the first and second modes may be recorded.

## Claims

### Claim 1

A method carried out by a system including a computing device and a video camera communicatively connected with the computing device, the method comprising:
operating in a first mode, wherein operating in the first mode comprises:
with the video camera, continually capturing video images in an image plane of the video camera of an icon displayed in a sub-region of a display region of a display device, wherein the display device is external to the computing device and video camera, wherein the sub-region of the display region is fractionally larger than the icon and maps to a corresponding sub-region of the image plane, and wherein the video images captured in the image plane comprise video frames;
for each respective video frame of at least a subset of the video frames, computationally comparing a template icon with the video image of the icon captured in the corresponding sub-region of the image plane during the respective video frame;
for each computational comparison for which the captured image of the icon during the respective video frame matches the template icon to within a matching threshold: (i) recording information associated with the respective video frame, and (ii) continuing operation in the first mode; and
upon occurrence of at least one computational comparison for which the captured image of the icon during a given one of the respective video frames fails to match the template icon to within the matching threshold: (i) recording information associated with the given one of the respective video frames, and (ii) switching to operation in a second mode;
and
operating in the second mode upon switching to operation in the second mode, wherein operating in the second mode comprises:
determining a new template icon that matches the captured image of the icon during the given one of the respective video frames to within the matching threshold;
setting the template icon to the new template icon; and
switching to operation in the first mode.

### Claim 2 (depends on 1)

The method of claim 1, wherein continually capturing the video images in the image plane of the video camera of the icon displayed in the sub-region of the display region of the display device comprises maintaining a threshold-limited fixed orientation of a field-of-view (FOV) direction of the video camera with respect to the display region of the display device during a time in which the video images are continually captured,
wherein the threshold-limited fixed orientation comprises an orientation with respect to the display region of the display device that remains constant to within a threshold amount of misalignment with respect to a fixed direction.

### Claim 3 (depends on 1)

The method of claim 1, wherein continually capturing the video images in the image plane of the video camera of the icon displayed in the sub-region of the display region of the display device comprises capturing a video image of the icon in the corresponding sub-region of the image plane during each respective video frame.

### Claim 4 (depends on 1)

The method of claim 1, wherein computationally comparing the template icon with the video image of the icon captured in the corresponding sub-region of the image plane during the respective video frame comprises computationally comparing the template icon with video image data captured only within the corresponding sub-region of the image plane.

### Claim 5 (depends on 4)

The method of claim 4, wherein computationally comparing the template icon with the video image data captured only within the corresponding sub-region of the image plane comprises applying normalized cross correlation template matching to the video image data captured only within the corresponding sub-region of the image plane.

### Claim 6 (depends on 4)

The method of claim 4, wherein computationally comparing the template icon with the video image data captured only within the corresponding sub-region of the image plane comprises inputting the video image data captured only within the corresponding sub-region of the image plane to an artificial neural network application configured for image recognition and trained with an image of the template icon.

### Claim 7 (depends on 1)

The method of claim 1, wherein each respective video frame has a respective time stamp marking a time during which the image of the icon was captured in image plane,
wherein recording the information associated with the respective video frame comprises recording the time stamp and information identifying the template icon,
and wherein recording the information associated with the given one of the respective video frames comprises recording the time stamp and information indicating the failure to match the template icon to within the matching threshold.

### Claim 8 (depends on 1)

The method of claim 1, wherein determining the new template icon that matches the captured image of the icon during the given one of the respective video frames to within the matching threshold comprises:
computationally comparing the captured image of the icon during the given one of the respective video frames with each of a plurality of candidate template icons until either: a match to within the matching threshold is found for a given candidate template icon, or no match to within the matching threshold is found for any of the plurality of candidate template icons;
if a match is found, setting the new icon to the given candidate template icon; and
if no match is found: (i) in an area of the image plane in which an area of the display region of the display device extending beyond the sub-region is captured, scanning for a further icon that matches one of the plurality of candidate template icons to within the matching threshold, (ii) setting the new icon to the one of the plurality of candidate template icons that matches the further icon, and (iii) setting the sub-region of the display region to a new sub-region in which the further icon is displayed.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
prior to operation in either of the first or second modes, operating in an initialization mode, wherein operation in the initialization mode comprises:
in an area of the image plane in which an area of the entire display region of the display device is captured, scanning for an initial icon that matches an initial template icon to within the matching threshold;
setting the icon to the initial template icon;
setting the sub-region of the display region to an initial sub-region in which the initial icon is displayed; and
switching to operation in the first mode.

### Claim 10

A system comprising:
one or more processors;
a video camera; and
memory storing instructions that, when executed by the one or more processors, cause the system to carry out operations including:
operating in a first mode, wherein operating in the first mode comprises:
with the video camera, continually capturing video images in an image plane of the video camera of an icon displayed in a sub-region of a display region of a display device, wherein the display device is external to the system, wherein the sub-region of the display region is fractionally larger than the icon and maps to a corresponding sub-region of the image plane, and wherein the video images captured in the image plane comprise video frames;
for each respective video frame of at least a subset of the video frames, computationally comparing a template icon with the video image of the icon captured in the corresponding sub-region of the image plane during the respective video frame;
for each computational comparison for which the captured image of the icon during the respective video frame matches the template icon to within a matching threshold: (i) recording information associated with the respective video frame, and (ii) continuing operation in the first mode; and
upon occurrence of at least one computational comparison for which the captured image of the icon during a given one of the respective video frames fails to match the template icon to within the matching threshold: (i) recording information associated with the given one of the respective video frames, and (ii) switching to operation in a second mode;
and
operating in the second mode upon switching to operation in the second mode, wherein operating in the second mode comprises:
determining a new template icon that matches the captured image of the icon during the given one of the respective video frames to within the matching threshold;
setting the template icon to the new template icon; and
switching to operation in the first mode.

### Claim 11 (depends on 10)

The system of claim 10, wherein continually capturing the video images in the image plane of the video camera of the icon displayed in the sub-region of the display region of the display device comprises maintaining a threshold-limited fixed orientation of a field-of-view (FOV) direction of the video camera with respect to the display region of the display device during a time in which the video images are continually captured,
wherein the threshold-limited fixed orientation comprises an orientation with respect to the display region of the display device that remains constant to within a threshold amount of misalignment with respect to a fixed direction.

### Claim 12 (depends on 10)

The system of claim 10, wherein continually capturing the video images in the image plane of the video camera of the icon displayed in the sub-region of the display region of the display device comprises capturing a video image of the icon in the corresponding sub-region of the image plane during each respective video frame.

### Claim 13 (depends on 10)

The system of claim 10, wherein computationally comparing the template icon with the video image of the icon captured in the corresponding sub-region of the image plane during the respective video frame comprises computationally comparing the template icon with video image data captured only within the corresponding sub-region of the image plane.

### Claim 14 (depends on 13)

The system of claim 13, wherein computationally comparing the template icon with the video image data captured only within the corresponding sub-region of the image plane comprises applying normalized cross correlation template matching to the video image data captured only within the corresponding sub-region of the image plane.

### Claim 15 (depends on 13)

The system of claim 13, wherein computationally comparing the template icon with the video image data captured only within the corresponding sub-region of the image plane comprises inputting the video image data captured only within the corresponding sub-region of the image plane to an artificial neural network application configured for image recognition and trained with an image of the template icon.

### Claim 16 (depends on 10)

The system of claim 10, wherein each respective video frame has a respective time stamp marking a time during which the image of the icon was captured in image plane,
wherein recording the information associated with the respective video frame comprises recording the time stamp and information identifying the template icon,
and wherein recording the information associated with the given one of the respective video frames comprises recording the time stamp and information indicating the failure to match the template icon to within the matching threshold.

### Claim 17 (depends on 10)

The system of claim 10, wherein determining the new template icon that matches the captured image of the icon during the given one of the respective video frames to within the matching threshold comprises:
computationally comparing the captured image of the icon during the given one of the respective video frames with each of a plurality of candidate template icons until either: a match to within the matching threshold is found for a given candidate template icon, or no match to within the matching threshold is found for any of the plurality of candidate template icons;
if a match is found, setting the new icon to the given candidate template icon; and
if no match is found: (i) in an area of the image plane in which an area of the display region of the display device extending beyond the sub-region is captured, scanning for a further icon that matches one of the plurality of candidate template icons to within the matching threshold, (ii) setting the new icon to the one of the plurality of candidate template icons that matches the further icon, and (iii) setting the sub-region of the display region to a new sub-region in which the further icon is displayed.

### Claim 18 (depends on 10)

The system of claim 10, wherein the operations further include:
prior to operation in either of the first or second modes, operating in an initialization mode, wherein operation in the initialization mode comprises:
in an area of the image plane in which an area of the entire display region of the display device is captured, scanning for an initial icon that matches an initial template icon to within the matching threshold;
setting the icon to the initial template icon;
setting the sub-region of the display region to an initial sub-region in which the initial icon is displayed; and
switching to operation in the first mode.

### Claim 19 (depends on 10)

The system of claim 10, further comprising a further video camera,
wherein operating in the first mode further comprises:
with the further video camera, continually capturing further video images in an image plane of the further video camera of a further icon displayed in a sub-region of a display region of a further display device, wherein the further display device is external to the system, wherein the sub-region of the display region of the further display device is fractionally larger than the further icon, and maps to a corresponding sub-region of the image plane of the further video camera, and wherein the further video images captured in the image plane of the further video camera comprise further video frames;
for each respective further video frame of at least a subset of the further video frames, computationally comparing a further template icon with the further video image of the further icon captured in the corresponding sub-region of the image plane of the further video camera during the respective further video frame;
for each computational comparison for which the captured image of the further icon during the respective further video frame matches the further template icon to within a further matching threshold: (i) recording information associated with the respective further video frame, and (ii) continuing operation in the first mode; and
upon occurrence of at least one computational comparison for which the captured image of the further icon during a given one of the respective further video frames fails to match the further template icon to within the further matching threshold: (i) recording information associated with the given one of the respective further video frames, and (ii) switching to operation in a second mode;
and wherein operating in the second mode further comprises:
determining a new further template icon that matches the captured image of the further icon during the given one of the respective further video frames to within the further matching threshold;
setting the further template icon to the new further template icon; and
switching to operation in the first mode.

### Claim 20

A non-transitory computer-readable storage medium, having stored thereon program instructions that, upon execution by one or more processors of a system comprising a video camera, cause the system to carry out operations including:
operating in a first mode, wherein operating in the first mode comprises:
with the video camera, continually capturing video images in an image plane of the video camera of an icon displayed in a sub-region of a display region of a display device, wherein the display device is external to the system, wherein the sub-region of the display region is fractionally larger than the icon and maps to a corresponding sub-region of the image plane, and wherein the video images captured in the image plane comprise video frames;
for each respective video frame of at least a subset of the video frames, computationally comparing a template icon with the video image of the icon captured in the corresponding sub-region of the image plane during the respective video frame;
for each computational comparison for which the captured image of the icon during the respective video frame matches the template icon to within a matching threshold: (i) recording information associated with the respective video frame, and (ii) continuing operation in the first mode; and
upon occurrence of at least one computational comparison for which the captured image of the icon during a given one of the respective video frames fails to match the template icon to within the matching threshold: (i) recording information associated with the given one of the respective video frames, and (ii) switching to operation in a second mode;
and
operating in the second mode upon switching to operation in the second mode, wherein operating in the second mode comprises:
determining a new template icon that matches the captured image of the icon during the given one of the respective video frames to within the matching threshold;
setting the template icon to the new template icon; and
switching to operation in the first mode.
