# US 2019/0379938 A1 · Computer-implemented system and method for determining attentiveness of user

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0379938 A1
- Document ID: US20190379938A1
- Status: Published application
- Kind: A1
- Application number: 16/357,048
- Filed: 2019-03-18
- Published: 2019-12-12
- Earliest claimed: 2018-06-07
- Assignees: Realeyes Oü
- Applicants: Realeyes Oü
- Inventors: Martin SALO; Elnar HAJIYEV; Attila SCHULC
- Web page: https://patentgrep.com/patent/US20190379938A1

## CPC

- G06Q 30/0277: {Online advertisement}
- H04N 21/4667: {Processing of monitored end-user data, e.g. trend analysis based on the log file of viewer selections}
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06N 20/20: Ensemble learning
- G06Q 30/0242: Determining effectiveness of advertisements
- H04N 21/252: {Processing of multiple end-users' preferences to derive collaborative data}
- H04N 21/25891: {being end-user preferences (retrieval of video data in a video database based on user preferences G06F16/739; arrangements for recognizing users' preferences H04H60/46; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18)}
- H04N 21/42201: {biosensors, e.g. heat sensor for presence detection, EEG sensors or any limb activity sensors worn by the user (input arrangements for interaction with the human body based on nervous system activity detection G06F3/015)}
- H04N 21/44218: {Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV programme (methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45)}
- H04N 21/4666: {using neural networks, e.g. processing the feedback provided by the user}
- H04N 21/84: Generation or processing of descriptive data, e.g. content descriptors {(systems specially adapted for using meta-information in broadcast systems H04H60/73)}
- H04N 21/8547: involving timestamps for synchronizing content
- G06N 3/08: Learning methods
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Disclosed herein is a method and system for collecting attentiveness information associated with a user's response to consuming a piece of media content. The attentiveness information is used to create an attentiveness-labelled behavioural data for the user's response. A computer-implemented attentiveness model may be generated by applying machine learning techniques to the a set of attentiveness-labelled behavioural data from multiple users. The system may comprise an annotation tool that facilitates human labelling of the user's response with attentiveness data. The resulting attentiveness model is therefore based on correlations indicative of attentiveness within the attentiveness-labelled behavioural data and/or physiological data that are based on real human cognition rather than a predetermined feature or combination of features.

## Claims

### Claim 1

A computer-implemented method of determining user attentiveness during media content consumption, the method comprising:
obtaining, at a collection server, response data from a client device, wherein the response data is collected for a user consuming media content on the client device, and wherein the response data comprises a data stream representative of variation over time of the user's behaviour whilst consuming the media content;
associating, at the collection server, the data stream with the media content;
displaying, at an annotation device, a dynamic representation of the response data concurrently with the media content to which it is associated;
receiving, at the annotation device, attentiveness data from an annotator, wherein the attentiveness data is an input score indicative of user attentiveness based on the dynamic representation; and
associating, at the annotation device, the attentiveness data with events in the data stream or media content to generate attentiveness-labelled response data.

### Claim 2 (depends on 1)

The method of claim 1, wherein the data stream comprises information indicative of time evolution of one or more response data parameters.

### Claim 3 (depends on 1)

The method of claim 1, wherein the response data comprises emotional state data obtained from facial image data of the user that is collected during consumption of the media content.

### Claim 4 (depends on 1)

The method of claim 1, wherein associating the data stream with the media content comprises synchronising the response data with the media content.

### Claim 5 (depends on 4)

The method of claim 4, wherein the response data is time stamped, and wherein synchronising the response data with the media content comprising matching time stamp data with media playback status data.

### Claim 6 (depends on 1)

The method of claim 1, wherein the data stream further comprises any of:
media control data,
media tag data,
media attentiveness saliency data, and
media playback status data.

### Claim 7 (depends on 1)

The method of claim 1, wherein generating the attentiveness-labelled response data comprises adding an attentiveness label parameter to the response data.

### Claim 8 (depends on 1)

The method of claim 1, wherein the attentiveness data is selected from any one of:
a binary indicator,
a plurality of preset levels, and
a sliding scale.

### Claim 9 (depends on 1)

The method of claim 1, wherein the response data comprises emotional state data, and wherein the method further comprises deriving a significance score or weighting for the emotion state data based on the received attentiveness data.

### Claim 10 (depends on 1)

The method of claim 1 including controlling the concurrently displayed dynamic representation of the response data and the media content to which it is associated.

### Claim 11 (depends on 1)

The method of claim 1 further comprising:
receiving, at an analysis server, attentiveness data relating to the dynamic representation from a plurality of annotators;
generating, by the analysis server, combined attentiveness data for the dynamic representation, the combined attentiveness data comprising an attentiveness parameter that is indicative of level of positive correlation between the attentiveness data from the plurality of annotators.

### Claim 12 (depends on 11)

The method of claim 11, wherein the input score for the attentiveness data is a binary indicator and the attentiveness parameter is a continuous variable.

### Claim 13 (depends on 1)

The method of claim 1 further comprising:
storing, in a data repository, attentiveness-labelled response data from multiple users;
extracting, from the data repository by an analysis server, an attentiveness-labelled response data training set;
establishing, at the analysis server, an objective for a machine learning algorithm; and
generating, using the machine learning algorithm, an attentiveness model from the attentiveness-labelled response data training set.

### Claim 14 (depends on 13)

The method of claim 13 further comprising:
obtaining, at the collection server, new response data from a client device, wherein the response data is collected for another user consuming media content on the client device, and wherein the response data comprises a data stream representative of variation over time of the user's behaviour whilst consuming the media content;
inputting the new response data to the attentiveness model to obtain an attentiveness score for one or more portions of the new response data.

### Claim 15 (depends on 13)

The method of claim 13, wherein extracting the attentiveness-labelled response data training set comprises applying a filter to the attentiveness-labelled response data in the repository.

### Claim 16 (depends on 13)

The method of claim 13, wherein the attentiveness-labelled response data in the training set comprises media tag data indicative of subject matter in the media content being consumed, and wherein the method further comprises:
obtaining new media tag data for another piece of media content, and
inputting the new media tag data to the attentiveness model to obtain an attractiveness score for one or more portions of the piece of media content.

### Claim 17 (depends on 1)

The method of claim 1 further comprising:
storing, in a data repository, attentiveness-labelled response data from multiple users;
generating, using the attentiveness-labelled response data, a plurality of attentiveness proxy sub-models, wherein each of the attentiveness proxy sub-models is trained based on a respective attentiveness proxy parameter to supply an output indicative of attentiveness from the response data; and
establishing an ensemble model that uses outputs from the plurality of attentiveness proxy sub-models to supply an attentiveness confidence score.

### Claim 18 (depends on 17)

The method of claim 17 further comprising:
obtaining, at the collection server, new response data from a client device, wherein the response data is collected for another user consuming media content on the client device, and wherein the response data comprises a data stream representative of variation over time of the user's behaviour whilst consuming the media content;
inputting the new response data to the plurality of attentiveness proxy sub-models to obtain an attentiveness score for one or more portions of the new response data; and
inputting the outputs of the plurality of attentiveness proxy sub-models to the ensemble model to obtain an attentiveness confidence score associated with the attentiveness score.

### Claim 19

A computer-implemented method of determining user attentiveness during media content consumption, the method comprising:
obtaining, at a collection server, response data from a client device, wherein the response data is collected for a user consuming media content on the client device, and wherein the response data comprises a data stream representative of variation over time of the user's behaviour whilst consuming the media content;
associating, at the collection server, the data stream with the media content;
displaying, at a plurality of annotation devices, a dynamic representation of the response data;
receiving, at the plurality of annotation devices, attentiveness data from an annotator, wherein the attentiveness data is an input score indicative of user attentiveness based on the dynamic representation;
receiving, at an analysis server from the plurality of annotation devices, attentiveness data relating to the dynamic representation from a plurality of annotators;
generating, by the analysis server, combined attentiveness data for the dynamic representation, the combined attentiveness data comprising an attentiveness parameter that is indicative of level of positive correlation between the attentiveness data from the plurality of annotators; and
associating, at the analysis server, the combined attentiveness data with events in the data stream or media content to generate attentiveness-labelled response data.

### Claim 20

A system for determining user attentiveness during media content consumption, the system comprising:
a collection server communicatively coupled via a network to a plurality of client devices, the collection server being configured to:
obtain response data from the plurality of client devices, wherein the response data is collected for a user consuming media content on the client device, and wherein the response data comprises a data stream representative of variation over time of the user's behaviour whilst consuming the media content; and
associate the data stream to the media content; and
an annotation device communicatively coupled to the collection server, the annotation device being configured to:
display a dynamic representation of the response data concurrently with the media content to which it is associated;
receive attentiveness data that comprises an input score indicative of user attentiveness based on the dynamic representation; and
associate the attentiveness data with events in the data stream or media content to generate attentiveness-labelled response data.
