# US 12,217,527 B2 · Methods and apparatus to determine an audience composition based on voice recognition, thermal imaging, and facial recognition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,217,527 B2
- Document ID: US12217527B2
- Status: Granted
- Kind: B2
- Application number: 18/363,378
- Filed: 2023-08-01
- Published: 2023-12-21
- Granted: 2025-02-04
- Earliest claimed: 2020-08-20
- Assignees: The Nielsen Company (US), LLC
- Applicants: The Nielsen Company (US), LLC
- Inventors: John T. LiVoti; Stanley Wellington Woodruff
- Web page: https://patentgrep.com/patent/US12217527B2

## CPC

- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- H04N 21/4223: Cameras (H04N23/00 takes precedence)
- G06Q 30/0201: Market modelling; Market analysis; Collecting market data
- G06Q 30/0242: Determining effectiveness of advertisements
- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- G06V 10/82: using neural networks
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 40/172: {Classification, e.g. identification}
- G10L 25/51: for comparison or discrimination
- G10L 25/78: Detection of presence or absence of voice signals (switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10)
- H04H 60/45: for identifying users
- H04N 21/25891: {being end-user preferences (retrieval of video data in a video database based on user preferences G06F16/739; arrangements for recognizing users' preferences H04H60/46; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18)}
- H04N 21/42202: {environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes}
- H04N 21/42203: {sound input device, e.g. microphone}
- H04N 21/44218: {Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV programme (methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45)}
- H04N 21/6582: {Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number (arrangements where receivers interact with the broadcast H04H20/38)}
- G10L 19/018: Audio watermarking, i.e. embedding inaudible data in the audio signal
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04H: BROADCAST COMMUNICATION (multiplex communication H04J; pictorial communication aspects of broadcast systems H04N)

## Abstract

Methods, apparatus, systems and articles of manufacture are disclosed. An example apparatus includes an audio detector to determine a first audience count based on signatures of audio data captured in the media environment, a thermal image detector to determine a heat blob count based on a frame of thermal image data captured in the media environment, and an audience image detector to identify at least one audience member based on a comparison of a frame of audience image data with a library of reference audience images, the audience image detector to perform the comparison in response to the first audience count not matching the heat blob count.

## Claims

### Claim 1

An apparatus comprising:
an audio sensor to generate audio monitoring data during presentation of media presented by a media presentation device in a media environment;
a thermal image sensor to generate thermal image data captured in the media environment;
a light image sensor to generate audience image data of a selected field of view of the media environment; and
at least one processor to execute computer readable instructions to perform a set of operations including:
based on the generated thermal image data, determining a thermal-indicated audience count that corresponds to a number of heat blobs within the generated thermal image data;
comparing the thermal-indicated audience count with a previously obtained audience count for the media environment;
determining, based on comparing the thermal-indicated audience count with the previously obtained audience count, that the thermal-indicated audience count differs from the previously obtained audience count;
based on the generated thermal image data, determining a position, within the generated thermal image data, that corresponds to an audience member;
defining the selected field of view of the media environment based on the determined position;
responsive to determining that the thermal-indicated audience count differs from the previously obtained audience count, controlling the light image sensor to generate the audience image data of the selected field of view; and
identifying an audience member based on a comparison of a frame of the audience image data with a library of reference audience images.

### Claim 2 (depends on 1)

The apparatus of claim 1, further including a user input receiver to obtain an input from one or more audience members to indicate the previously obtained audience count.

### Claim 3 (depends on 2)

The apparatus of claim 2, further including a user interface to prompt the one or more audience members to indicate the previously obtained audience count, and wherein the set of operations further include prompting the one or more audience members to indicate a current audience count following a threshold time period after identifying the audience member.

### Claim 4 (depends on 3)

The apparatus of claim 3, wherein the threshold time period is reset upon at least one of a subsequent identification of another audience member, or a subsequent re-identification of the audience member.

### Claim 5 (depends on 1)

The apparatus of claim 1, further including a network communication interface, and wherein the operations further include transmitting, via the network communication interface, audience meter data including data indicative of at least the identified audience member.

### Claim 6 (depends on 5)

The apparatus of claim 5, wherein the audio monitoring data is indicative of a media identifier corresponding to the media presented by the media presentation device, and wherein the transmitted audience meter data further includes the media identifier.

### Claim 7 (depends on 1)

The apparatus of claim 1, wherein the operations further includes:
determining a number of distinct voices represented within the audio monitoring data; and
determining an audio-indicated audience count based on the determined number of distinct voices.

### Claim 8

A method performed by computing system including at least one processor, the method comprising:
obtaining thermal image data of a media environment during presentation of media presented by a media presentation device in the media environment;
determining, based on the thermal image data, a thermal-indicated audience count that corresponds to a number of heat blobs within the generated thermal image data;
comparing the thermal-indicated audience count with a previously obtained audience count for the media environment;
determining, based on comparing the thermal-indicated audience count with the previously obtained audience count, that the thermal-indicated audience count differs from the previously obtained audience count;
determining, based on the thermal image data, a position, within the generated thermal image data, that corresponds to an audience member;
defining a field of view of the media environment based on the determined position;
responsive to determining that the thermal-indicated audience count differs from the previously obtained audience count, controlling a light image sensor to generate audience image data of the defined field of view; and
identifying an audience member based on a comparison of a frame of the audience image data with a library of reference audience images.

### Claim 9 (depends on 8)

The method of claim 8, further including obtaining an input, via a user interface, indicating the previously obtained audience count.

### Claim 10 (depends on 9)

The method of claim 9, further including:
prompting, via the user interface, one or more audience members to indicate the previously obtained audience count, wherein obtaining the input indicating the previously obtained audience count is responsive to prompting the one or more audience members; and
subsequently prompting the one or more audience members to indicate a current audience count following a threshold time period after identifying the audience member.

### Claim 11 (depends on 10)

The method of claim 10, further including resetting the threshold time period upon at least one of a subsequent identification of another audience member, or a subsequent re-identification of the audience member.

### Claim 12 (depends on 8)

The method of claim 8, further including transmitting, via a network communication interface, audience meter data including data indicative of at least the identified audience member.

### Claim 13 (depends on 8)

The method of claim 8, further including: obtaining audio monitoring data, via an audio sensor, during presentation of the media presented by the media presentation device, wherein the audio monitoring data is indicative of a media identifier corresponding to the media presented by the media presentation device.

### Claim 14 (depends on 13)

The method of claim 13, further including:
determining a number of distinct voices represented within the audio monitoring data; and
determining an audio-indicated audience count based on the determined number of distinct voices.

### Claim 15

A non-transitory computer readable storage medium having stored therein instructions that, upon execution by at least one processor, cause performance of:
obtaining thermal image data of a media environment during presentation of media presented by a media presentation device in the media environment;
determining, based on the thermal image data, a thermal-indicated audience count that corresponds to a number of heat blobs within the generated thermal image data;
comparing the thermal-indicated audience count with a previously obtained audience count for the media environment;
determining, based on comparing the thermal-indicated audience count with the previously obtained audience count, that the thermal-indicated audience count differs from the previously obtained audience count;
determining, based on the thermal image data, a position, within the generated thermal image data, that corresponds to an audience member;
defining a field of view of the media environment based on the determined position;
responsive to determining that the thermal-indicated audience count differs from the previously obtained audience count, controlling a light image sensor to generate audience image data of the defined field of view; and
identifying an audience member based on a comparison of a frame of the audience image data with a library of reference audience images.

### Claim 16 (depends on 15)

The non-transitory computer readable storage medium of claim 15, wherein the instructions further cause, upon execution by the at least one processor, performance of:
obtaining an input, via a user interface, indicating the previously obtained audience count.

### Claim 17 (depends on 16)

The non-transitory computer readable storage medium of claim 16, wherein the instructions further cause, upon execution by the at least one processor, performance of:
prompting, via the user interface, one or more audience members to indicate the previously obtained audience count, wherein obtaining the input indicating the previously obtained audience count is responsive to prompting the one or more audience members; and
subsequently prompting the one or more audience members to indicate a current audience count following a threshold time period after identifying the audience member.

### Claim 18 (depends on 17)

The non-transitory computer readable storage medium of claim 17, wherein the instructions further cause, upon execution by the at least one processor, performance of resetting the threshold time period upon at least one of a subsequent identification of another audience member, or a subsequent re-identification of the audience member.

### Claim 19 (depends on 15)

The non-transitory computer readable storage medium of claim 15, wherein the instructions further cause, upon execution by the at least one processor, performance of transmitting, via a network communication interface, audience meter data including data indicative of at least the identified audience member.

### Claim 20 (depends on 15)

The non-transitory computer readable storage medium of claim 15, wherein the instructions further cause, upon execution by the at least one processor, performance of:
obtaining audio monitoring data, via an audio sensor, during presentation of the media presented by the media presentation device, wherein the audio monitoring data is indicative of a media identifier corresponding to the media presented by the media presentation device;
determining a number of distinct voices represented within the audio monitoring data; and
determining an audio-indicated audience count based on the determined number of distinct voices.