# US 12,008,798 B2 · Automated facial recognition library construction with privacy protection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,798 B2
- Document ID: US12008798B2
- Status: Granted
- Kind: B2
- Application number: 17/558,991
- Filed: 2021-12-22
- Published: 2023-06-22
- Granted: 2024-06-11
- Earliest claimed: 2021-12-22
- Assignees: The Nielsen Company (US), LLC
- Applicants: The Nielsen Company (US), LLC
- Inventors: Kevin Keqiang Deng
- Web page: https://patentgrep.com/patent/US12008798B2

## CPC

- G06V 10/762: using clustering, e.g. of similar faces in social networks
- G06F 21/6254: {by anonymising data, e.g. decorrelating personal data from the owner's identification}
- G06V 10/763: {Non-hierarchical techniques, e.g. based on statistics of modelling distributions}
- G06V 40/168: {Feature extraction; Face representation}
- G06V 40/172: {Classification, e.g. identification}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Methods and systems for automated construction of an anonymized facial recognition library are disclosed. A camera of a client device may capture a first plurality of images of faces of members of a panel of viewers of media content presented on a content presentation device collocated with the client device during viewing sessions. A first machine learning (ML) model may be applied to the first plurality to generate a second plurality of feature vectors, each associated with a different one of the images. One or more clusters of feature vectors of the second plurality may be computationally determined within a vector space of the feature vectors. A respective centroid feature vector may be determined for each respective cluster, and assigned a unique ID. A respective association between each cluster ID and a respective name ID may be determined based on panel-member information received at the client device.

## Claims

### Claim 1

A method for automated construction of an anonymized facial recognition library carried out by a client device comprising one or more processors and a camera, the method comprising:
with the camera, capturing a first plurality of images of faces of members of a panel of viewers of media content presented on a content presentation device collocated with the client device during one or more viewing sessions carried out over a particular time interval;
applying a first machine learning (ML) model implemented on the client device to the first plurality to generate a second plurality of feature vectors, each associated with a different one of the images of the faces, wherein the first ML model is trained to extract facial-image feature vectors from images that contain one or more faces;
computationally determining one or more clusters of feature vectors of the second plurality within a vector space of the feature vectors;
for each respective cluster in the vector space, determining a respective centroid feature vector marking the center of the respective cluster, and respectively assigning a unique cluster identifier (ID) to the respective centroid feature vector, wherein the anonymized facial recognition library comprises the respective centroid feature vectors and their respectively assigned unique cluster IDs; and
determining a respective association between each cluster ID and a respective name ID based on panel-member information received at the client device, wherein each respective name ID is unique to a different one of the panel members, and wherein each respective association associates one of the respective name IDs with one or more cluster IDs.

### Claim 2 (depends on 1)

The method of claim 1, wherein capturing the first plurality of images of the faces of members of the panel during the one or more viewing sessions carried out over the particular time interval comprises: during each respective viewing session, capturing at least one facial image of each of those panel members present at the respective viewing session,
and wherein computationally determining the one or more clusters of the feature vectors of the second plurality comprises:
successively aggregating the feature vectors generated from each successive viewing session of the one or more viewing sessions into a session group of feature vectors; and
successively determining one or more candidate clusters of feature vectors for the session group after each aggregation.

### Claim 3 (depends on 2)

The method of claim 2, wherein computationally determining the one or more clusters of the feature vectors of the second plurality further comprises:
determining a particular successive viewing session beyond which the respective centroids of candidate clusters determined for each successive aggregation remain unchanged to within a threshold deviation,
wherein the particular time interval spans at least all of the successive viewing sessions up to and including the particular successive viewing session.

### Claim 4 (depends on 1)

The method of claim 1, wherein capturing the first plurality of images of the faces of members of the panel during the one or more viewing sessions carried out over the particular time interval comprises:
capturing one or more versions of facial images of each of the members of the panel over a course of the one or more viewing sessions, wherein the one or more versions of facial images of any given panel member correspond to one or more images of the given panel member's face captured under different capture conditions, the capture conditions being at least one of lighting, distance from the camera to the given panel member, physical pose of the any given panel member, or physical posture of the given panel member,
and wherein computationally determining the one or more clusters of the feature vectors of the second plurality comprises:
determining respective clusters of feature vectors, each representing one or more versions of the given panel member's facial image that are sufficiently similar for their corresponding feature vectors to cluster in the vector space.

### Claim 5 (depends on 1)

The method of claim 1, wherein computationally determining the one or more clusters of the feature vectors of the second plurality comprises:
determining respective locations of each of the second plurality of feature vectors within the vector space; and
identifying respective regions in the vector space, each region having a respective group of feature vectors all within a first threshold distance from a respective statistical centroid of the respective group, and the respective statistical centroid being separated from every other respective group's statistical centroid by at least a second threshold distance.

### Claim 6 (depends on 1)

The method of claim 1, wherein determining the respective association between each cluster ID and the respective name ID based on the panel-member information received at the client device comprises:
for at least one respective cluster, presenting a digital image corresponding to a representative feature vector of the respective cluster in an interactive user interface, wherein the interactive user interface is implemented in an application program of an end-user computing device communicatively connected to the client device; and
responsive to one or more prompts presented in the interactive user interface requesting an identifying name for each presented digital image, receiving via the interactive interface input associating each respective name ID with one or more respective cluster IDs of the presented digital images.

### Claim 7 (depends on 1)

The method of claim 1, wherein determining the respective association between each cluster ID and the respective name ID based on the panel-member information received at the client device comprises:
receiving panel-member information comprising the respective name ID, age and identified gender of each panel member;
for each respective cluster, determining a representative digital image corresponding to a representative feature vector of the respective cluster, wherein the representative feature vector is one of the respective centroid feature vector of the respective cluster, or one of the feature vectors of the respective cluster;
computationally predicting a statistically most likely match between each representative digital image and the age and identified gender of a different one of the panel members, by applying a second ML model implemented on the client device to the received panel-member information and to the representative digital image of each respective cluster, wherein the second ML model is trained to predict statistical matches between images of people and age and identified gender of the people;
for each match above a threshold likelihood, associating the cluster ID of the representative feature vector corresponding to the representative image with the respective name ID of the age and identified gender of the predicted matching panel member; and
for each match not above the threshold likelihood, prompting a designated panel member via an interactive interface for an identifying name associated with the representative digital image.

### Claim 8 (depends on 7)

The method of claim 7, wherein the panel-member information furthers comprises a height of each panel member,
and wherein the method further comprises:
determining relative heights of images of the panel member derived from the representative digital images; and
using the relative heights to adjust the likelihood of the predicted matches.

### Claim 9 (depends on 1)

The method of claim 1, wherein the client device is communicatively connected with a remote audience measurement server device, and wherein the method further comprises:
providing the respective associations between the cluster IDs and the respective name IDs to the audience measurement server;
capturing one or more audience images of one or more panel members who are present at each of one or more target viewing sessions during which media content is presented on the content presentation device;
for each of the one or more target viewing sessions, applying the first ML model to the one or more audience images to generate one or more corresponding audience feature vectors respectively associated with the one or more audience images, and determining which respective clusters the one or more audience feature vectors belong to; and
transmitting one or more audience reports to the audience measurement server, the one or more audience reports comprising information correlating the one or more target viewing sessions with at least one of: (i) the cluster IDs of the determined respective clusters, or (ii) the name IDs of the panel members present at the one or more target viewing sessions,
wherein the one or more target viewing sessions consist of at least one of: one or more viewing sessions during the particular time interval, or one or more viewing sessions subsequent to the particular time interval.

### Claim 10

A client device configured for automated construction of an anonymized facial recognition library, the client device comprising:
a camera;
one or more processors; and
memory storing instructions that, when executed by the one or more processors, cause the client device to carry out operations including:
with the camera, capturing a first plurality of images of faces of members of a panel of viewers of media content presented on a content presentation device collocated with the client device during one or more viewing sessions carried out over a particular time interval;
applying a first machine learning (ML) model implemented on the client device to the first plurality to generate a second plurality of feature vectors, each associated with a different one of the images of the faces, wherein the first ML model is trained to extract facial-image feature vectors from images that contain one or more faces;
computationally determining one or more clusters of feature vectors of the second plurality within a vector space of the feature vectors;
for each respective cluster in the vector space, determining a respective centroid feature vector marking the center of the respective cluster, and respectively assigning a unique cluster identifier (ID) to the respective centroid feature vector, wherein the anonymized facial recognition library comprises the respective centroid feature vectors and their respectively assigned unique cluster IDs; and
determining a respective association between each cluster ID and a respective name ID based on panel-member information received at the client device, wherein each respective name ID is unique to a different one of the panel members, and wherein each respective association associates one of the respective name IDs with one or more cluster IDs.

### Claim 11 (depends on 10)

The client device of claim 10, wherein capturing the first plurality of images of the faces of members of the panel during the one or more viewing sessions carried out over the particular time interval comprises: during each respective viewing session, capturing at least one facial image of each of those panel members present at the respective viewing session,
and wherein computationally determining the one or more clusters of the feature vectors of the second plurality comprises:
successively aggregating the feature vectors generated from each successive viewing session of the one or more viewing sessions into a session group of feature vectors; and
successively determining one or more candidate clusters of feature vectors for the session group after each aggregation.

### Claim 12 (depends on 11)

The client device of claim 11, wherein computationally determining the one or more clusters of the feature vectors of the second plurality further comprises:
determining a particular successive viewing session beyond which the candidate clusters determined for each successive aggregation remain unchanged to within a threshold deviation,
wherein the particular time interval spans at least all of the successive viewing sessions up to and including the particular successive viewing session.

### Claim 13 (depends on 10)

The client device of claim 10, wherein capturing the first plurality of images of the faces of members of the panel during the one or more viewing sessions carried out over the particular time interval comprises:
capturing one or more versions of facial images of each of the members of the panel over a course of the one or more viewing sessions, wherein the one or more versions of facial images of any given panel member correspond to one or more images of the given panel member's face captured under different capture conditions, the capture conditions being at least one of lighting, distance from the camera to the given panel member, physical pose of the given panel member, or physical posture of the given panel member,
and wherein computationally determining the one or more clusters of the feature vectors of the second plurality comprises:
determining respective clusters of feature vectors, each representing one or more versions of the given panel member's facial image that are sufficiently similar for their corresponding feature vectors to cluster in the vector space.

### Claim 14 (depends on 10)

The client device of claim 10, wherein computationally determining the one or more clusters of the feature vectors of the second plurality comprises:
determining respective locations of each of the second plurality of feature vectors within the vector space; and
identifying respective regions in the vector space, each region having a respective group of feature vectors all within a first threshold distance from a respective statistical centroid of the respective group, and the respective statistical centroid being separated from every other respective group's statistical centroid by at least a second threshold distance.

### Claim 15 (depends on 10)

The client device of claim 10, wherein determining the respective association between each cluster ID and the respective name ID, based on the panel-member information received at the client device comprises:
for at least one respective cluster, presenting a digital image corresponding to a representative feature vector of the respective cluster in an interactive user interface, wherein the interactive user interface is implemented in an application program of an end-user computing device communicatively connected to the client device; and
responsive to one or more prompts presented in the interactive user interface requesting an identifying name for each presented digital image, receiving via the interactive interface input associating each respective name ID with one or more respective cluster IDs of the presented digital images.

### Claim 16 (depends on 10)

The client device of claim 10, wherein determining the respective association between each cluster ID and the respective name ID, based on the panel-member information received at the client device comprises:
receiving panel-member information comprising the respective name ID, age and identified gender of each panel member;
for each respective cluster, determining a representative digital image corresponding to a representative feature vector of the respective cluster, wherein the representative feature vector is one of the respective centroid feature vector of the respective cluster, or one of the feature vectors of the respective cluster;
computationally predicting a statistically most likely match between each representative digital image and the age and identified gender of a different one of the panel members, by applying a second ML model implemented on the client device to the received panel-member information and to the representative digital image of each respective cluster, wherein the second ML model is trained to predict statistical matches between images of people and age and identified gender of the people;
for each match above a threshold likelihood, associating the cluster ID of the representative feature vector corresponding to the representative image with the respective name ID of the age and identified gender of the predicted matching panel member; and
for each match not above the threshold likelihood, prompting a designated panel member via an interactive interface for an identifying name associated with the representative digital image.

### Claim 17 (depends on 16)

The client device of claim 16, wherein the panel-member information furthers comprises a height of each panel member,
and wherein the operations further include:
determining relative heights of images of the panel member derived from the representative digital images; and
using the relative heights to adjust the likelihood of the predicted matches.

### Claim 18 (depends on 10)

The client device of claim 10, wherein the client device is communicatively connected with a remote audience measurement server device, and wherein the operations further include:
providing the respective associations between the cluster IDs and the respective name IDs to the audience measurement server;
capturing one or more audience images of one or more panel members who are present at each of one or more target viewing sessions during which media content is presented on the content presentation device;
for each of the one or more target viewing sessions, applying the first ML model to the one or more audience images to generate one or more corresponding audience feature vectors respectively associated with the one or more audience images, and determining which respective clusters the one or more audience feature vectors belong to; and
transmitting one or more audience reports to the audience measurement server, the one or more audience reports comprising information correlating the one or more target viewing sessions with at least one of: (i) the cluster IDs of the determined respective clusters, or (ii) the name IDs of the panel members present at the one or more target viewing sessions,
wherein the one or more target viewing sessions consist of at least one of: one or more viewing sessions during the particular time interval, or one or more viewing sessions subsequent to the particular time interval.

### Claim 19

A non-transitory computer-readable medium having instructions stored thereon that, when carried out by one or more processors of a client device having a camera, cause the client device to carry out operations including:
with the camera, capturing a first plurality of images of faces of members of a panel of viewers of media content presented on a content presentation device collocated with the client device during one or more viewing sessions carried out over a particular time interval;
applying a first machine learning (ML) model implemented on the client device to the first plurality to generate a second plurality of feature vectors, each associated with a different one of the images of the faces, wherein the first ML model is trained to extract facial-image feature vectors from images that contain one or more faces;
computationally determining one or more clusters of feature vectors of the second plurality within a vector space of the feature vectors;
for each respective cluster in the vector space, determining a respective centroid feature vector marking the center of the respective cluster, and respectively assigning a unique cluster identifier (ID) to the respective centroid feature vector, wherein the anonymized facial recognition library comprises the respective centroid feature vectors and their respectively assigned unique cluster IDs; and
determining a respective association between each cluster ID and a respective name ID based on panel-member information received at the client device, wherein each respective name ID is unique to a different one of the panel members, and wherein each respective association associates one of the respective name IDs with one or more cluster IDs.

### Claim 20 (depends on 19)

The non-transitory computer-readable medium of claim 19, wherein the client device is communicatively connected with a remote audience measurement server device, and wherein the operations further include:
providing the respective associations between the cluster IDs and the respective name IDs to the audience measurement server;
capturing one or more audience images of one or more panel members who are present at each of one or more target viewing sessions during which media content is presented on the content presentation device;
for each of the one or more target viewing sessions, applying the first ML model to the one or more audience images to generate one or more corresponding audience feature vectors respectively associated with the one or more audience images, and determining which respective clusters the one or more audience feature vectors belong to; and
transmitting one or more audience reports to the audience measurement server, the one or more audience reports comprising information correlating the one or more target viewing sessions with at least one of: (i) the cluster IDs of the determined respective clusters, or (ii) the name IDs of the panel members present at the one or more target viewing sessions,
wherein the one or more target viewing sessions consist of at least one of: one or more viewing sessions during the particular time interval, or one or more viewing sessions subsequent to the particular time interval.