# US 12,119,019 B2 · Privacy-preserving social interaction measurement

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,119,019 B2
- Document ID: US12119019B2
- Status: Granted
- Kind: B2
- Application number: 17/578,217
- Filed: 2022-01-18
- Published: 2023-07-20
- Granted: 2024-10-15
- Earliest claimed: 2022-01-18
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Julian Maclaren; Karolis Misiunas; Vahe Tshitoyan; Brian Foo; Kelly Dobson
- Web page: https://patentgrep.com/patent/US12119019B2

## CPC

- A61B 5/02416: {using photoplethysmograph signals, e.g. generated by infrared radiation (A61B5/14552 takes precedence)}
- A61B 5/7264: {Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems}
- G06F 21/6245: {Protecting personal data, e.g. for financial or medical purposes}
- G06Q 10/10: Office automation; Time management
- G06Q 10/40: Business processes related to social networking or social networking services
- G06Q 50/22: Social work or social welfare, e.g. community support activities or counselling services
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 21/028: using properties of sound source
- G10L 25/51: for comparison or discrimination
- G10L 25/78: Detection of presence or absence of voice signals (switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10)
- H04R 1/04: Structural association of microphone with electric circuitry therefor (in electric hearing aids H04R25/00)
- H04R 1/406: {microphones}
- H04R 3/005: {for combining the signals of two or more microphones (specially adapted for hearing aids H04R25/407)}
- A61B 5/165: {Evaluating the state of mind, e.g. depression, anxiety}
- A61B 5/681: {Wristwatch-type devices}
- A61B 2503/12: Healthy persons not otherwise provided for, e.g. subjects of a marketing survey
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)

## Abstract

Various systems, devices, and methods for social interaction measurement that preserve privacy are presented. An audio signal can be captured using a microphone. The audio signal can be processed using an audio-based machine learning model that is trained to detect the presence of speech. The audio signal can be discarded such that content of the audio signal is not stored after the audio signal is processed using the machine learning model. An indication of whether speech is present within the audio signal can be output based at least in part on processing the audio signal using the audio-based machine learning model.

## Claims

### Claim 1

A wearable device, comprising:
a housing;
an attachment mechanism, attached with the housing, that permits the wearable device to be worn by a wearer;
a microphone housed by the housing;
a photoplethysmography (PPG) sensor, housed by the housing; and
a processing system comprising at least one processor, communicatively coupled with the microphone and housed by the housing, wherein the processing system is configured to:
receive an audio signal from the microphone;
receive a PPG signal from the PPG sensor;
process the audio signal using an audio-based machine learning model stored by the processing system, wherein the audio-based machine learning model is trained to detect a presence of speech;
process the PPG signal received from the PPG sensor using a PPG-based machine learning model stored by the processing system;
using a first output of the audio-based machine learning model and a second output of the PPG-based machine learning model, determine whether the audio signal comprises speech spoken by the wearer of the wearable device;
output an indication of whether speech spoken by the wearer is present within the audio signal based at least in part on processing the audio signal using the audio-based machine learning model; and
discard the audio signal such that the audio signal is not stored after the indication of whether speech is present is output.

### Claim 2 (depends on 1)

The wearable device of claim 1, wherein the processing system is further configured to:
process the first output of the audio-based machine learning model and the second output of the PPG-based machine learning model using a second-level machine learning model, wherein the second-level machine learning model outputs the indication of whether speech is present, wherein:
the indication of whether speech is present is indicative of whether speech spoken by the wearer is present.

### Claim 3 (depends on 2)

The wearable device of claim 2, further comprising a plurality of microphones, that comprises the microphone, in communication with the processing system; and
the processing system is further configured to:
process a plurality of audio streams from the plurality of microphones using a directionality analyzer, wherein:
the second-level machine learning model further uses a third output from the directionality analyzer; and
the indication of whether speech is present is further indicative of whether the wearer is participating in a person-to-person conversation.

### Claim 4 (depends on 1)

The wearable device of claim 1, further comprising:
an inertial measurement unit (IMU), housed by the housing and in communication with the processing system, that outputs an inertial signal to the processing system, wherein the processing system is further configured to:
process the inertial signal received from the IMU using an inertial-based machine learning model stored by the processing system; and
using the first output of the audio-based machine learning model, the second output of the PPG-based machine learning model, and a third output of the inertial-based machine learning model, determine whether the audio signal comprises speech spoken by the wearer of the wearable device.

### Claim 5 (depends on 1)

The wearable device of claim 1, wherein the attachment mechanism comprises a band that permits the wearable device to be worn on a wrist of the wearer.

### Claim 6 (depends on 5)

The wearable device of claim 5, wherein the wearable device is a smartwatch that further comprises:
a graphical electronic display, housed by the housing and in communication with the processing system; and
a wireless communication interface, housed by the housing and in communication with the processing system that enables the processing system to communicate wirelessly with a smartphone.

### Claim 7 (depends on 1)

The wearable device of claim 1, wherein the processing system is further configured to:
calculate an amount of time in a day that the wearer is socially engaged based at least on the indication of whether speech is present within the audio signal; and
cause the amount of time in the day that the wearer is socially engaged to be output to the wearer.

### Claim 8 (depends on 1)

The wearable device of claim 1, wherein the audio signal from the microphone is only input to the audio-based machine learning model.

### Claim 9 (depends on 1)

The wearable device of claim 1, wherein the processing system is further configured to:
perform a signal separation analysis on a plurality of audio signals received from a plurality of microphones of the wearable device, wherein an output of the signal separation analysis is an indication of a number of audio sources.

### Claim 10

A method for social interaction measurement, the method comprising:
capturing, by a microphone of a wearable device, an audio signal;
capturing photoplethysmography (PPG) data using a PPG sensor of the wearable device;
processing, by a processing system of the wearable device, the audio signal using an audio-based machine learning model, wherein the audio-based machine learning model is trained to detect a presence of speech;
processing the PPG data received from the PPG sensor using a PPG-based machine learning model stored by the processing system, wherein the PPG-based machine learning model is trained to detect the presence of speech based on the PPG data;
discarding, by the wearable device, the audio signal such that content of the audio signal is not stored after the audio signal is processed using the audio-based machine learning model;
using a first output of the audio-based machine learning model and a second output of the PPG-based machine learning model, determine whether the audio signal comprises speech spoken by a wearer of the wearable device; and
outputting, by the processing system, an indication of whether speech by the wearer is present within the audio signal based at least in part on processing the audio signal using the audio-based machine learning model.

### Claim 11 (depends on 10)

The method for social interaction measurement of claim 10, further comprising:
processing the first output of the audio-based machine learning model and the second output of the PPG-based machine learning model using a second-level machine learning model, wherein the second-level machine learning model outputs the indication of whether speech is present.

### Claim 12 (depends on 11)

The method for social interaction measurement of claim 11, further comprising:
wherein capturing the audio signal comprises capturing a plurality of audio signals using a plurality of microphones of the wearable device.

### Claim 13 (depends on 12)

The method for social interaction measurement of claim 12, further comprising:
performing a signal separation analysis on the plurality of audio signals received from the plurality of microphones of the wearable device, wherein an output of the signal separation analysis is an indication of a number of audio sources.

### Claim 14 (depends on 12)

The method for social interaction measurement of claim 12, further comprising:
processing a plurality of audio streams from the plurality of microphones using a directionality analyzer, wherein:
the second-level machine learning model further uses a third output from the directionality analyzer; and
the indication of whether speech is present is further indicative of whether the wearer is participating in a face-to-face conversation.