# US 2020/0286490 A1 · Methods and systems for processing audio signals containing speech data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0286490 A1
- Document ID: US20200286490A1
- Status: Published application
- Kind: A1
- Application number: 16/852,383
- Filed: 2020-04-17
- Published: 2020-09-10
- Earliest claimed: 2017-10-18
- Assignees: SoapBox Labs Ltd.
- Applicants: SoapBox Labs Ltd.
- Inventors: Patricia Scanlon
- Web page: https://patentgrep.com/patent/US20200286490A1

## CPC

- G10L 17/04: Training, enrolment or model building
- G10L 17/06: Decision making techniques; Pattern matching strategies
- G06F 16/61: Indexing; Data structures therefor; Storage structures
- G06F 16/636: {by using biological or physiological data}
- G06Q 30/0185: {Product, service or business identity fraud}
- G06Q 50/265: {Personal security, identity or safety}
- G10L 15/25: using position of the lips, movement of the lips or face analysis
- G10L 17/10: Multimodal systems, i.e. based on the integration of multiple recognition engines or fusion of expert systems
- G06F 21/32: using biometric data, e.g. fingerprints, iris scans or voiceprints
- G10L 25/78: Detection of presence or absence of voice signals (switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

Methods and systems for processing audio signals containing speech data are disclosed. Biometric data associated with at least one speaker are extracted from an audio input. A match is determined between the extracted biometric data and stored biometric data associated with a consenting user profile, where a consenting user profile is a user profile associated with a record indicating consent to store biometric data. If a match is determined to exist with such a profile, the speech data is stored in an archive after processing. If no such match is determined, or if the extracted biometric data includes data from a speaker not having a consenting user profile, the speech data is discarded, optionally after having been processed. The system and method provides a safeguard against transferring to storage data of users, particularly minors or children, for whom a verified and valid consent has not been obtained from an authorised adult.

## Claims

### Claim 1

A method comprising:
storing one or more user profiles that are each associated with one of one or more users of a computing system, wherein each user profile is associated with a voiceprint that was generated to uniquely characterize voice characteristics of a respective user of the one or more users of the computing system, and wherein at least one of the stored user profiles is a consenting user profile that is associated with a record indicating consent to store biometric data;
processing an audio signal containing speech data from at least one speaker to extract biometric data associated with said at least one speaker, wherein the speech data is received from the at least one speaker at the computing system;
determining whether said extracted biometric data corresponds to a voiceprint associated with a consenting user profile;
responsive to determining that said extracted biometric data corresponds to the voiceprint associated with the consenting user profile, performing at least one of:
(i) processing the speech data; or
(ii) storing the speech data in an archive;
responsive to determining that said extracted biometric data does not correspond to the voiceprint associated with the consenting user profile:
deleting the speech data within a predetermined time period.

### Claim 2 (depends on 1)

The method of claim 1, wherein:
the extracted biometric data includes biometric data associated with more than one speaker, and
determining whether said extracted biometric data corresponds to the voiceprint associated with the consenting user profile comprises determining whether the biometric data of each of said speakers matches a voiceprint associated with a respective consenting user profile.

### Claim 3 (depends on 2)

The method of claim 2, wherein the speech data is deleted within a predetermined period of time responsive to determining that the biometric data of each of said speakers does not corresponds to a voiceprint associated with a respective consenting user profile.

### Claim 4 (depends on 1)

The method of claim 1, wherein the speech data is deleted within a predetermined period of time responsive to determining that the extracted biometric data includes biometric data associated with more than one speaker.

### Claim 5 (depends on 1)

The method of claim 1, wherein responsive to determining that said extracted biometric data does not correspond to the voiceprint associated with the consenting user profile, the speech data is deleted without being processed further.

### Claim 6 (depends on 5)

The method of claim 5, wherein said predetermined time period is immediately after determining that said extracted biometric data does not correspond to the voiceprint associated with the consenting user profile.

### Claim 7 (depends on 1)

The method of claim 1, wherein responsive to determining that said extracted biometric data does not correspond to the voiceprint associated with the consenting user profile, the speech data is processed before being deleted within said predetermined time period.

### Claim 8 (depends on 7)

The method of claim 7, wherein said predetermined time period is immediately after processing the speech data.

### Claim 9 (depends on 1)

The method of claim 1, further comprising creating a consenting user profile, wherein creating a consenting user profile comprises:
verifying the credentials of a first user of the computing system against a data source to ensure that the first user is authorised to provide consent to store speech data;
initialising a user profile associated with a second user, on the instruction of the first user;
receiving speech data of the second user;
extracting biometric data from said second user speech data;
storing said biometric data and associating said biometric data with said user profile; and
storing said user profile as a consenting user profile.

### Claim 10 (depends on 1)

The method of claim 1, wherein determining whether said extracted biometric data corresponds to a voiceprint associated with a consenting user profile further comprises matching additional non-speech biometric data acquired during the same use of the system by the same user against stored non-speech biometric data associated with a consenting user profile.

### Claim 11 (depends on 10)

The method of claim 10, wherein said additional non-speech biometric data is selected from:
a. image data of the user's face;
b. iris pattern data;
c. fingerprint data;
d. hand geometry data;
e. palm blood vessel pattern data
f. retinal blood vessel pattern data;
g. mouth movement data; or
h. behavioural data.

### Claim 12 (depends on 10)

The method of claim 10, wherein the user profile creation further comprises acquiring and storing said non-speech biometric data during profile creation.

### Claim 13 (depends on 1)

The method of claim 1, wherein determining whether said extracted biometric data corresponds to a voiceprint associated with a consenting user profile comprises determining a match against a user profile of a logged-in user.

### Claim 14 (depends on 13)

The method of claim 13, wherein the logging in of a user is responsive to the detection of biometric data associated with the user.

### Claim 15 (depends on 1)

The method of claim 1, wherein determining whether said extracted biometric data corresponds to a voiceprint associated with a consenting user profile comprises determining a match against both consenting user profiles and non-consenting user profiles, wherein a non-consenting user profile is a user profile not associated with a record indicating consent to store biometric data.

### Claim 16 (depends on 15)

The method of claim 15, further comprising creating a non-consenting user profile, wherein creating a non-consenting user profile comprises:
initialising a user profile associated with a third user;
receiving speech data of the third user;
extracting biometric data from said third user speech data;
storing said biometric data and associating said biometric data with said user profile; and
storing said user profile as a non-consenting user profile.

### Claim 17 (depends on 1)

The method of claim 1, further comprising updating the stored biometric data associated with a consenting user profile on the basis of said extracted biometric data.

### Claim 18

A computing system programmed to process an audio signal containing speech data, the computing system comprising:
an audio input;
a visual input;
a data store storing one or more user profiles that are each associated with one of one or more users of a computing system, wherein each user profile is associated with a voiceprint that was generated to uniquely characterize voice characteristics of a respective user of the one or more users of the computing system, and wherein at least one of the stored user profiles is a consenting user profile that is associated with a record indicating consent to store biometric data;
an interface to a storage archive storing speech data; and
a processor, coupled to the audio input, visual input and the data store, to:
extract biometric data from an audio signal received via the audio input;
determine whether said extracted biometric data corresponds to a voiceprint associated with a consenting user profile;
responsive to determining that said extracted biometric data corresponds to the voiceprint associated with the consenting user profile, performing at least one of:
(i) processing the speech data; or
(ii) storing the speech data in an archive;
responsive to determining that said extracted biometric data does not correspond to the voiceprint associated with the consenting user profile:
delete the speech data within a predetermined time period.

### Claim 19

A method, comprising:
storing one or more user profiles that are each associated with one of one or more users of a computing system, wherein each user profile is associated with stored biometric data that uniquely characterizes a respective user of the one or more users of the computing system, and wherein at least one of the stored user profiles is a consenting user profile that is associated with a record indicating consent to store biometric data;
processing said image data of at least one user speaking to extract biometric data associated with said at least one user;
determining whether said extracted biometric data matches stored biometric data associated with a consenting user profile;
responsive to a match determined to exist between said extracted biometric data and a consenting user profile, performing one of:
(iii) processing the image data; and
(iv) storing the image data in an archive;
responsive to a match not determined to exist between said extracted biometric data and a consenting user profile:
deleting the image data within a predetermined time period.

### Claim 20 (depends on 19)

The method of claim 19, wherein processing the image data comprises analysing the image data to determine speech content based on the movements of the user's mouth, lips or jaw, and wherein analysing the data to determine speech content is combined with analysis of an audio signal containing speech data of said user, captured simultaneously with said image data, such that the determination of speech content is based on both audio-based and image-based inputs.
