# US 2022/0044688 A1 · Sample-efficient representation learning for real-time latent speaker state characterization

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044688 A1
- Document ID: US20220044688A1
- Status: Published application
- Kind: A1
- Application number: 17/115,382
- Filed: 2020-12-08
- Published: 2022-02-10
- Earliest claimed: 2020-08-04
- Assignees: OTO Systems Inc.
- Applicants: OTO Systems Inc.
- Inventors: Valentin Alain Jean Perret; Nicolas Lucien Perony; Nándor Kedves
- Web page: https://patentgrep.com/patent/US20220044688A1

## CPC

- G10L 17/18: Artificial neural networks; Connectionist approaches
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/048: Activation functions
- G06N 3/049: Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G10L 17/02: Preprocessing operations, e.g. segment selection; Pattern representation or modelling, e.g. based on linear discriminant analysis [LDA] or principal components; Feature selection or extraction
- G10L 17/08: Use of distortion metrics or a particular distance between probe pattern and reference templates
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems, methods, and non-transitory computer-readable media can provide audio waveform data that corresponds to a voice sample to a temporal convolutional network for evaluation. The temporal convolutional network can pre-process the audio waveform data and can output an identity embedding associated with the audio waveform data. The identity embedding associated with the voice sample can be obtained from the temporal convolutional network. Information describing a speaker associated with the voice sample can be determined based at least in part on the identity embedding.

## Claims

### Claim 1

A computer-implemented method comprising:
providing, by a computing system, audio waveform data that corresponds to a voice sample to a temporal convolutional network for evaluation, wherein the temporal convolutional network pre-processes the audio waveform data and outputs an identity embedding associated with the audio waveform data;
obtaining, by the computing system, the identity embedding associated with the voice sample from the temporal convolutional network; and
determining, by the computing system, information describing a speaker associated with the voice sample based at least in part on the identity embedding.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein determining the information describing the speaker further comprises:
determining, by the computing system, an identity of the speaker associated with the voice sample based at least in part on the identity embedding.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein determining the information describing the speaker further comprises:
determining, by the computing system, that the speaker associated with the voice sample matches a known speaker based at least in part on the identity embedding.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the temporal convolutional network is trained based on a triplet loss function that evaluates a plurality of triplets, wherein a triplet includes an anchor voice sample, a positive voice sample that has a threshold level of similarity to the anchor voice sample, and a negative voice sample that does not have a threshold level of similarity to the anchor voice sample.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein a triplet of voice samples is selected based on semi-hard triplet mining.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the temporal convolutional network is trained based on at least one of: contrastive loss, lifted structures loss, N-pair loss, angular loss, or SoftTriple loss.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the temporal convolutional network embeds the voice sample in a continuous high-dimensional identity space.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein the temporal convolutional network includes a plurality of temporal convolutional neural residual blocks (TCN blocks) that pre-process the audio waveform data.

### Claim 9 (depends on 8)

The computer-implemented method of claim 8, wherein a TCN block applies a set of temporal causal convolutions to the audio waveform data.

### Claim 10 (depends on 8)

The computer-implemented method of claim 8, wherein a TCN block includes a pair of consecutive one-dimensional convolutional layers using causal padding, and wherein the pair of consecutive one-dimensional convolutional layers are bypassed by an additive skip connection.

### Claim 11

A system comprising:
at least one processor; and
a memory storing instructions that, when executed by the at least one processor, cause the system to perform a method comprising:
providing audio waveform data that corresponds to a voice sample to a temporal convolutional network for evaluation, wherein the temporal convolutional network pre-processes the audio waveform data and outputs an identity embedding associated with the audio waveform data;
obtaining the identity embedding associated with the voice sample from the temporal convolutional network; and
determining information describing a speaker associated with the voice sample based at least in part on the identity embedding.

### Claim 12 (depends on 11)

The system of claim 11, wherein determining the information describing the speaker further causes the system to perform:
determining an identity of the speaker associated with the voice sample based at least in part on the identity embedding.

### Claim 13 (depends on 11)

The system of claim 11, wherein determining the information describing the speaker further causes the system to perform:
determining that the speaker associated with the voice sample matches a known speaker based at least in part on the identity embedding.

### Claim 14 (depends on 11)

The system of claim 11, wherein the temporal convolutional network is trained based on a triplet loss function that evaluates a plurality of triplets, wherein a triplet includes an anchor voice sample, a positive voice sample that has a threshold level of similarity to the anchor voice sample, and a negative voice sample that does not have a threshold level of similarity to the anchor voice sample.

### Claim 15 (depends on 14)

The system of claim 14, wherein a triplet of voice samples is selected based on semi-hard triplet mining.

### Claim 16

A non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing system, cause the computing system to perform a method comprising:
providing audio waveform data that corresponds to a voice sample to a temporal convolutional network for evaluation, wherein the temporal convolutional network pre-processes the audio waveform data and outputs an identity embedding associated with the audio waveform data;
obtaining the identity embedding associated with the voice sample from the temporal convolutional network; and
determining information describing a speaker associated with the voice sample based at least in part on the identity embedding.

### Claim 17 (depends on 16)

The non-transitory computer-readable storage medium of claim 16, wherein determining the information describing the speaker further causes the computing system to perform:
determining an identity of the speaker associated with the voice sample based at least in part on the identity embedding.

### Claim 18 (depends on 16)

The non-transitory computer-readable storage medium of claim 16, wherein determining the information describing the speaker further causes the computing system to perform:
determining that the speaker associated with the voice sample matches a known speaker based at least in part on the identity embedding.

### Claim 19 (depends on 16)

The non-transitory computer-readable storage medium of claim 16, wherein the temporal convolutional network is trained based on a triplet loss function that evaluates a plurality of triplets, wherein a triplet includes an anchor voice sample, a positive voice sample that has a threshold level of similarity to the anchor voice sample, and a negative voice sample that does not have a threshold level of similarity to the anchor voice sample.

### Claim 20 (depends on 19)

The non-transitory computer-readable storage medium of claim 19, wherein a triplet of voice samples is selected based on semi-hard triplet mining.
