# US 12,217,549 B1 · System and method for matching a voice sample to a facial image based on voice and image metaproperties

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,217,549 B1
- Document ID: US12217549B1
- Status: Granted
- Kind: B1
- Application number: 18/632,397
- Filed: 2024-04-11
- Granted: 2025-02-04
- Earliest claimed: 2024-04-11
- Assignees: Corsound AI Ltd
- Applicants: Corsound AI Ltd
- Inventors: Nir Schwartz; Arkady Krishtul
- Web page: https://patentgrep.com/patent/US12217549B1

## CPC

- G06V 40/70: Multimodal biometrics, e.g. combining information from different biometric modalities
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/776: Validation; Performance evaluation
- G06V 40/172: {Classification, e.g. identification}
- G06V 40/179: {metadata assisted face recognition}
- G10L 17/02: Preprocessing operations, e.g. segment selection; Pattern representation or modelling, e.g. based on linear discriminant analysis [LDA] or principal components; Feature selection or extraction
- G10L 17/04: Training, enrolment or model building
- G10L 17/10: Multimodal systems, i.e. based on the integration of multiple recognition engines or fusion of expert systems
- G06V 2201/10: Recognition assisted with metadata
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

System and method for matching a voice sample to a facial image may include, using a processor: obtaining a voice sample and a facial image; calculating a plurality of voice metaproperties from the voice sample; calculating a plurality of image metaproperties from image sample, where each of the image metaproperties corresponds to one of the voice metaproperties; and determining a level of match between the voice sample and the facial image, based on the plurality of voice metaproperties and the plurality of image metaproperties.

## Claims

### Claim 1

A method for matching a voice sample to a facial image, the method comprising, using a processor:
obtaining a voice sample and a facial image;
calculating a plurality of voice metaproperties from the voice sample;
calculating a plurality of image metaproperties from the facial image, wherein each of the image metaproperties corresponds to one of the voice metaproperties, wherein each of the voice metaproperties and the image metaproperties comprises a probability distribution providing the probabilities that the voice metaproperty or the image metaproperty equals certain values of the metaproperty; and
determining a level of match between the voice sample and the facial image, based on the plurality of voice metaproperties and the plurality of image metaproperties, wherein determining whether the voice sample matches the facial image is performed by:
calculating a distance between each of the voice metaproperties and the corresponding image metaproperty;
calculating weights for a weighted sum operation by training a classifier and deriving the weights from the parameters of the classifier;
calculating the weighted sum of the distances; and
determining that the voice sample matches the facial image if the weighted sum satisfies a threshold condition, and that the voice sample does not match the facial image otherwise,
wherein the classifier is trained by:
obtaining a labelled dataset comprising a plurality of matching pairs labelled as matching pairs, and a plurality of unmatching pairs, labelled as unmatching pairs, wherein each of the matching pairs comprises a matching labelled voice sample and labelled facial image, and each of the unmatching pairs comprises an unmatching labelled voice sample and labelled facial image;
calculating, for each of the labelled voice samples, the plurality of voice metaproperties from the labelled voice sample;
calculating, for each of the labelled facial images, the plurality of image metaproperties from the labelled facial image; and
using the plurality of voice metaproperties and the plurality of image metaproperties of the plurality of matching pairs and the plurality of unmatching pairs, and the associated labels, to train the classifier.

### Claim 2 (depends on 1)

The method of claim 1, wherein the classifier is a binary classifier.

### Claim 3 (depends on 1)

The method of claim 1, wherein calculating one voice metaproperty of the plurality of voice metaproperties comprises:
encoding the voice sample into a voice representation using a voice encoder; and
calculating the one voice metaproperty of the plurality of voice metaproperties from the voice representation;
and wherein calculating one image metaproperty of the plurality of image metaproperties comprises:
encoding the facial image into an image representation using a face encoder; and
calculating one image metaproperty of the plurality of image metaproperties from the image representation.

### Claim 4 (depends on 1)

The method of claim 1, wherein each of the metaproperties comprises an identity characteristic of an individual.

### Claim 5 (depends on 1)

The method of claim 1, wherein the metaproperties comprise at least one property selected from: age, gender, weight, body mass index (BMI), skin tone, mother tongue and accent.

### Claim 6 (depends on 1)

The method of claim 1, comprising:
obtaining a plurality of facial images;
determining a level of match between the voice sample and each the facial images;
selecting the facial images with a highest level of match; and
reconstructing the face of the speaker in the voice sample by fusing the selected facial images.

### Claim 7

A system for matching a voice sample to a facial image, the system comprising:
a memory; and
a processor configured to:
obtain a voice sample and a facial image;
calculate a plurality of voice metaproperties from the voice sample;
calculate a plurality of image metaproperties from the facial image, wherein each of the image metaproperties corresponds to one of the voice metaproperties, wherein each of the voice metaproperties and the image metaproperties comprises a probability distribution providing the probabilities that the voice metaproperty or the image metaproperty equals certain values of the metaproperty; and
determine a level of match between the voice sample and the facial image, based on the plurality of voice metaproperties and the plurality of image metaproperties, by:
calculating a distance between each of the voice metaproperties and the corresponding image metaproperty;
calculating a weighted sum of the distances; and
determining that the voice sample matches the facial image if the weighted sum satisfies a threshold condition, and that the voice sample does not match the facial image otherwise,
wherein the processor is configured to calculate weights for the weighted sum operation by training a classifier and deriving the weights from the parameters of the classifier, and
wherein the processor is configured to train the classifier by:
obtaining a labelled dataset comprising a plurality of matching pairs labelled as matching pairs, and a plurality of unmatching pairs, labelled as unmatching pairs, wherein each of the matching pairs comprises a matching labelled voice sample and labelled facial image, and each of the unmatching pairs comprises an unmatching labelled voice sample and labelled facial image;
calculating, for each of the labelled voice samples, the plurality of voice metaproperties from the labelled voice sample;
calculating, for each of the labelled facial images, the plurality of image metaproperties from the labelled facial image; and
using the plurality of voice metaproperties and the plurality of image metaproperties of the plurality of matching pairs and the plurality of unmatching pairs, and the associated labels, to train the classifier.

### Claim 8 (depends on 7)

The system of claim 7, wherein the classifier is a binary classifier.

### Claim 9 (depends on 7)

The system of claim 7, wherein the processor is configured to calculate one voice metaproperty of the plurality of voice metaproperties by:
encoding the voice sample into a voice representation using a voice encoder; and
calculating the one voice metaproperty of the plurality of voice metaproperties from the voice representation;
and wherein calculating one image metaproperty of the plurality of image metaproperties comprises:
encoding the facial image into an image representation using a face encoder; and
calculating one image metaproperty of the plurality of image metaproperties from the image representation.

### Claim 10 (depends on 7)

The system of claim 7, wherein each of the metaproperties comprises an identity characteristic of an individual.

### Claim 11 (depends on 7)

The system of claim 7, wherein the metaproperties comprise at least one property selected from: age, gender, weight, body mass index (BMI), skin tone, mother tongue and accent.

### Claim 12 (depends on 7)

The system of claim 7, wherein the processor is configured to:
obtain a plurality of facial images;
determine a level of match between the voice sample and each the facial images;
selecting the facial images with a highest level of match; and
reconstruct the face of the speaker in the voice sample by fusing the selected facial images.