Back to results
US 12,694,708 B2Granted
Context-based speaker counter for a speaker diarization system
AssigneeGoogle Inc.
InventorsSourish Chaudhuri, Lev Finkelstein
Life of this patent
Estimating the term…
Abstract
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for determining the number of speakers in a video and a corresponding audio using visual context. In one aspect, a method includes detecting within the video multiple speakers, determining a bounding box for each detected speaker that includes the detected person and objects within a threshold distance of the detected person in an image frame, determining a unique descriptor for that person based in part on image information depicting the objects within the bounding box, determining a cardinality of unique speakers in the video, providing to the speaker diarization system the cardinality of unique speakers.
Claims16
A method performed by data processing apparatus, the method comprising:
- obtaining a video that includes a plurality of image frames and corresponding audio;
- detecting, within the video, multiple persons speaking depicted in the video without using biometric information to identify the persons speaking, the detection of each person based on the face of the person speaking detected in a face detection bounding box;
- in response to detecting the multiple persons speaking depicted in the video, determining, for each detected person, a bounding box that includes the detected person and objects within a threshold distance of the detected person in the image frame, the bounding box larger than the face detection bounding box and including the face detection bounding box;
- determining, for each detected person, from image information included within the bounding box for the detected person and without using biometric data that identifies the detected person, a unique descriptor for that person, the unique descriptor based in part on image information depicting the objects within the bounding box, each unique descriptor for the detected person not including biometric data that identifies the person, including determining a histogram of pixel activations for each of multiple frames, the histogram of pixel activations determined from local factors from the image information included within the bounding box;
- determining a cardinality of unique descriptors determined for the video, including:
- generating histogram clusters of the histograms, wherein each histogram cluster includes only histogram that are within a threshold distance of each other;
- and
- determining the cardinality of unique descriptors as the number of histogram clusters;
- providing, to a speaker diarization system that determines unique speakers of the corresponding audio for the video, at least the cardinality of unique descriptors.
6 more elements
A system, comprising:
- a data processing apparatus;
- and
- a non-transitory computer readable medium storing instructions executable by the data processing apparatus and that upon such execution cause the data processing apparatus to perform operations comprising:
- obtaining a video that includes a plurality of image frames and corresponding audio;
- detecting, within the video, multiple persons speaking depicted in the video without using biometric information to identify the persons speaking, the detection of each person based on face of the person speaking detected in a face detection bounding box;
- in response to detecting the multiple persons speaking depicted in the video, determining, for each detected person, a bounding box that includes the detected person and objects within a threshold distance of the detected person in the image frame, the bounding box larger than the face detection bounding box and including the face detection bounding box;
- determining, for each detected person, from image information included within the bounding box for the detected person and without using biometric data that identifies the detected person, a unique descriptor for that person, the unique descriptor based in part on image information depicting the objects within the bounding box, each unique descriptor for the detected person not including biometric data that identifies the person, including determining a histogram of pixel activations for each of multiple frames, the histogram of pixel activations determined from local factors from the image information included within the bounding box;
- determining a cardinality of unique descriptors determined for the video, including:
- generating histogram clusters of the histograms, wherein each histogram cluster includes only histogram that are within a threshold distance of each other;
- and
- determining the cardinality of unique descriptors as the number of histogram clusters;
- providing, to a speaker diarization system that determines unique speakers of the corresponding audio for the video, at least the cardinality of unique descriptors.
9 more elements
A non-transitory computer readable medium storing instructions executable by a data processing apparatus and that upon such execution cause the data processing apparatus to perform operations comprising:
- obtaining a video that includes a plurality of image frames and corresponding audio;
- obtaining a video that includes a plurality of image frames and corresponding audio;
- detecting, within the video, multiple persons speaking depicted in the video without using biometric information to identify the persons speaking, the detection of each person based on face of the person speaking detected in a face detection bounding box;
- in response to detecting the multiple persons speaking depicted in the video, determining, for each detected person, a bounding box that includes the detected person and objects within a threshold distance of the detected person in the image frame, the bounding box larger than the face detection bounding box and including the face detection bounding box;
- determining, for each detected person, from image information included within the bounding box for the detected person and without using biometric data that identifies the detected person, a unique descriptor for that person, the unique descriptor based in part on image information depicting the objects within the bounding box, each unique descriptor for the detected person not including biometric data that identifies the person, including determining a histogram of pixel activations for each of multiple frames, the histogram of pixel activations determined from local factors from the image information included within the bounding box;
- determining a cardinality of unique descriptors determined for the video, including:
- generating histogram clusters of the histograms, wherein each histogram cluster includes only histogram that are within a threshold distance of each other;
- and
- determining the cardinality of unique descriptors as the number of histogram clusters;
- providing, to a speaker diarization system that determines unique speakers of the corresponding audio for the video, at least the cardinality of unique descriptors.
7 more elements
Dependencies are inferred from the claim text.