Back to results
US 12,387,728 B2Granted
Utterance section detection device, utterance section detection method, and storage medium
AssigneePanasonic Intellectual Property Management Co., Ltd.
InventorYoshifumi Hirose
Life of this patent
Estimating the term…
Abstract
Provided is an utterance section detection device including: a first lip shape estimation module configured to estimate a first lip shape of an utterer, based on sound data including a voice of the utterer, a second lip shape estimation module configured to estimate a second lip shape of the utterer; based on image data in which an image of at least a face of the utterer is photographed; and an utterance section detection module configured to detect an utterance section in which the utterer is vocalizing in the sound data, based on changes in the first lip shape and changes in the second lip shape.
Claims18
An utterance section detection device, comprising:
- a memory having one or more instructions stored thereon;
- and
- a processor, wherein the processor executes the one or more instructions to cause the utterance section detection device to:
- acquire sound data from a microphone device, wherein the sound data is a voice of an utterer;
- estimate a first lip shape of the utterer based on the sound data;
- acquire image data from a camera device, wherein the image data is an image of at least a face of the utterer photographed by the camera device;
- estimate a second lip shape of the utterer based on the image data;
- estimate a vocal tract shape of the utterer from the sound data using a prediction model as a vocal tract sound source model that predicts a sampled value of a voice waveform of the sound data;
- calculate, based on the estimated vocal tract shape, a first opening degree indicating a first degree of opening between lips as a first parameter digitizing the estimated first lip shape;
- extract a lip region of the utterer from the image data, and calculate, based on the extracted lip region, a second opening degree indicating a second degree of opening between lips as a second parameter digitizing the second lip shape;
- calculate a correlation value indicating how much the first parameter and the second parameter are correlated;
- detect as an utterance section a section of the sound data in which the utterer is vocalizing in the sound data from changes in the estimated first lip shape and changes in the estimated second lip shape, wherein the section is one in which the calculated correlation value is larger than a predetermined threshold value;
- create voice data by trimming a segment that is the utterance section out of the sound data;
- and
- output the created voice data to a display device connected to the utterance section detection device.
12 more elements
An utterance section detection method of an utterance section detection device that comprises a memory storing one or more instructions thereon and a processor that executes the one or more instructions for detecting, in sound data that is a voice of an utterer, an utterance section in which the utterer is vocalizing, the utterance section detection method comprising:
- acquiring, via a microphone device, the sound data, wherein the sound data is a voice of the utterer;
- acquiring, from a camera device, image data in which an image of at least a face of the utterer is photographed by the camera device;
- estimating changes in a first lip shape of the utterer based on the sound data;
- estimating changes in a second lip shape of the utterer based on the image data;
- and
- estimating a vocal tract shape of the utterer from the sound data using a prediction model as a vocal tract sound source model that predicts a sampled value of a voice waveform of the sound data;
- calculating, based on the estimated vocal tract shape, a first opening degree indicating a first degree of opening between lips as a first parameter digitizing the estimated first lip shape;
- extracting a lip region of the utterer from the image data, and calculate, based on the extracted lip region, a second opening degree indicating a second degree of opening between lips as a second parameter digitizing the second lip shape;
- calculating a correlation value indicating how much the first parameter and the second parameter are correlated;
- detecting as the utterance section a section of the sound data from the estimated changes in the first lip shape and the estimated changes in the second lip shape, wherein the section is one in which the calculated correlation value is larger than a predetermined threshold value;
- outputting to a display device connected to the utterance section detection device the utterance section, wherein the utterance section is displayed by the display device.
8 more elements
An utterance section detection device, comprising:
- a memory having one or more instructions stored thereon;
- and
- a processor, wherein the processor executes the one or more instructions to cause the utterance section detection device to:
- acquire sound data from a microphone device, wherein the sound data is a voice of an utterer;
- estimate a first lip shape of the utterer based on the sound data;
- acquire image data from a camera device, wherein the image data is an image of at least a face of the utterer photographed by the camera device;
- estimate a second lip shape of the utterer based on the image data;
- estimate a vocal tract shape of the utterer from the sound data using a prediction model as a vocal tract sound source model that predicts a sampled value of a voice waveform of the sound data;
- calculate, based on the estimated vocal tract shape, a first opening degree indicating a first degree of opening between lips as a first parameter digitizing the estimated first lip shape;
- extract a lip region of the utterer from the image data, and calculate, based on the extracted lip region, a second opening degree indicating a second degree of opening between lips as a second parameter digitizing the second lip shape;
- calculate a correlation value indicating how much the first parameter and the second parameter are correlated;
- detect as an utterance section a section of the sound data in which the utterer is vocalizing in the sound data from changes in the estimated first lip shape and changes in the estimated second lip shape, wherein the section is one in which the calculated correlation value is larger than a predetermined threshold value;
- and
- output to a display device connected to the utterance section detection device the utterance section, wherein the utterance section is displayed by the display device.
11 more elements
Dependencies are inferred from the claim text.