Back to results
US 12,592,237 B2Granted
Driver interface with voice and image control
AssigneeSoundHound AI IP, LLC
InventorsZili Li, Cristina Vasconcelos
Life of this patent
Estimating the term…
Abstract
A driver interface for use within an automobile provides responses to voice commands issued for example by a driver of the automobile. The interface includes a camera and microphone for capturing image data such as gestures and audio data from the automobile driver. The image data and audio data are processed to extract image and linguistic features from the image and audio data, which image and linguistic features are processed to interpret and infer a meaning of the voice command.
Claims19
A driver interface system comprising:
- a camera interface enabled to capture images of a driver and an environment surrounding the driver;
- an image processor enabled to extract, from captured images of the driver and an environment surrounding the driver, visual features, the visual features being an abstraction of the captured images, the abstraction being a compressed representation of the captured images;
- a microphone interface enabled to capture speech;
- a speech recognizer enabled to extract, from the captured speech, linguistic features;
- a natural language processor enabled to infer a voice command from the linguistic features and the visual features;
- and
- an output interface for rendering a response to the voice command;
- and
- wherein the speech recognizer uses a machine-learned linguistic model based on a combination of audio feature tensors and visual feature tensors.
6 more elements
A driver interface system comprising:
- a camera interface enabled to capture images of a driver and an environment surrounding the driver;
- a neural network image processor enabled to extract, from captured images of the driver and an environment surrounding the driver, visual features, the visual features being an abstraction of the captured images determined by the neural network image processor, the abstraction being a compressed representation of the captured images with reduced information content as compared to the captured images;
- a microphone interface enabled to capture speech;
- a speech recognizer enabled to extract, from the captured speech, linguistic features;
- and
- a natural language processor enabled to infer a voice command from the linguistic features and the visual features, including visual features absracted from an environment surrounding the driver;
- wherein the image processor, the speech recognizer, and the natural language processor are jointly configured by joint training, such that the extraction of the visual feature tensors and audio feature tensors is optimized in combination with the natural language processor to minimize errors in inferring the voice command.
4 more elements
A driver interface system comprising:
- a camera interface enabled to capture images of a driver and an environment surrounding the driver;
- an image processor enabled to extract, from captured images of the driver and an environment surrounding the driver, visual feature tensors, the visual feature tensors being an abstraction of the captured images, the abstraction being a compressed representation of the captured images with reduced information content as compared to the captured images;
- a microphone interface enabled to capture speech;
- a speech recognizer enabled to extract, from the captured speech, audio feature tensors;
- a natural language processor enabled to concatonate or merge the visual feature vectors and audio feature vectors to infer a voice command;
- wherein the image processor, the speech recognizer, and the natural language processor are jointly configured by joint training, such that the extraction of the visual feature tensors and audio feature tensors is optimized in combination with the natural language processor to minimize errors in inferring the voice command;
- and
- an output interface for rendering a response to the voice command.
5 more elements
Dependencies are inferred from the claim text.