US 12,597,416 B1Granted

Multi-branched network for event detection

AssigneeAmazon Technologies, Inc.

InventorsMashhour Solh, Ameya Patil, Steven Sensarn

Life of this patent

Estimating the term…

Loading patent…

Abstract

A system that performs event detection using a multi-branched network for sensor fusion. For example, a device may detect when a tap event occurs on a surface of the device using a combination of microphone audio data and sensor data, such as motion data generated by a motion sensor. Prior to combining these inputs for further inference, the device may use separate neural networks to independently extract features from the audio data and the sensor data. This improves an accuracy of tap detection and enables detection of additional tap gestures and/or other types of event/activity detection, such as typing detection. The multi-branched network may generate fused data by processing audio features, motion data, raw audio data, raw accelerometer data, and/or additional sensor data. Depending on the inputs, a number of branches, a branch depth, and/or a number of event detectors may vary without departing from the disclosure.

Claims20

Dependent claims

A computer-implemented method, the method comprising:

receiving, from an accelerometer component of a device, linear acceleration data representing motion of the device;
receiving audio data corresponding to audio captured by at least one microphone of the device;
processing the audio data to determine first data, the first data representing average root-mean-squared (RMS) values and inter-channel level difference (ILD) values;
determining, using the linear acceleration data and at least a first convolutional layer of a machine learning model, first feature data corresponding to the motion of the device, the first feature data representing first values derived from the linear acceleration data;
determining, using the first data and at least a second convolutional layer of the machine learning model, second feature data corresponding to the audio, the second feature data representing second values derived from the first data;
generating third feature data by concatenating the first feature data and the second feature data;
determining, using the third feature data and at least a third convolutional layer of the machine learning model, fourth feature data;
detecting, using the fourth feature data, a first physical interaction with the device, the first physical interaction comprising at least one of a swipe, tap, or button press;
and
performing a first action in response to detecting the first physical interaction.

7 more elements

A computer-implemented method, the method comprising:

determining first data corresponding to a first sensor component of a device, the first data representing output of the first sensor component during a first time window;
determining second data corresponding to audio captured by at least one microphone of the device, the second data representing a portion of the audio captured during the first time window;
determining, using the first data and at least a first neural network of a machine learning model, first feature data;
determining, using the second data and at least a second neural network of the machine learning model, second feature data;
generating third feature data using the first feature data and the second feature data;
determining, using the third feature data and at least a third neural network of the machine learning model, fourth feature data;
detecting, using the fourth feature data, a first event corresponding to a first physical interaction with the device;
and
performing a first action in response to detecting the first event.

6 more elements

A system comprising:

at least one processor;
and
memory including instructions operable to be executed by the at least one processor to cause the system to:
determine first data corresponding to a first sensor component of a device, the first data representing output of the first sensor component during a first time window;
determine second data corresponding to audio captured by at least one microphone of the device, the second data representing a portion of the audio captured during the first time window;
determine, using the first data and at least a first neural network of a machine learning model, first feature data;
determine, using the second data and at least a second neural network of the machine learning model, second feature data;
generate third feature data using the first feature data and the second feature data;
determine, using the third feature data and at least a third neural network of the machine learning model, fourth feature data;
detect, using the fourth feature data, a first event corresponding to a first physical interaction with the device;
and
perform a first action in response to detecting the first event.

9 more elements

Dependencies are inferred from the claim text.