# US 11,443,758 B2 · Anomalous sound detection with timbre separation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,443,758 B2
- Document ID: US11443758B2
- Status: Granted
- Kind: B2
- Application number: 17/171,259
- Filed: 2021-02-09
- Published: 2022-08-11
- Granted: 2022-09-13
- Earliest claimed: 2021-02-09
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Shu Morikuni; Michiaki Tatsubori; Phongtharin Vinayavekhin; Ryuki Tachibana; Tadanobu Inoue
- Web page: https://patentgrep.com/patent/US11443758B2

## CPC

- G05B 23/0275: {Fault isolation and identification, e.g. classify fault; estimate cause or root of failure}
- G10L 25/51: for comparison or discrimination
- G01M 99/005: {Testing of complete machines, e.g. washing-machines or mobile phones (testing of machine parts G01M13/00; testing of electric apparatus or components G01R31/50)}
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/09: Supervised learning
- G10L 25/30: using neural networks
- G05B 2219/37337: Noise, acoustic emission, sound
- G06N 3/048: Activation functions
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G01M: TESTING STATIC OR DYNAMIC BALANCE OF MACHINES OR STRUCTURES; TESTING OF STRUCTURES OR APPARATUS, NOT OTHERWISE PROVIDED FOR
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods, systems, and computer program products for detecting anomalous behavior include reconstructing a waveform of a target device from an input waveform using a target autoencoder. A waveform of unrelated sound events is reconstructed from the input waveform using an environmental autoencoder. The input waveform is classified to determine that the input waveform is produced by anomalous behavior of the target device using a classifier, based on the reconstructed waveform of the target device and the reconstructed waveform of the unrelated sound events. An automatic response to the anomalous behavior is generated.

## Claims

### Claim 1

A computer program product for detecting anomalous behavior, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions being executable by a hardware processor to cause the hardware processor to:
reconstruct a waveform of a target device from an input waveform using a target autoencoder;
reconstruct a waveform of unrelated sound events from the input waveform using an environmental autoencoder;
classify the input waveform to determine that the input waveform is produced by anomalous behavior of the target device using a classifier, based on the reconstructed waveform of the target device and the reconstructed waveform of the unrelated sound events; and
generate an automatic response to the anomalous behavior.

### Claim 2 (depends on 1)

The computer program product of claim 1, wherein the program instructions further cause the hardware processor to concatenate a current reconstructed waveform of the target device, a current reconstructed waveform of the unrelated sound events, and a current original waveform that is used to generate the current reconstructed waveforms to form a concatenated signal.

### Claim 3 (depends on 2)

The computer program product of claim 2, wherein the classifier accepts the concatenated signal as an input.

### Claim 4 (depends on 1)

The computer program product of claim 1, wherein the program instructions further cause the hardware processor to generate the input waveform using input from a sensor device in an environment that includes the target device.

### Claim 5 (depends on 1)

The computer program product of claim 1, wherein the first environmental autoencoder is trained using sample waveforms of multiple different sounds, unrelated to operation of the target device.

### Claim 6 (depends on 1)

The computer program product of claim 1, wherein the program instructions further cause the hardware processor to reconstruct a second waveform of unrelated sound events from the input waveform using at least one additional environmental autoencoder.

### Claim 7 (depends on 6)

The computer program product of claim 6, wherein the program instructions further cause the hardware processor to combine a subset of the input waveform, the reconstructed waveform output of the target autoencoder, and reconstructed waveform outputs of the first environmental autoencoder and the at least one additional environmental autoencoder, to form a combined signal as input to the classifier.

### Claim 8 (depends on 1)

The computer program product of claim 1, wherein the automatic response includes an action selected from the group consisting of changing settings on the target device, changing qualities of the environment, and turning off the target device or other nearby equipment.

### Claim 9

A computer-implemented method for training an anomalous behavior classifier, comprising:
training a target autoencoder, using training data that represents normal operation of a target device, to generate a reconstructed target device waveform from a mixed input waveform;
training a first environmental autoencoder, using training data that represents a first set of unrelated sound events, to generate a reconstructed unrelated waveform from a mixed input waveform; and
training a neural network classifier, using reconstructed target device waveforms and unrelated waveforms based on the training data, to generate an output that classifies whether an input waveform represents anomalous behavior of the target device.

### Claim 10 (depends on 9)

The method of claim 9, wherein training the neural network classifier includes concatenating a reconstructed training waveform from the target autoencoder, a reconstructed environmental waveform of the first environmental autoencoder, and an original training waveform for use as a training input to the classifier.

### Claim 11 (depends on 9)

The method of claim 9, wherein the first environmental autoencoder is trained using sample waveforms of multiple different sounds, unrelated to operation of the target device.

### Claim 12 (depends on 9)

The method of claim 9, further comprising training at least one additional environmental autoencoder, using training data that represents respective sets of unrelated sound events, to generate a reconstructed unrelated waveform from a mixed input waveform.

### Claim 13

A system for detecting anomalous behavior, comprising:
a hardware processor;
a memory that stores computer program instructions, the computer program instructions executable by the hardware processor to cause the hardware processor to:
reconstruct a waveform of a target device from an input waveform using a target autoencoder;
reconstruct a waveform of unrelated sound events from the input waveform using an environmental autoencoder;
classify the input waveform to determine that the input waveform is produced by anomalous behavior of the target device using a classifier, based on the reconstructed waveform of the target device and the reconstructed waveform of the unrelated sound events; and
generate an automatic response to the anomalous behavior.

### Claim 14 (depends on 13)

The system of claim 13, wherein the computer program instructions are further executable by the hardware processor to cause the hardware processor to concatenate an output waveform of the target autoencoder current, an output waveform of the first autoencoder, and the input waveform.

### Claim 15 (depends on 14)

The system of claim 14, wherein the classifier classifies the concatenated waveforms.

### Claim 16 (depends on 13)

The system of claim 13, further comprising a sensor that generates the input waveform in an environment that includes the target device.

### Claim 17 (depends on 13)

The system of claim 13, wherein the first environmental autoencoder is trained using sample waveforms of multiple different sounds, unrelated to operation of the target device.

### Claim 18 (depends on 13)

The system of claim 13, wherein the computer program instructions are further executable by the hardware processor to cause the hardware processor to reconstruct waveforms of unrelated sound events using an additional environmental autoencoder.

### Claim 19 (depends on 18)

The system of claim 18, wherein the computer program instructions are further executable by the hardware processor to cause the hardware processor to combine a subset of the input waveform, a reconstructed waveform output of the target autoencoder, and reconstructed waveform outputs of the first environmental autoencoder and the at least one additional environmental autoencoder.

### Claim 20 (depends on 13)

The system of claim 13, wherein the computer program instructions are further executable by the hardware processor to cause the hardware processor to perform an action selected from the group consisting of changing settings on the target device, changing qualities of the environment, and turning off the target device or other nearby equipment.