# US 2021/0174140 A1 · Sensor triggered sound clip capturing for machine learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174140 A1
- Document ID: US20210174140A1
- Status: Published application
- Kind: A1
- Application number: 16/704,429
- Filed: 2019-12-05
- Published: 2021-06-10
- Earliest claimed: 2019-12-05
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Lisa Seacat DeLuca; David Kaminsky
- Web page: https://patentgrep.com/patent/US20210174140A1

## CPC

- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/2321: using statistics or function optimisation, e.g. modelling of probability density functions
- G06N 20/00: Machine learning
- G07C 5/00: Registering or indicating the working of vehicles
- G16Y 20/10: relating to the environment, e.g. temperature; relating to location
- G16Y 40/35: Management of things, i.e. controlling in accordance with a policy or in order to achieve specified objectives
- G06N 5/047: Pattern matching networks; Rete networks
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G07C: TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
- G16Y: INFORMATION AND COMMUNICATION TECHNOLOGY SPECIALLY ADAPTED FOR THE INTERNET OF THINGS [IoT]

## Abstract

A method for automatically training a machine learning system to detect and identify a sensor triggering event associated with an internet of things (IoT) device is provided. The method may include capturing sensor data and capturing sound clips associated with the IoT device. The method may further include identifying the sensor triggering event associated with the IoT device. The method may further include sending an alert of the identified sensor triggering event. The method may also include correlating the captured sensor data, the captured sound clips, and the identified sensor triggering event. The method may further include identifying a second sensor triggering event by determining similarities between the correlated data associated with the identified sensor triggering event and additional sensor and sound data that is captured based on the second sensor triggering event.

## Claims

### Claim 1

A method for automatically training a machine learning system to detect and identify a sensor triggering event associated with an internet of things (IoT) device, the method comprising:
in response to a first sensor triggering event, using one or more sensors associated with the sensor analysis unit to capture sensor data and using an acoustic detection device to capture one or more sound clips associated with the IoT device;
based on the captured sensor data and the captured one or more sound clips, identifying the first sensor triggering event associated with the IoT device; and
correlating and storing the captured sensor data, the captured one or more sound clips, and the first identified sensor triggering event and providing the correlated and stored data as training data to one or more computer machine learning systems.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
receiving, by the IoT device, one or more rules to control event collection by the sensor analysis unit and the acoustic detection device that is associated with the IoT device, wherein event collection includes capturing the sensor data and the one or more sound clips associated with the IoT device in response to the first sensor triggering event, and wherein the one or more rules to control event collection by the sensor analysis unit and the acoustic detection device comprise a first set of rules to determine when and how long to collect sensor data associated with the one or more sensors and to collect sound associated with the acoustic detection device in response to determining a type of the first sensor triggering event.

### Claim 3 (depends on 3)

The method of claim 3, wherein using the one or more sensors associated with the sensor analysis unit to capture the sensor data further comprises:
capturing the sensor data based on a second set of rules that determines whether to capture the sensor data from a first set sensors that include the one or more sensors that detected the first sensor triggering event and capturing the sensor data from a second set of different sensors.

### Claim 4 (depends on 1)

The method of claim 1, wherein identifying the first sensor triggering event further comprises:
transmitting the captured sensor data and the captured one or more sound clips to the one or more computer machine learning system according to a third set of rules based on a determination of whether the first sensor triggering event is normal or abnormal;
determining similarities between the captured sensor data, stored sensor data, and online sensor data;
determining similarities between the captured one or more sound clips, stored one or more sound clips, and online one or more sound clips;
identifying the first sensor triggering event by determining that the captured sensor data and the captured one or more sound clips have a degree of similarity to stored or online data associated with a previously identified event, and identifying the previously identified event as the first sensor triggering event; and
sending an alert comprising a notification of the first identified sensor triggering event to one or more computing devices associated with a user of the IoT device.

### Claim 5 (depends on 1)

The method of claim 1, wherein identifying the first sensor triggering event further comprises:
presenting the identified first sensor triggering event with a confidence score.

### Claim 6 (depends on 5)

The method of claim 5, wherein the alert comprises the captured one or more sound clips, the captured sensor data, and the confidence score associated with the first identified sensor triggering event.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
in response to detecting a second sensor triggering event, identifying the second sensor triggering event by determining similarities between the correlated and stored data associated with the first identified sensor triggering event and additional sensor and sound data that is captured based on the second sensor triggering event.

### Claim 8

A computer system for automatically training a machine learning system to detect and identify a sensor triggering event associated with an internet of things (IoT) device, comprising:
one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage devices, and program instructions stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, wherein the computer system is capable of performing a method comprising:
in response to a first sensor triggering event, using one or more sensors associated with a sensor analysis unit to capture sensor data and using an acoustic detection device to capture one or more sound clips associated with the IoT device;
based on the captured sensor data and the captured one or more sound clips, identifying the first sensor triggering event associated with the IoT device; and
correlating and storing the captured sensor data, the captured one or more sound clips, and the first identified sensor triggering event and providing the correlated and stored data as training data to one or more computer machine learning systems.

### Claim 9 (depends on 8)

The computer system of claim 8, further comprising:
receiving, by the IoT device, one or more rules to control event collection by the sensor analysis unit and the acoustic detection device that is associated with the IoT device, wherein event collection includes capturing the sensor data and the one or more sound clips associated with the IoT device in response to the first sensor triggering event, and wherein the one or more rules to control event collection by the sensor analysis unit and the acoustic detection device comprise a first set of rules to determine when and how long to collect sensor data associated with the one or more sensors and to collect sound associated with the acoustic detection device in response to determining a type of the first sensor triggering event.

### Claim 10 (depends on 9)

The computer system of claim 9, wherein using the one or more sensors associated with the sensor analysis unit to capture the sensor data further comprises:
capturing the sensor data based on a second set of rules that determines whether to capture the sensor data from a first set sensors that include the one or more sensors that detected the first sensor triggering event and capturing the sensor data from a second set of different sensors.

### Claim 11 (depends on 8)

The computer system of claim 8, wherein identifying the first sensor triggering event further comprises:
transmitting the captured sensor data and the captured one or more sound clips to the one or more computer machine learning systems according to a third set of rules based on a determination of whether the first sensor triggering event is normal or abnormal;
determining similarities between the captured sensor data, stored sensor data, and online sensor data;
determining similarities between the captured one or more sound clips, stored one or more sound clips, and online one or more sound clips;
identifying the first sensor triggering event by determining that the captured sensor data and the captured one or more sound clips have a degree of similarity to stored or online data associated with a previously identified event, and identifying the previously identified event as the first sensor triggering event; and
sending an alert comprising a notification of the first identified sensor triggering event to one or more computing devices associated with a user of the IoT device.

### Claim 12 (depends on 8)

The computer system of claim 8, wherein identifying the first sensor triggering event further comprises:
presenting the identified first sensor triggering event with a confidence score.

### Claim 13 (depends on 12)

The computer system of claim 12, wherein the alert comprises the captured one or more sound clips, the captured sensor data, and the confidence score associated with the first identified sensor triggering event.

### Claim 14 (depends on 8)

The computer system of claim 8, further comprising:
in response to detecting a second sensor triggering event, identifying the second sensor triggering event by determining similarities between the correlated and stored data associated with the first identified sensor triggering event and additional sensor and sound data that is captured based on the second sensor triggering event.

### Claim 15

A computer program product for automatically training a machine learning system to detect and identify a sensor triggering event associated with an internet of things (IoT) device, comprising:
one or more tangible computer-readable storage devices and program instructions stored on at least one of the one or more tangible computer-readable storage devices, the program instructions executable by a processor, the program instructions comprising:
program instructions to, in response to a first sensor triggering event, use one or more sensors associated with the sensor analysis unit to capture sensor data and use an acoustic detection device to capture one or more sound clips associated with the IoT device;
program instructions to, based on the captured sensor data and the captured one or more sound clips, identify the first sensor triggering event associated with the IoT device; and
program instructions to correlate and store the captured sensor data, the captured one or more sound clips, and the first identified sensor triggering event and providing the correlated and stored data as training data to one or more computer machine learning systems.

### Claim 16 (depends on 15)

The computer program product of claim 15, further comprising:
program instructions to receive, by the IoT device, one or more rules to control event collection by the sensor analysis unit and the acoustic detection device that is associated with the IoT device, wherein event collection includes capturing the sensor data and the one or more sound clips associated with the IoT device in response to the first sensor triggering event, and wherein the one or more rules to control event collection by the sensor analysis unit and the acoustic detection device comprise a first set of rules to determine when and how long to collect sensor data associated with the one or more sensors and to collect sound associated with the acoustic detection device in response to determining a type of the first sensor triggering event.

### Claim 17 (depends on 16)

The computer program product of claim 16, wherein the program instructions to use the one or more sensors associated with the sensor analysis unit to capture the sensor data further comprises:
program instructions to capture the sensor data based on a second set of rules that determines whether to capture the sensor data from a first set sensors that include the one or more sensors that detected the first sensor triggering event and capture the sensor data from a second set of different sensors.

### Claim 18 (depends on 15)

The computer program product of claim 15, wherein the program instructions to identify the first sensor triggering event further comprises:
program instructions to transmit the captured sensor data and the captured one or more sound clips to the one or more computer machine learning systems according to a third set of rules based on a determination of whether the first sensor triggering event is normal or abnormal;
program instructions to determine similarities between the captured sensor data, stored sensor data, and online sensor data; and
program instructions to determine similarities between the captured one or more sound clips, stored one or more sound clips, and online one or more sound clips; and
program instructions to identify the first sensor triggering event by determining that the captured sensor data and the captured one or more sound clips have a degree of similarity to stored or online data associated with a previously identified event, and identifying the previously identified event as the first sensor triggering event; and
program instructions to send an alert comprising a notification of the first identified sensor triggering event to one or more computing devices associated with a user of the IoT device.

### Claim 19 (depends on 15)

The computer program product of claim 15, wherein the program instructions to identify the first sensor triggering event comprises:
program instructions to present the identified first sensor triggering event with a confidence score.

### Claim 20 (depends on 19)

The method of claim 19, wherein the alert comprises the captured one or more sound clips, the captured sensor data, and the confidence score associated with the first identified sensor triggering event.
