# US 11,443,741 B2 · System and method for providing portable natural language processing interface across multiple appliances

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,443,741 B2
- Document ID: US11443741B2
- Status: Granted
- Kind: B2
- Application number: 16/844,979
- Filed: 2020-04-09
- Published: 2020-07-23
- Granted: 2022-09-13
- Earliest claimed: 2018-10-15
- Assignees: Midea Group Co., Ltd.
- Applicants: Midea Group Co., Ltd.
- Inventors: Haibin Huang; Chen Zhang; Xin Liu
- Web page: https://patentgrep.com/patent/US11443741B2

## CPC

- G06F 40/30: Semantic analysis
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 40/216: using statistical methods
- G10L 15/063: {Training}
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
- G10L 2015/0635: {updating or merging of old and new templates; Mean values; Weighting}
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L 2015/228: {of application context}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A natural language processing (NLP) apparatus includes a housing, a built-in voice input interface; a built-in data communication interface configured to establish data communication with multiple types of appliances; a built-in NLP module, and a built-in control device. A first voice input is received through the built-in voice input interface; if the target appliance is a first appliance of a first appliance type, the first voice input is processed using a first NLP model of the built-in NLP module to obtain a first machine command, and the first machine command is sent via the built-in data communication interface to the first appliance; and if the target appliance is a second appliance of a second appliance type, the first voice input is processed using a second NLP model of the built-in NLP module, and the second machine command is sent via the built-in data communication interface to the second appliance.

## Claims

### Claim 1

A natural language processing (NLP) apparatus, comprising:
a housing;
a built-in voice input interface configured to accept voice-based inputs from a user;
a built-in data communication interface configured to establish data communication with multiple types of appliances configured to respond to distinct sets of machine commands to perform their respective appliance operations;
a built-in NLP module, wherein the NLP module includes a respective NLP model for each of the multiple types of appliances;
a built-in control device configured to detect an activation input for activating the built-in voice input interface and a target selection input for selecting a target appliance for a voice input received after activating the built-in voice input interface;
one or more processors; and
memory storing instructions, the instructions, when executed by the one or more processors, cause the processors to perform operations comprising:
while the built-in voice input interface remains activated and a current selection of the target appliance remains unchanged, receiving a first voice input through the built-in voice input interface; and
in response to receiving the first voice input:
in accordance with a determination that the target appliance is a first appliance of a first appliance type, processing the first voice input using a first NLP model of the built-in NLP module that corresponds to the first appliance type to obtain a first machine command, and sending, via the built-in data communication interface, the first machine command to the first appliance; and
in accordance with a determination that the target appliance is a second appliance of a second appliance type that is distinct from the first appliance type, processing the first voice input using a second NLP model of the built-in NLP module that corresponds to the second appliance type to obtain a second machine command, and sending, via the built-in data communication interface, the second machine command to the second appliance.

### Claim 2 (depends on 1)

The NLP apparatus of claim 1, including:
a coupling interface that is configured to physically connect the NLP apparatus with one or more of the multiple types of appliances, wherein the built-in control device detects the activation input for activating the built-in voice input interface when the coupling interface is physically connected to an appliance of a respective type of the multiple types of appliances.

### Claim 3 (depends on 1)

The NLP apparatus of claim 1, including:
a coupling interface that is configured to physically connect the NLP apparatus with one or more of the multiple types of appliances, wherein, when the coupling interface is physically connected to an appliance of a respective type of the multiple types of appliances, the built-in control device selects the appliance of the respective type of the multiple types of appliances as the target appliance for the voice input received after activating the built-in voice input interface.

### Claim 4 (depends on 1)

The NLP apparatus of claim 1, including:
a plurality of touch-activated portions on the housing of the NLP apparatus, wherein the built-in control device detects the activation input for activating the built-in voice input interface when at least one of the plurality of touch-activated portions on the housing of the NLP apparatus is activated by a touch input, and the built-in control device selects the target appliance in accordance with determining that said one of the plurality of touch-activated portions on the housing of the NLP apparatus is preassigned to be associated with the target appliance.

### Claim 5 (depends on 1)

The NLP apparatus of claim 1, including:
an orientation sensor that is configured to determine a current orientational state of the NLP apparatus in a physical environment, wherein the built-in control device selects the target appliance in accordance with determining that the current orientational state of the NLP apparatus is preassigned to be associated with the target appliance.

### Claim 6 (depends on 1)

The NLP apparatus of claim 1, wherein:
the built-in control device is configured to detect the activation input for activating the built-in voice input interface when detecting that the NLP apparatus has been brought within a threshold distance of an appliance of a respective one of the multiple types of appliances.

### Claim 7 (depends on 6)

The NLP apparatus of claim 6, wherein:
the built-in control device selects the target appliance in accordance with detecting that the NLP apparatus is within the threshold distance of the target appliance.

### Claim 8 (depends on 1)

The NLP apparatus of claim 1, wherein the operations include:
activating the built-in data communication interface to establish data communication with the target appliance; and
receiving a message including an appliance type identifier identifying a respective appliance type of the target appliance via the activated built-in data communication interface.

### Claim 9 (depends on 1)

The NLP apparatus of claim 1, wherein:
before detecting the activation input, the built-in voice input interface is inactive.

### Claim 10 (depends on 1)

The NLP apparatus of claim 1, wherein:
the built-in voice input interface is kept activated for a predetermined period of time after the built-in voice input interface is activated in response to the activation input.

### Claim 11 (depends on 1)

The NLP apparatus of claim 1, wherein:
the built-in data communication interface corresponds to a wireless communication protocol selected from the group consisting of: Bluetooth, Wireless Fidelity (Wi-Fi), near-field communication (NFC), ZigBee, ultrasound, acoustics, and magnetic resonance.

### Claim 12 (depends on 1)

The NLP apparatus of claim 1, wherein the operations further include:
prior to receiving the first voice input, obtaining and installing a plurality of NLP models in the built-in NLP module in accordance with user selections of a plurality of sets of appliance functions for a plurality of appliance types respectively, wherein each NLP model corresponds to a respective set of appliance functions for a respective appliance type.

### Claim 13 (depends on 1)

The NLP apparatus of claim 1, wherein the operations further include:
after selecting the target appliance, establishing a first wireless data communication with the target appliance.

### Claim 14 (depends on 1)

The NLP apparatus of claim 1, including:
a rechargeable battery enclosed in the housing, wherein the battery is rechargeable by the target appliance via the built-in data communication interface.

### Claim 15 (depends on 14)

The NLP apparatus of claim 14, wherein the operations include:
in response to starting to charge the rechargeable battery, searching for a user device by broadcasting data communication request messages, wherein the user device is configured to communicate with an NLP model server that trains a plurality of NLP models including the first NLP model and the second NLP model.

### Claim 16 (depends on 1)

The NLP apparatus of claim 1, wherein the operations include:
collecting voice data from one or more voice inputs of the user associated with managing one or more appliances respectively; and
in accordance with a determination that the NLP apparatus has established a data communication connection with a user device that is capable of communicating with an NLP model server configured to train a plurality of NLP models including the first NLP model and the second NLP model, sending the collected voice data to the user device for forwarding to the NLP model server to update one or more NLP models corresponding to the one or more appliances of one or more appliance types respectively.

### Claim 17

A non-transitory computer-readable storage medium storing instructions, the instructions, when executed by one or more processors of a natural language processing (NLP) apparatus, cause the processors to perform operations, comprising:
an NLP apparatus, comprising:
a housing;
a built-in voice input interface configured to accept voice-based inputs from a user;
a built-in data communication interface configured to establish data communication with multiple types of appliances configured to respond to distinct sets of machine commands to perform their respective appliance operations;
a built-in NLP module, wherein the NLP module includes a respective NLP model for each of the multiple types of appliances; and
a built-in control module configured to detect an activation input for activating the built-in voice input interface and a target selection input for selecting a target appliance for a voice input received after activating the built-in voice input interface:
while the built-in voice input interface remains activated and a current selection of the target appliance remains unchanged, receiving a first voice input through the built-in voice input interface; and
in response to receiving the first voice input:
in accordance with a determination that the target appliance is a first appliance of a first appliance type, processing the first voice input using a first NLP model of the built-in NLP module that corresponds to the first appliance type to obtain a first machine command, and sending, via the built-in data communication interface, the first machine command to the first appliance; and
in accordance with a determination that the target appliance is a second appliance of a second appliance type that is distinct from the first appliance type, processing the first voice input using a second NLP model of the built-in NLP module that corresponds to the second appliance type to obtain a second machine command, and sending, via the built-in data communication interface, the second machine command to the second appliance.

### Claim 18 (depends on 17)

The non-transitory computer-readable storage medium of claim 17, wherein the operations include:
activating the built-in data communication interface to establish data communication with the target appliance; and
receiving a message including an appliance type identifier identifying a respective appliance type of the target appliance via the activated built-in data communication interface.

### Claim 19 (depends on 17)

The non-transitory computer-readable storage medium of claim 17, wherein the operations further include:
prior to receiving the first voice input, obtaining and installing a plurality of NLP models in the built-in NLP module in accordance with user selections of a plurality of sets of appliance functions for a plurality of appliance types respectively, wherein each NLP model corresponds to a respective set of appliance functions for a respective appliance type.

### Claim 20 (depends on 17)

The non-transitory computer-readable storage medium of claim 17, wherein the operations include:
collecting voice data from one or more voice inputs of the user associated with managing one or more appliances respectively; and
in accordance with a determination that the NLP apparatus has established a data communication connection with a user device that is capable of communicating with an NLP model server configured to train a plurality of NLP models including the first NLP model and the second NLP model, sending the collected voice data to the user device for forwarding to the NLP model server to update one or more NLP models corresponding to the one or more appliances of one or more appliance types respectively.