# US 12,223,282 B2 · Intelligent automated assistant in a home environment > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,223,282 B2 - Document ID: US12223282B2 - Status: Granted - Kind: B2 - Application number: 17/497,797 - Filed: 2021-10-08 - Published: 2022-02-10 - Granted: 2025-02-11 - Earliest claimed: 2016-06-09 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Garett R. Nell; Roshni Malani; Sean P. Brown; Benjamin L Brumbaugh - Web page: https://patentgrep.com/patent/US12223282B2 ## CPC - G06F 40/35: Discourse or dialogue representation - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G10L 15/1822: {Parsing for meaning understanding} - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - H04L 12/282: {based on user interaction within the home (receiver circuitry for displaying additional information being controlled by a remote control apparatus H04N21/42204)} - H04L 12/2823: {Reporting information sensed by appliance or service execution status of appliance services in a home automation network (device-related reporting H04L43/065; arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station, in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom H04Q9/00)} - G06F 40/284: Lexical analysis, e.g. tokenisation or collocates - G10L 2015/223: {Execution procedure of a spoken command} - G10L 2015/226: {using non-speech characteristics} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract Systems and processes for operating an intelligent automated assistant are provided. In one example process, discourse input representing a user request is received. The process determines whether the discourse input relates to a device of an established location. In response to determining that the discourse input relates to a device of an established location, a data structure representing a set of devices of the established location is retrieved. The process determines, using the data structure, a user intent corresponding to the discourse input, the user intent associated with an action to be performed by a device of the set of devices, and a criterion to be satisfied prior to performing the action. The action and the device are stored in association with the criterion, where, in accordance with a determination that the criterion is satisfied, the action is performed by the device. ## Claims ### Claim 1 An electronic device, comprising: one or more processors; and memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: receiving a user request defining a request for a first device of an established location to perform an action and a condition that is required to be satisfied prior to performing the action, wherein the condition is defined using a relative term in the user request; determining, based on the user request and the relative term, an actionable intent of performing the action with the first device of the established location when an actual value of a characteristic of a second device of the established location satisfies the condition, wherein determining the actionable intent includes: responsive to receiving the user request, retrieving a historical distribution of values for the characteristic of the second device of the established location, determining, based on the historical distribution of values for the characteristic of the second device of the established location, a quantitative threshold value corresponding to the relative term defining the condition, wherein the quantitative threshold value is a predetermined percentile of the historical distribution; receiving data associated with the characteristic of the second device of the established location; determining, based on the received data, that the actual value of the characteristic of the second device satisfies the condition as defined by the quantitative threshold value; and in accordance with a determination that the actual value of the characteristic of the second device satisfies the condition as defined by the quantitative threshold value, causing the first device to perform the action. ### Claim 2 (depends on 1) The electronic device of claim 1, wherein the characteristic of the second device of the established location is a device characteristic of a sensor of the second device of the established location. ### Claim 3 (depends on 1) The electronic device of claim 1, wherein determining, based on the received data, that the actual value of the characteristic of the second device satisfies the condition as defined by the quantitative threshold value includes: determining whether the actual value of the characteristic of the second device is greater than the quantitative threshold value. ### Claim 4 (depends on 1) The electronic device of claim 1, wherein determining, based on the received data, that the actual value of the characteristic of the second device satisfies the condition as defined by the quantitative threshold value includes: determining whether the actual value of the characteristic of the second device is less than the quantitative threshold value. ### Claim 5 (depends on 1) The electronic device of claim 1, wherein causing the first device to perform the action further comprises: providing instructions that cause the first device of the established location to perform the action. ### Claim 6 (depends on 1) The electronic device of claim 1, wherein the actionable intent is associated with a second condition, and wherein causing the first device to perform the action is further performed in accordance with a determination that the second condition is satisfied. ### Claim 7 (depends on 6) The electronic device of claim 6, wherein satisfying the second condition requires that a current time be equal to a reference time. ### Claim 8 (depends on 6) The electronic device of claim 6, wherein the one or more programs further include instructions for: receiving second data associated with the second condition; and determining from the received second data whether the second condition is satisfied. ### Claim 9 (depends on 1) The electronic device of claim 1, wherein determining the actionable intent of performing the action with the first device of the established location when the actual value of the characteristic of the second device of the established location satisfies the condition includes: retrieving a data structure representing a plurality of devices of the established location. ### Claim 10 (depends on 9) The electronic device of claim 9, wherein the one or more programs further include instructions for: determining one or more possible device characteristics corresponding to the user request; determining, based on the data structure, one or more candidate devices of the established location; and selecting, based on the data structure and on the possible device characteristics, the first device from the one or more candidate devices. ### Claim 11 (depends on 9) The electronic device of claim 9, wherein the one or more programs further include instructions for: determining, based on the data structure, the second device based on a relationship between the second device and the first device. ### Claim 12 (depends on 1) The electronic device of claim 1, wherein the characteristic of the second device is a temperature characteristic. ### Claim 13 (depends on 1) The electronic device of claim 1, wherein the characteristic of the second device is a time characteristic. ### Claim 14 (depends on 1) The electronic device of claim 1, wherein the characteristic of the second device is an illuminance characteristic. ### Claim 15 (depends on 1) The electronic device of claim 1, wherein the characteristic of the second device is a sound intensity characteristic. ### Claim 16 A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions which, when executed by one or more processors of an electronic device, cause the electronic device to: receive a user request defining a request for a first device of an established location to perform an action and a condition that is required to be satisfied prior to performing the action, wherein the condition is defined using a relative term in the user request; determine, based on the user request and the relative term, an actionable intent of performing the action with the first device of the established location when an actual value of a characteristic of a second device of the established location satisfies the condition, wherein determining the actionable intent includes: responsive to receiving the user request, retrieve a historical distribution of values for the characteristic of the second device of the established location; determine, based on the historical distribution of values for the characteristic of the second device of the established location, a quantitative threshold value corresponding to the relative term defining the condition, wherein the quantitative threshold value is a predetermined percentile of the historical distribution; receive data associated with the characteristic of the second device of the established location; determine, based on the received data, that the actual value of the characteristic of the second device satisfies the condition as defined by the quantitative threshold value; and in accordance with a determination that the actual value of the characteristic of the second device satisfies the condition as defined by the quantitative threshold value, cause the first device to perform the action. ### Claim 17 A method for operating a digital assistant, the method comprising: at an electronic device with a processor and memory: receiving a user request defining a request for a first device of an established location to perform an action and a condition that is required to be satisfied prior to performing the action, wherein the condition is defined using a relative term in the user request; determining, based on the user request and the relative term, an actionable intent of performing the action with the first device of the established location when an actual value of a characteristic of a second device of the established location satisfies the condition, wherein determining the actionable intent includes: responsive to receiving the user request, retrieving a historical distribution of values for the characteristic of the second device of the established location; determining, based on the historical distribution of values for the characteristic of the second device of the established location, a quantitative threshold value corresponding to the relative term defining the condition, wherein the quantitative threshold value is a predetermined percentile of the historical distribution; receiving data associated with the characteristic of the second device of the established location; determining, based on the received data, that the actual value of the characteristic of the second device satisfies the condition as defined by the quantitative threshold value; and in accordance with a determination that the actual value of the characteristic of the second device satisfies the condition as defined by the quantitative threshold value, causing the first device to perform the action. ### Claim 18 (depends on 16) The non-transitory computer-readable storage medium of claim 16, wherein the characteristic of the second device of the established location is a device characteristic of a sensor of the second device of the established location. ### Claim 19 (depends on 16) The non-transitory computer-readable storage medium of claim 16, wherein determining, based on the received data, that the actual value of the characteristic of the second device satisfies the condition as defined by the quantitative threshold value includes: determining whether the actual value of the characteristic of the second device is greater than the quantitative threshold value. ### Claim 20 (depends on 16) The non-transitory computer-readable storage medium of claim 16, wherein determining, based on the received data, that the actual value of the characteristic of the second device satisfies the condition as defined by the quantitative threshold value includes: determining whether the actual value of the characteristic of the second device is less than the quantitative threshold value. ### Claim 21 (depends on 16) The non-transitory computer-readable storage medium of claim 16, wherein causing the first device to perform the action further comprises: providing instructions that cause the first device of the established location to perform the action. ### Claim 22 (depends on 16) The non-transitory computer-readable storage medium of claim 16, wherein the actionable intent is associated with a second condition, and wherein causing the first device to perform the action is further performed in accordance with a determination that the second condition is satisfied. ### Claim 23 (depends on 22) The non-transitory computer-readable storage medium of claim 22, wherein satisfying the second condition requires that a current time be equal to a reference time. ### Claim 24 (depends on 22) The non-transitory computer-readable storage medium of claim 22, wherein the one or more programs further comprise instructions which, when executed by one or more processors of an electronic device, cause the electronic device to: receive second data associated with the second condition; and determine from the received second data whether the second condition is satisfied. ### Claim 25 (depends on 16) The non-transitory computer-readable storage medium of claim 16, wherein determining the actionable intent of performing the action with the first device of the established location when the actual value of the characteristic of the second device of the established location satisfies the condition includes: retrieving a data structure representing a plurality of devices of the established location. ### Claim 26 (depends on 25) The non-transitory computer-readable storage medium of claim 25, wherein the one or more programs further comprise instructions which, when executed by one or more processors of an electronic device, cause the electronic device to: determine one or more possible device characteristics corresponding to the user request; determine, based on the data structure, one or more candidate devices of the established location; and select, based on the data structure and on the possible device characteristics, the first device from the one or more candidate devices. ### Claim 27 (depends on 25) The non-transitory computer-readable storage medium of claim 25, wherein the one or more programs further comprise instructions which, when executed by one or more processors of an electronic device, cause the electronic device to: determine, based on the data structure, the second device based on a relationship between the second device and the first device. ### Claim 28 (depends on 16) The non-transitory computer-readable storage medium of claim 16, wherein the characteristic of the second device is a temperature characteristic. ### Claim 29 (depends on 16) The non-transitory computer-readable storage medium of claim 16, wherein the characteristic of the second device is a time characteristic. ### Claim 30 (depends on 16) The non-transitory computer-readable storage medium of claim 16, wherein the characteristic of the second device is an illuminance characteristic. ### Claim 31 (depends on 16) The non-transitory computer-readable storage medium of claim 16, wherein the characteristic of the second device is a sound intensity characteristic. ### Claim 32 (depends on 17) The method of claim 17, wherein the characteristic of the second device of the established location is a device characteristic of a sensor of the second device of the established location. ### Claim 33 (depends on 17) The method of claim 17, wherein determining, based on the received data, that the actual value of the characteristic of the second device satisfies the condition as defined by the quantitative threshold value includes: determining whether the actual value of the characteristic of the second device is greater than the quantitative threshold value. ### Claim 34 (depends on 17) The method of claim 17, wherein determining, based on the received data, that the actual value of the characteristic of the second device satisfies the condition as defined by the quantitative threshold value includes: determining whether the actual value of the characteristic of the second device is less than the quantitative threshold value. ### Claim 35 (depends on 17) The method of claim 17, wherein causing the first device to perform the action further comprises: providing instructions that cause the first device of the established location to perform the action. ### Claim 36 (depends on 17) The method of claim 17, wherein the actionable intent is associated with a second condition, and wherein causing the first device to perform the action is further performed in accordance with a determination that the second condition is satisfied. ### Claim 37 (depends on 36) The method of claim 36, wherein satisfying the second condition requires that a current time be equal to a reference time. ### Claim 38 (depends on 36) The method of claim 36, further comprising: receiving second data associated with the second condition; and determining from the received second data whether the second condition is satisfied. ### Claim 39 (depends on 17) The method of claim 17, wherein determining the actionable intent of performing the action with the first device of the established location when the actual value of the characteristic of the second device of the established location satisfies the condition includes: retrieving a data structure representing a plurality of devices of the established location. ### Claim 40 (depends on 39) The method of claim 39, further comprising: determining one or more possible device characteristics corresponding to the user request; determining, based on the data structure, one or more candidate devices of the established location; and selecting, based on the data structure and on the possible device characteristics, the first device from the one or more candidate devices. ### Claim 41 (depends on 39) The method of claim 39, further comprising: determining, based on the data structure, the second device based on a relationship between the second device and the first device. ### Claim 42 (depends on 17) The method of claim 17, wherein the characteristic of the second device is a temperature characteristic. ### Claim 43 (depends on 17) The method of claim 17, wherein the characteristic of the second device is a time characteristic. ### Claim 44 (depends on 17) The method of claim 17, wherein the characteristic of the second device is an illuminance characteristic. ### Claim 45 (depends on 17) The method of claim 17, wherein the characteristic of the second device is a sound intensity characteristic.