# US 12,602,106 B2 · Ambience-driven user experience > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,106 B2 - Document ID: US12602106B2 - Status: Granted - Kind: B2 - Application number: 18/088,300 - Filed: 2022-12-23 - Published: 2023-10-19 - Granted: 2026-04-14 - Earliest claimed: 2020-06-29 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Benjamin H. Boesel; David H. Huang; Jonathan Perron; Shih-Sang Chiu - Web page: https://patentgrep.com/patent/US12602106B2 ## CPC - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking} - G02B 27/0172: {characterised by optical features} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G02B 2027/0138: {comprising image capture systems, e.g. camera} - G02B 2027/014: {comprising information/image processing systems} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS ## Abstract In accordance with various implementations, a method is performed at an electronic device with one or more processors, a non-transitory memory, and a display. The method includes determining an engagement score associated with an object that is visible at the display. The engagement score characterizes a level of user engagement with respect to the object. The method includes, in response to determining that the engagement score satisfies an engagement criterion, determining an ambience vector associated with the object and presenting content based on the ambience vector. The ambience vector represents a target ambient environment. ## Claims ### Claim 1 A method comprising: at an electronic device with one or more processors, a non-transitory memory, and a display: displaying, on the display, a representation of an operating environment; determining an engagement score associated with an object within the operating environment, wherein the engagement score characterizes a level of user engagement with respect to the object; and in response to determining that the engagement score satisfies an engagement criterion: obtaining one or more semantic values of the object and generating an ambience vector associated with the object, wherein the ambience vector specifies at least one of a sound, a mood or weather conditions for the representation of the operating environment derived from the one or more semantic values of the object; and presenting content in the representation of the operating environment by outputting audio that corresponds to the sound or by displaying a virtual object that corresponds to the mood or the weather conditions specified by the ambience vector. ### Claim 2 (depends on 1) The method of claim 1, further comprising: detecting, via an input device, a user input; and in response to detecting the user input: in accordance with a determination that the user input is directed towards presenting content associated with the object, presenting the content; and in accordance with a determination that the user input is not directed towards presenting the content associated with the object, foregoing presenting the content. ### Claim 3 (depends on 1) The method of claim 1, wherein the electronic device is in a first mode of operation, and wherein presenting the content is further based on determining that the electronic device is in the first mode of operation. ### Claim 4 (depends on 3) The method of claim 3, further comprising detecting, via an input device, a user input directing the electronic device to enter the first mode of operation, wherein presenting the content is further based on detecting the user input. ### Claim 5 (depends on 1) The method of claim 1, wherein the electronic device includes an extremity tracker that outputs extremity tracking data, and wherein determining the engagement score is based on a function of the extremity tracking data. ### Claim 6 (depends on 1) The method of claim 1, wherein the electronic device includes an eye tracker that outputs eye tracking data, and wherein determining the engagement score is based on a function of the eye tracking data. ### Claim 7 (depends on 1) The method of claim 1, wherein the engagement score satisfies the engagement criterion when the user engagement spatially corresponds to the object. ### Claim 8 (depends on 1) The method of claim 1, wherein the object corresponds to a physical object. ### Claim 9 (depends on 8) The method of claim 8, further comprising obtaining a semantic value associated with the physical object, wherein the ambience vector is based on the semantic value. ### Claim 10 (depends on 8) The method of claim 8, wherein the electronic device includes an image sensor, wherein the image sensor is associated with a capture region that includes the physical object, wherein the image sensor outputs image data representing the physical object, and wherein the display displays the image data representing the physical object. ### Claim 11 (depends on 8) The method of claim 8, wherein the display corresponds to a see-through display, and wherein the physical object is visible through the see-through display. ### Claim 12 (depends on 1) The method of claim 1, wherein the object corresponds to a computer-generated object. ### Claim 13 (depends on 1) The method of claim 1, wherein the electronic device includes a speaker, wherein the content corresponds to audio content, and wherein presenting the content includes playing, via the speaker, the audio content. ### Claim 14 (depends on 13) The method of claim 13, further comprising: identifying a second object based on the ambience vector; determining a spatial relationship between the electronic device and the second object; and playing the audio content, as spatial audio, based on a function of the spatial relationship. ### Claim 15 (depends on 1) The method of claim 1, wherein the content corresponds to displayable content, and wherein presenting the content includes displaying, on the display, the displayable content. ### Claim 16 (depends on 15) The method of claim 15, further comprising: identifying a second object based on the ambience vector; and determining a spatial relationship between the electronic device and the second object; wherein the displayable content is displayed based on a function of the spatial relationship. ### Claim 17 (depends on 15) The method of claim 15, wherein the displayable content satisfies a depth threshold with respect to the display. ### Claim 18 (depends on 1) The method of claim 1, wherein the ambience vector includes one or more values that characterize an ambient light, an ambient sound, or ambient weather of a target ambience of the representation of the operating environment. ### Claim 19 (depends on 1) The method of claim 1, further comprising: in response to determining that the engagement score satisfies an engagement criterion: transmitting, to a second electronic device, an instruction to change an operating characteristic of the operating environment based on the ambience vector. ### Claim 20 A non-transitory memory storing one or more programs, which, when executed by one or more processors of a device, cause the device to: display, on a display, a representation of an operating environment; determine an engagement score associated with an object within the operating environment, wherein the engagement score characterizes a level of user engagement with respect to the object; and in response to determining that the engagement score satisfies an engagement criterion: obtain one or more semantic values of the object and generate an ambience vector associated with the object, wherein the ambience vector specifies at least one of a sound, a mood or weather conditions for the representation of the operating environment derived from the one or more semantic values of the object; and present content in the representation of the operating environment by outputting audio that corresponds to the sound or by displaying a virtual object that corresponds to the mood or the weather conditions specified by the ambience vector. ### Claim 21 An electronic device comprising: one or more processors; a non-transitory memory; a display; and one or more programs, wherein the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors, the one or more programs including instructions for: displaying, on the display, a representation of an operating environment; determining an engagement score associated with an object within the operating environment, wherein the engagement score characterizes a level of user engagement with respect to the object; and in response to determining that the engagement score satisfies an engagement criterion: obtain one or more semantic values of the object and generate an ambience vector associated with the object, wherein the ambience vector specifies at least one of a sound, a mood or weather conditions for the representation of the operating environment derived from the one or more semantic values of the object; and presenting content in the representation of the operating environment by outputting audio that corresponds to the sound or by displaying a virtual object that corresponds to the mood or the weather conditions specified by the ambience vector.