# US 12,222,512 B2 · Displaying content based on state information > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,222,512 B2 - Document ID: US12222512B2 - Status: Granted - Kind: B2 - Application number: 18/339,121 - Filed: 2023-06-21 - Published: 2024-02-22 - Granted: 2025-02-11 - Earliest claimed: 2022-08-18 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Gregory Lutter - Web page: https://patentgrep.com/patent/US12222512B2 ## CPC - G02B 27/0172: {characterised by optical features} - G02B 2027/0138: {comprising image capture systems, e.g. camera} - G02B 2027/014: {comprising information/image processing systems} - G02B 2027/0141: {characterised by the informative content of the display} - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS ## Abstract A head-mounted device may determine contextual information by analyzing sensor data. The head-mounted device may use computer vision analysis to determine contextual information from images of the physical environment around the head-mounted device. Instead or in addition, the head-mounted device may determine contextual information by receiving state information directly from external equipment within the physical environment. Based on the received state information, the head-mounted device may display content, play audio, change a device setting on the head-mounted device, the external equipment, and/or additional external equipment, and/or may open an application. The head-mounted device may receive the state information in accordance with identifying the external equipment in images of the physical environment. The head-mounted device may receive the state information and then obtain and analyze sensor data to determine contextual information in response to the received state information being insufficient. ## Claims ### Claim 1 An electronic device comprising: one or more sensors comprising one or more cameras; communication circuitry; one or more processors; and memory storing instructions configured to be executed by the one or more processors, the instructions for: capturing, via the one or more cameras, one or more images; analyzing the one or more images, wherein analyzing the one or more images comprises detecting an external electronic device in the one or more images; obtaining, via the communication circuitry, state information from the external electronic device; and in accordance with a determination that the state information does not satisfy a criterion: capturing, via the one or more cameras, one or more additional images; estimating state information for the external electronic device based on the one or more additional images; and presenting content that is based on the estimated state information for the external electronic device. ### Claim 2 (depends on 1) The electronic device defined in claim 1, wherein the instructions further comprise instructions for: ceasing to analyze the one or more images in response to obtaining, via the communication circuitry, the state information from the external electronic device. ### Claim 3 (depends on 1) The electronic device defined in claim 1, wherein analyzing the one or more images comprises determining a first position of the external electronic device and wherein the instructions further comprise instructions for: determining a second position at which to present the content based on the first position of the external electronic device, wherein presenting the content comprises presenting the content at the second position. ### Claim 4 (depends on 3) The electronic device defined in claim 3, wherein the external electronic device is positioned in a physical environment, wherein analyzing the one or more images to determine the first position of the external electronic device comprises determining the first position of the external electronic device within the physical environment, and wherein the electronic device further comprises: one or more displays configured to emit light from a transparent structure towards an eye box, wherein the second position overlaps the external electronic device when viewed from the eye box. ### Claim 5 (depends on 1) The electronic device defined in claim 1, wherein the instructions further comprise instructions for: in response to detecting the external electronic device in the one or more images: wirelessly pairing with the external electronic device; and transmitting, via the communication circuitry, a request for state information to the external electronic device. ### Claim 6 (depends on 1) The electronic device defined in claim 1, wherein the state information comprises an application running on the external electronic device, comprises application information for an application running on the external electronic device, identifies content being displayed on a display in the external electronic device, or identifies audio being played on a speaker in the external electronic device. ### Claim 7 (depends on 1) The electronic device defined in claim 1, wherein the instructions further comprise instructions for: changing one or more settings in the electronic device, the external electronic device, or an additional external electronic device based on the state information received from the external electronic device. ### Claim 8 A method of operating an electronic device that comprises one or more sensors and communication circuitry, the one or more sensors comprising one or more cameras and the method comprising: capturing, via the one or more cameras, one or more images; analyzing the one or more images, wherein analyzing the one or more images comprises detecting an external electronic device in the one or more images; obtaining, via the communication circuitry, state information from the external electronic device; and in accordance with a determination that the state information does not satisfy a criterion: capturing, via the one or more cameras, one or more additional images; estimating state information for the external electronic device based on the one or more additional images; and presenting content that is based on the estimated state information for the external electronic device. ### Claim 9 (depends on 8) The method defined in claim 8, further comprising: ceasing to analyze the one or more images in response to obtaining, via the communication circuitry, the state information from the external electronic device. ### Claim 10 (depends on 8) The method defined in claim 8, wherein analyzing the one or more images comprises determining a first position of the external electronic device and wherein the method further comprises: determining a second position at which to present the content based on the first position of the external electronic device, wherein presenting the content comprises presenting the content at the second position. ### Claim 11 (depends on 10) The method defined in claim 10, wherein the external electronic device is positioned in a physical environment, wherein analyzing the one or more images to determine the first position of the external electronic device comprises determining the first position of the external electronic device within the physical environment, wherein the electronic device further comprises one or more displays configured to emit light from a transparent structure towards an eye box, and wherein the second position overlaps the external electronic device when viewed from the eye box. ### Claim 12 (depends on 8) The method defined in claim 8, further comprising: in response to detecting the external electronic device in the one or more images: wirelessly pairing with the external electronic device; and transmitting, via the communication circuitry, a request for state information to the external electronic device. ### Claim 13 (depends on 8) The method defined in claim 8, wherein the state information comprises an application running on the external electronic device, comprises application information for an application running on the external electronic device, identifies content being displayed on a display in the external electronic device, or identifies audio being played on a speaker in the external electronic device. ### Claim 14 (depends on 8) The method defined in claim 8, further comprising: changing one or more settings in the electronic device, the external electronic device, or an additional external electronic device based on the state information received from the external electronic device. ### Claim 15 A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device that comprises one or more sensors and communication circuitry, the one or more sensors comprising one or more cameras and the one or more programs including instructions for: capturing, via the one or more cameras, one or more images; analyzing the one or more images, wherein analyzing the one or more images comprises detecting an external electronic device in the one or more images; obtaining, via the communication circuitry, state information from the external electronic device; and in accordance with a determination that the state information does not satisfy a criterion: capturing, via the one or more cameras, one or more additional images; estimating state information for the external electronic device based on the one or more additional images; and presenting content that is based on the estimated state information for the external electronic device. ### Claim 16 (depends on 15) The non-transitory computer-readable storage medium defined in claim 15, wherein the instructions further comprise instructions for: ceasing to analyze the one or more images in response to obtaining, via the communication circuitry, the state information from the external electronic device. ### Claim 17 (depends on 15) The non-transitory computer-readable storage medium defined in claim 15, wherein analyzing the one or more images comprises determining a first position of the external electronic device and wherein the instructions further comprise instructions for: determining a second position at which to present the content based on the first position of the external electronic device, wherein presenting the content comprises presenting the content at the second position. ### Claim 18 (depends on 17) The non-transitory computer-readable storage medium defined in claim 17, wherein the external electronic device is positioned in a physical environment, wherein analyzing the one or more images to determine the first position of the external electronic device comprises determining the first position of the external electronic device within the physical environment, wherein the electronic device further comprises one or more displays configured to emit light from a transparent structure towards an eye box, and wherein the second position overlaps the external electronic device when viewed from the eye box. ### Claim 19 (depends on 15) The non-transitory computer-readable storage medium defined in claim 15, wherein the instructions further comprise instructions for: in response to detecting the external electronic device in the one or more images: wirelessly pairing with the external electronic device; and transmitting, via the communication circuitry, a request for state information to the external electronic device. ### Claim 20 (depends on 15) The non-transitory computer-readable storage medium defined in claim 15, wherein the state information comprises an application running on the external electronic device, comprises application information for an application running on the external electronic device, identifies content being displayed on a display in the external electronic device, or identifies audio being played on a speaker in the external electronic device. ### Claim 21 (depends on 15) The non-transitory computer-readable storage medium defined in claim 15, wherein the instructions further comprise instructions for: changing one or more settings in the electronic device, the external electronic device, or an additional external electronic device based on the state information received from the external electronic device.