# US 10,503,996 B2 · Context-aware display of objects in mixed environments > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,503,996 B2 - Document ID: US10503996B2 - Status: Granted - Kind: B2 - Application number: 15/670,836 - Filed: 2017-08-07 - Published: 2017-11-23 - Granted: 2019-12-10 - Earliest claimed: 2015-05-12 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Chris M. Novak - Web page: https://patentgrep.com/patent/US10503996B2 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06V 10/768: {using context analysis, e.g. recognition aided by known co-occurring patterns} - G06V 20/20: in augmented reality scenes - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Concepts and technologies are described herein for providing a context-aware display of objects in mixed environments. The computing device displays an image on an interface. Data associated with at least one object is obtained from one or more resources, such as a camera, sensor and/or an external device. The obtained data is analyzed to determine if the object is of interest. Any number of factors, including a context associated with the object, may be used to determine if an object is of interest. The computing device modifies a display property, such as a transparency parameter, of a selected portion of the image, depending on obtained data and/or contextual data. Depending on the obtained data and/or contextual data, the computing device may display a visual indicator representing the object, an image of the object within the selected portion of the image, and/or any other object or indicator. ## Claims ### Claim 1 A device comprising: a processor; and memory having executable instructions stored thereupon which, when executed by the processor, cause the device to: display a virtual reality image on an interface; obtain data associated with a real-world object that is not viewable via the interface, wherein the data comprises at least one of shape data or sound data; determine a type of the real-world object based at least in part on the at least one of the shape data or the sound data; determine that the real-world object is moving relative to a location of the device; based at least in part on the determining that the real-world object is moving relative to the location of the device, display a visual indicator on a portion of the virtual reality image, wherein the visual indicator represents the type of the real-world object; and transition the display of the visual indicator on the portion of the virtual reality image into a real-world view of the real-world object as the real-world object moves closer to the location of the device. ### Claim 2 (depends on 1) The device of claim 1, wherein transitioning the display of the visual indicator on the portion of the virtual reality image into the real-world view of the real-world object as the real-world object moves closer to the location of the device comprises increasing a level of transparency for the portion of the virtual reality image when the real-world object moves from a first distance to a second distance with respect to the location of the device, the first distance being greater than the second distance. ### Claim 3 (depends on 1) The device of claim 1, wherein the memory has further executable instructions, which cause the device to adjust a transparency of the portion of the virtual reality image to provide the real-world view of the real-world object through the interface as the real-world object moves closer to the location of the device. ### Claim 4 (depends on 1) The device of claim 1, wherein the data associated with the real-world object comprises the sound data and the memory has further executable instructions, which cause the device to: determine that the sound data is being emitted by the real-world object toward the device; assign a priority to the real-world object based at least in part on the determining that the sound data is being emitted by the real-world object toward the device; and determine, based at least in part on the priority, at least one of a size or a position of the visual indicator on the virtual reality image. ### Claim 5 (depends on 1) The device of claim 1, wherein the memory has further executable instructions, which cause the device to modify a status of an application controlling the virtual reality image based at least in part on the determining that the real-world object is moving relative to the location of the device. ### Claim 6 (depends on 5) The device of claim 5, wherein the modifying the status of the application includes at least one of reducing a volume control associated with the application or pausing one or more processes of the application. ### Claim 7 (depends on 1) The device of claim 1, wherein the memory has further executable instructions, which cause the device to: determine that the real-world object is within a predefined distance of the location of the device; and display the visual indicator on the portion of the virtual reality image based at least in part on determining that the real-world object is within the predefined distance of the location of the device. ### Claim 8 (depends on 7) The device of claim 7, wherein the predefined distance is configured to be set by a user of the device. ### Claim 9 A device comprising: a processor; and memory having executable instructions stored thereupon which, when executed by the processor, cause the device to: display a virtual reality image on an interface; obtain data associated with a real-world object that is not viewable via the interface, wherein the data comprises at least one of shape data or pattern data; determine, based at least in part on the at least one of the shape data or the pattern data, a type of the real-world object; based at least in part on the determining the type of the real-world object, display a visual indicator on a portion of the virtual reality image, wherein the visual indicator represents the type of the real-world object; and transition the display of the visual indicator on the portion of the virtual reality image into a real-world view of the real-world object as the real-world object moves closer to a location of the device. ### Claim 10 (depends on 9) The device of claim 9, wherein the data further comprises location data and the memory has further executable instructions, which cause the device to: determine, based at least in part on the location data, a location of the real-world object relative to the location of the device; and display the visual indicator on the portion of the virtual reality image based at least in part on the location of the real-world object relative to the location of the device. ### Claim 11 (depends on 10) The device of claim 10, wherein the memory has further executable instructions, which cause the device to: determine, based at least in part on the location data, that the real-world object is moving relative to the location of the device; and adjust a transparency of the visual indicator to provide the real-world view of the real-world object based at least in part on the determining that the real-world object is moving relative to the location of the device. ### Claim 12 (depends on 9) The device of claim 9, wherein transitioning the display of the visual indicator on the portion of the virtual reality image into the real-world view of the real-world object as the real-world object moves closer to the location of the device comprises increasing a level of transparency for the portion of the virtual reality image when the real-world object moves from a first distance to a second distance with respect to the location of the device, the first distance being greater than the second distance. ### Claim 13 (depends on 10) The device of claim 10, wherein the memory has further executable instructions, which cause the device to modify a status of an application controlling the virtual reality image based at least in part on the location of the real-world object relative to the location of the device. ### Claim 14 (depends on 13) The device of claim 13, wherein the modifying the status of the application includes at least one of reducing a volume control associated with the application or pausing one or more processes of the application. ### Claim 15 (depends on 9) The device of claim 9, wherein the data further comprises sound data and the memory has further executable instructions, which cause the device to determine the type of the real-world object based at least in part on the sound data. ### Claim 16 (depends on 9) The device of claim 9, wherein the memory has further executable instructions, which cause the device to: determine that the real-world object is within a predefined distance of the location of the device; and display the visual indicator on the portion of the virtual reality image based at least in part on determining that the real-world object is within the predefined distance of the location of the device. ### Claim 17 A method comprising: displaying a virtual reality image on an interface associated with a device; obtaining, by the device, data associated with a real-world object that is not viewable via the interface, wherein the data comprises at least one of shape data or pattern data; determining, based at least in part on the at least one of the shape data or the pattern data, a type of the real-world object; based at least in part on the determining the type of the real-world object, displaying a visual indicator on a portion of the virtual reality image, wherein the visual indicator represents the type of the real-world object; and transitioning the display of the visual indicator on the portion of the virtual reality image into a real-world view of the real-world object as the real-world object moves closer to a location of the device. ### Claim 18 (depends on 17) The method of claim 17, wherein the data further comprises location data and the method further comprises: determining, based at least in part on the location data, that the real-world object is moving relative to the location of the device; and adjust a transparency of the visual indicator to provide the real-world view of the real-world object based at least in part on the determining that the real-world object is moving relative to the location of the device. ### Claim 19 (depends on 18) The method of claim 18, further comprising modifying a status of an application controlling the virtual reality image based at least in part on the determining that the real-world object is moving relative to the location of the device, wherein the modifying the status of the application includes at least one of reducing a volume control associated with the application or pausing one or more processes of the application. ### Claim 20 (depends on 17) The method of claim 17, wherein the data further comprises sound data and the method further comprises determining the type of the real-world object based at least in part on the sound data.