# US 2018/0322706 A1 · Contextual applications in a mixed reality environment > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2018/0322706 A1 - Document ID: US20180322706A1 - Status: Published application - Kind: A1 - Application number: 15/972,020 - Filed: 2018-05-04 - Published: 2018-11-08 - Earliest claimed: 2017-05-05 - Assignees: Unity IPR ApS - Inventors: Sylvio Herve Drouin; Gregory Lionel Xavier Jean Palmaro; Dioselin Alejandra Gonzalez Rosillo - Web page: https://patentgrep.com/patent/US20180322706A1 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G02B 27/0172: {characterised by optical features} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range - G06V 20/64: Three-dimensional [3D] objects - G02B 2027/0138: {comprising image capture systems, e.g. camera} - G02B 2027/0178: {Eyeglass type (eyeglass details G02C)} - G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method of implementing a plurality of contextual applications within a mixed reality (MR) environment on an MR-capable device of a user is disclosed. At least one real-world object is identified in the MR environment by applying an object recognition algorithm to one or more attributes of the at least one real-world object that are captured by sensors of the MR-capable device. A first contextual application of the plurality of contextual applications is used to determine an association between a first set of contextual triggers and a second contextual application of the plurality of contextual applications. A second contextual application is initiated based on a satisfying of the at least one contextual trigger. A function is invoked within the second contextual application based on an interaction of the user with at least one virtual object satisfying a second set of contextual triggers associated with the second contextual application. ## Claims ### Claim 1 A system comprising: one or more computer processors; one or more computer memories; and a set of instructions incorporated into the one or more computer memories, the set of instructions configuring the one or more computer processors to perform operations for implementing a plurality of contextual applications within a mixed reality (MR) environment on an MR-capable device of a user, the operations comprising: identifying at least one real-world object in the MR environment by applying an object recognition algorithm to one or more attributes of the at least one real-world object that are captured by sensors of the MR-capable device; using a first contextual application of the plurality of contextual applications to determine an association between a first set of contextual triggers and a second contextual application of the plurality of contextual applications, the first contextual application being a launcher application; initiating the second contextual application based on a satisfying of the at least one contextual trigger, the initiating including presenting at least one virtual object that is associated with the second contextual application in the mixed reality environment; and invoking a function within the second contextual application based on an interaction of the user with the at least one virtual object satisfying a second set of contextual triggers associated with the second contextual application. ### Claim 2 (depends on 1) The system of claim 1, wherein the applying of the object recognition algorithm includes using artificial intelligence to perform the identifying of the at least one real-world object in the MR environment. ### Claim 3 (depends on 1) The system of claim 1, wherein the presenting of the at least one virtual object by the second contextual application is based on a correspondence between the user and a shared display layer associated with the second contextual application running on a second MR-capable device. ### Claim 4 (depends on 1) The system of claim 1, wherein the interaction of the user includes using the at least one real-world object as a controller. ### Claim 5 (depends on 1) The system of claim 1, wherein the initiating of the second contextual application includes applying diminished reality technology to modify a depiction of the at least one real-world object on the MR-capable device. ### Claim 6 (depends on 4) The system of claim 4, wherein the using of the at least one real-world object as a controller includes manipulating the real-world object in three dimensions to control an action within the second contextual application, the action defined using an application program interface of the second contextual application. ### Claim 7 (depends on 1) The system of claim 1, wherein the first set of contextual triggers includes a combination of elements including at least two of data pertaining to a time of the identifying, data pertaining to a location of the MR device, and data pertaining to a set of additional real-world objects identified in the MR environment. ### Claim 8 A method comprising: incorporating a set of instructions into one or more computer memories, the set of instructions configuring one or more computer processors to perform operations for implementing a plurality of contextual applications within a mixed reality (MR) environment on an MR-capable device of a user, the operations comprising: identifying at least one real-world object in the MR environment by applying an object recognition algorithm to one or more attributes of the at least one real-world object that are captured by sensors of the MR-capable device; using a first contextual application of the plurality of contextual applications to determine an association between a first set of contextual triggers and a second contextual application of the plurality of contextual applications, the first contextual application being a launcher application; initiating the second contextual application based on a satisfying of the at least one contextual trigger, the initiating including presenting at least one virtual object that is associated with the second contextual application in the mixed reality environment; and invoking a function within the second contextual application based on an interaction of the user with the at least one virtual object satisfying a second set of contextual triggers associated with the second contextual application. ### Claim 9 (depends on 8) The method of claim 8, wherein the applying of the object recognition algorithm includes using artificial intelligence to perform the identifying of the at least one real-world object in the MR environment. ### Claim 10 (depends on 8) The method of claim 8, wherein the presenting of the at least one virtual object by the second contextual application is based on a correspondence between the user and a shared display layer associated with the second contextual application running on a second MR-capable device. ### Claim 11 (depends on 8) The method of claim 8, wherein the interaction of the user includes using the at least one real-world object as a controller. ### Claim 12 (depends on 8) The method of claim 8, wherein the initiating of the second contextual application includes applying diminished reality technology to modify a depiction of the at least one real-world object on the MR-capable device. ### Claim 13 (depends on 11) The method of claim 11, wherein the using of the at least one real-world object as a controller includes manipulating the real-world object in three dimensions to control an action within the second contextual application, the action defined using an application program interface of the second contextual application. ### Claim 14 The method of 8, wherein the first set of contextual triggers includes a combination of elements including at least two of data pertaining to a time of the identifying, data pertaining to a location of the MR device, and data pertaining to a set of additional real-world objects identified in the MR environment. ### Claim 15 A non-transitory machine-readable medium embodying a set of instructions, the set of instructions configuring one or more computer processors to perform operations for implementing a plurality of contextual applications within a mixed reality (MR) environment on an MR-capable device of a user, the operations comprising: identifying at least one real-world object in the MR environment by applying an object recognition algorithm to one or more attributes of the at least one real-world object that are captured by sensors of the MR-capable device; using a first contextual application of the plurality of contextual applications to determine an association between a first set of contextual triggers and a second contextual application of the plurality of contextual applications, the first contextual application being a launcher application; initiating the second contextual application based on a satisfying of the at least one contextual trigger, the initiating including presenting at least one virtual object that is associated with the second contextual application in the mixed reality environment; and invoking a function within the second contextual application based on an interaction of the user with the at least one virtual object satisfying a second set of contextual triggers associated with the second contextual application. ### Claim 16 (depends on 15) The non-transitory machine-readable medium of claim 15, wherein the applying of the object recognition algorithm includes using artificial intelligence to perform the identifying of the at least one real-world object in the MR environment. ### Claim 17 (depends on 15) The non-transitory machine-readable medium of claim 15, wherein the presenting of the at least one virtual object by the second contextual application is based on a correspondence between the user and a shared display layer associated with the second contextual application running on a second MR-capable device. ### Claim 18 (depends on 15) The non-transitory machine-readable medium of claim 15, wherein the interaction of the user includes using the at least one real-world object as a controller. ### Claim 19 (depends on 15) The non-transitory machine-readable medium of claim 15, wherein the initiating of the second contextual application includes applying diminished reality technology to modify a depiction of the at least one real-world object on the MR-capable device. ### Claim 20 (depends on 18) The non-transitory machine-readable medium of claim 18, wherein the using of the at least one real-world object as a controller includes manipulating the real-world object in three dimensions to control an action within the second contextual application, the action defined using an application program interface of the second contextual application.