# US 12,008,717 B2 · Artificial reality environment control through an artificial reality environment schema > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,717 B2 - Document ID: US12008717B2 - Status: Granted - Kind: B2 - Application number: 17/369,267 - Filed: 2021-07-07 - Published: 2023-01-12 - Granted: 2024-06-11 - Earliest claimed: 2021-07-07 - Assignees: Meta Platforms Technologies, LLC - Applicants: Meta Platforms Technologies, LLC - Inventors: Gioacchino Noris; Michal Hlavac; Paul Timothy Furgale; Johannes Joachim Schmid; Anush Mohan; Christopher Richard Tanner - Web page: https://patentgrep.com/patent/US12008717B2 ## CPC - G06F 9/451: Execution arrangements for user interfaces - G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/012: {Head tracking input arrangements} - G06F 16/909: using geographical or spatial information, e.g. location (spatiotemporally dependent retrieval from the web G06F16/9537) - G06F 8/24: {Object-oriented} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract An artificial reality environment (XRE) schema is defined that supports anel-controlling interactions between various artificial reality actors. The XRE schema includes a set of definitions for an XRE, independent of type of artificial reality device. The definitions in the XRE schema can include standards for both interfaces and data objects. The XRE schema can define XR elements in terms of entities and components of a space, organized according to a hierarchy. Each entity can represent a real or virtual object or space, within the XRE, defined by a name and a collection of one or more components. Each component (as part of an entity) can define aspects and expose information about the entity. The XRE schema can specify structures that that allow actors (e.g., producers, instantiators, and consumers) to define and perform actions in relation to XRE elements. ## Claims ### Claim 1 A method for device-agnostic representation of an artificial reality environment (XRE) using an XRE schema, the method comprising: defining multiple features, in the artificial reality environment, each as an element defined by the XRE schema as either an entity or a component, wherein the elements include at least one entity and each entity has a defined name, definition, and component list, wherein the elements include at least one component and at least one component has a defined name, definition, entity list, and query list, and wherein at least one element comprises one or more quantity guarantees with respect to its component list or entity list; performing, for the elements defined by the XRE schema, application programming interface element constructor calls, wherein constructed instances for the elements are returned in response to the element constructor calls, and wherein at least one of the called element constructors that corresponds to the at least one element enforces the one or more quantity guarantees with respect to its component list or entity list; and adding a node for each of the constructed instances of each element to a directed acyclic graph hierarchy for the artificial reality environment, wherein the hierarchy has a pre-defined type of a root node that corresponds to an entity element, wherein each edge in the hierarchy specifies a parent/child link between elements where the parent element contains the child element; wherein the entity corresponding to the root node links to a physical boundaries component linked to one or more wall, floor, and/or ceiling entities; and wherein the method is performed on a first type of artificial reality device and at least one of the elements is accessed by a second type of artificial reality device with a type different from the first type of artificial reality device. ### Claim 2 (depends on 1) The method of claim 1, wherein each entity only links to components and each component only links to entities. ### Claim 3 (depends on 1) The method of claim 1, wherein the at least one element comprises two elements including a first entity and a first component, wherein the first entity specifies multiple component quantity guarantees, each component quantity guarantee defining an amount of components of a particular type the first entity is linked to; and wherein the first component specifies at least one entity quantity guarantee, each entity quantity guarantee defining an amount of entities of a particular type the first component is linked to. ### Claim 4 (depends on 1) The method of claim 1, wherein the root node is for a room entity specifying an area that is traversable without going through a doorway, taking stairs, or taking an elevator. ### Claim 5 (depends on 1) The method of claim 1, wherein each entity element is linked to an anchor component specifying where in 3D space the entity exists in relation to an origin point. ### Claim 6 (depends on 1) The method of claim 1, wherein the hierarchy is a first hierarchy and the root node is a first root node, and wherein the entity corresponding to the first root node includes a component that defines a connection to a second root node of a second hierarchy for another artificial reality environment. ### Claim 7 (depends on 1) The method of claim 1, wherein multiple of the elements, defined by the XRE schema, were defined by a producer process including: defining an entity specifying at least an entity name and a list of components for the entity with associated component amount guarantees; and for at least one particular component in the list of components, defining the particular component specifying a component name, a list of entities for the particular component with associated entity amount guarantees, and a query list for functions configured to act in relation to the particular component. ### Claim 8 (depends on 1) The method of claim 1, wherein performing the application programming interface element constructor calls further comprises: obtaining element parameters corresponding to one or more element constructors defined for the elements, wherein, for a particular one of the elements, one or more of the element parameters were generated by machine perception systems that automatically identified aspects of the real world around the artificial reality device of the first type; and calling the one or more element constructors. ### Claim 9 (depends on 1) The method of claim 1, wherein performing the application programming interface element constructor calls further comprises: obtaining element parameters corresponding to one or more element constructors defined for the elements, wherein at least one element constructor is defined by a producer process; and calling the one or more element constructors. ### Claim 10 (depends on 1) The method of claim 1, wherein at least one particular element, of the elements, is acted upon by a consumer by: obtaining a handle for the at least one particular element; and accessing, via the handle, a query from a query list of the at least one particular element or setting or getting, via the handle, an element part of the at least one particular element. ### Claim 11 (depends on 1) The method of claim 1, wherein the first type of artificial reality device is a virtual reality device and the second type of artificial reality device is a mixed reality device. ### Claim 12 (depends on 1) The method of claim 1, wherein at least four of the elements are a plane, a line, a point, and a mesh fundamental component elements, each linked to an entity. ### Claim 13 (depends on 1) The method of claim 1, wherein at least one particular element of the elements is a purpose manager component linked to one or more purposed zone entities and/or one or more purposed surfaces entities; wherein each of the one or more purposed zone entities is linked to at least an anchor component defining a position in 3D space, a volume component defining a volume in the 3D space, and a purpose component specifying a natural language purpose of the purposed zone; and wherein each of the one or more purposed surface entities is linked to at least an anchor component defining a position in 3D space, a plane component defining a plane in the 3D space, and a purpose component specifying a natural language purpose of the purposed surface. ### Claim 14 A computing system for device-agnostic representation of an artificial reality environment (XRE) using an XRE schema, the computing system comprising: one or more processors; and one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising: defining multiple features, in the artificial reality environment, each as an element defined by the XRE schema as either an entity or a component, wherein the elements include at least one entity and each entity has a defined name and component list, wherein the elements include at least one component with a defined name, entity list, and query list, and wherein at least one element comprises one or more quantity guarantees with respect to its component list or entity list; performing, for the elements defined by the XRE schema, application programming interface element constructor calls, wherein constructed instances for the elements are returned in response to the element constructor calls, and wherein at least one of the called element constructors that corresponds to the at least one element enforces the one or more quantity guarantees with respect to its component list or entity list; and adding a node for each of the constructed instances of each element to a hierarchy for the artificial reality environment, wherein the hierarchy has a pre-defined type of a root node that corresponds to an entity element, wherein each edge in the hierarchy specifies a parent/child link between elements, wherein the entity corresponding to the root node links to a physical boundaries component linked to one or more wall, floor, and/or ceiling entities, and wherein the computing system is a first type of artificial reality device and at least one of the elements is accessed by a second type of artificial reality device with a type different from the first type of artificial reality device. ### Claim 15 (depends on 14) The computing system of claim 14, wherein the entity corresponding to the root node links to a physical boundaries component linked to a wall entity. ### Claim 16 (depends on 14) The computing system of claim 14, wherein the computing system is a mixed reality device; and wherein at least one of the elements is accessed by a virtual reality device different from the mixed reality device. ### Claim 17 (depends on 14) The computing system of claim 14, wherein at least four of the elements are a plane, a line, a point, and a mesh fundamental component elements, each linked to an entity. ### Claim 18 A non-transitory computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process for device-agnostic representation of an artificial reality environment (XRE) using an XRE schema, the process comprising: defining multiple features, in the artificial reality environment, each as an element defined by the XRE schema as either an entity or a component, wherein the elements include at least one entity and each entity has a defined name, definition, and component list, wherein the elements include at least one component and at least one component has a defined name, definition, entity list, and query list, and wherein at least one element comprises one or more quantity guarantees with respect to its component list or entity list; performing, for the elements defined by the XRE schema, application programming interface element constructor calls, wherein constructed instances for the elements are returned in response to the element constructor calls, and wherein at least one of the called element constructors that corresponds to the at least one element enforces the one or more quantity guarantees with respect to its component list or entity list; and adding a node for each of the constructed instances of each element to a directed acyclic graph hierarchy for the artificial reality environment, wherein the hierarchy has a pre-defined type of a root node that corresponds to an entity element, wherein each edge in the hierarchy specifies a parent/child link between elements where the parent element contains the child element; and wherein the entity corresponding to the root node links to a physical boundaries component linked to one or more wall, floor, and/or ceiling entities; wherein the method is performed on a first type of artificial reality device and at least one of the elements is accessed by a second type of artificial reality device with a type different from the first type of artificial reality device. ### Claim 19 (depends on 18) The non-transitory memory of claim 18, wherein the entity corresponding to the root node links to a physical boundaries component linked to a floor entity. ### Claim 20 (depends on 18) The non-transitory memory of claim 18, wherein the at least one element comprises two elements including a first entity and a first component, wherein the first entity specifies multiple component quantity guarantees, each component quantity guarantee defining an amount of components of a particular type the first entity is linked to; and wherein the first component specifies at least one entity quantity guarantee, each entity quantity guarantee defining an amount of entities of a particular type the first component is linked to.