# US 11,307,647 B2 · Artificial reality triggered by physical object > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,307,647 B2 - Document ID: US11307647B2 - Status: Granted - Kind: B2 - Application number: 16/567,563 - Filed: 2019-09-11 - Published: 2021-03-11 - Granted: 2022-04-19 - Earliest claimed: 2019-09-11 - Assignees: Facebook Technologies, LLC - Applicants: Facebook Technologies, LLC - Inventors: Lars Anders Bond; Christopher John Ocampo; Stefano Zanetti - Web page: https://patentgrep.com/patent/US11307647B2 ## CPC - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G02B 27/0172: {characterised by optical features} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors - G06T 11/00: Two-dimensional [2D] image generation - G02B 2027/0138: {comprising image capture systems, e.g. camera} - G02B 2027/014: {comprising information/image processing systems} - G02B 2027/0178: {Eyeglass type (eyeglass details G02C)} - G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract This disclosure describes an artificial reality system that presents artificial reality content or artificial reality effects based on, or in response to, interactions with one or more physical objects within a physical environment. In one example, this disclosure describes a method that includes determining that a user has performed a trigger action on a trigger object; responsive to determining that the user has performed the trigger action, generating artificial reality content; presenting the artificial reality content on a display associated with a head-mounted display; determining that the user has performed a de-trigger action; and responsive to determining that the user has performed the de-trigger action, discontinuing presentation of the artificial reality content. ## Claims ### Claim 1 A system comprising: an image capture system configured to capture image data representative of a physical environment including a plurality of physical objects; a head-mounted display (HMD) worn by a user; a mapping engine configured to determine, based on the image data, a map of the physical environment including position information about the HMD and a trigger object included in the plurality of physical objects, wherein the trigger object is an object capable of supporting the user in a sitting position; and an application engine configured to: determine that the user has performed a trigger action on the trigger object, wherein the trigger action is the user sitting on the trigger object, responsive to determining that the user has performed the trigger action, generate an artificial reality environment by overlaying artificial reality content on an image of the physical environment, present the artificial reality environment on a display associated with the HMD, determine that the user has performed a de-trigger action, and responsive to determining that the user has performed the de-trigger action, cease presentation of the artificial reality environment. ### Claim 2 (depends on 1) The system of claim 1, wherein the de-trigger action is the user standing up after sitting on the trigger object. ### Claim 3 (depends on 1) The system of claim 1, wherein the image is a first image, and wherein to cease presentation of the artificial reality environment, the application engine is further configured to: present a second image of the physical environment. ### Claim 4 (depends on 1) The system of claim 1, wherein the artificial reality environment is a triggered artificial reality environment, wherein the application engine is further configured to: present, prior to determining that the user has performed a trigger action on the trigger object, an idle artificial reality environment. ### Claim 5 (depends on 4) The system of claim 4, wherein to cease presentation of the triggered artificial reality environment, the application engine is further configured to: present the idle artificial reality environment. ### Claim 6 (depends on 4) The system of claim 4, wherein to generate the triggered artificial reality environment, the application engine is further configured to perform a mode change responsive to determining that the user has performed the trigger action; and wherein to cease presentation of the triggered artificial reality environment, the application engine is further configured to perform another mode change responsive to determining that the user has performed the de-trigger action. ### Claim 7 (depends on 1) The system of claim 1, wherein the application engine is further configured to: detect input identifying the trigger object, wherein the trigger object is an arbitrary physical object within the physical environment. ### Claim 8 (depends on 1) The system of claim 1, wherein the plurality of physical objects includes a device capable of accepting input, and wherein to generate the artificial reality environment, the application engine is further configured to: perform an action on an attribute of the physical environment, including one or more of: changing lighting in the physical environment, presenting music in the physical environment, or providing input to the device. ### Claim 9 (depends on 1) The system of claim 1, wherein to generate the artificial reality environment, the application engine is further configured to: generate the artificial reality environment based on information about at least one of: attributes of the physical environment, time, date, trigger object type, trigger action type, or environmental attributes. ### Claim 10 (depends on 1) The system of claim 1, wherein to generate the artificial reality environment, the application engine is further configured to: identify the user; and generate the artificial reality environment based on information about the user. ### Claim 11 (depends on 10) The system of claim 10, wherein to identify the user, the application engine is further configured to: identify the user based on biometric information detected by the HMD. ### Claim 12 (depends on 10) The system of claim 10, wherein to identify the user, the application engine is further configured to: identify the user based on input detected by the HMD in response to a prompt presented by the HMD. ### Claim 13 (depends on 10) The system of claim 10, wherein the information about the user is profile information associated with the user. ### Claim 14 (depends on 10) The system of claim 10, wherein the information about the user is based on an object possessed by the user. ### Claim 15 (depends on 10) The system of claim 10, wherein the information about the user is information derived from a calendar maintained by the user. ### Claim 16 A method comprising: capturing, by an artificial reality system, image data representative of a physical environment including a plurality of physical objects including a head-mounted display (HMD) worn by a user; determining, by the artificial reality system and based on the image data, a map of the physical environment including position information about the HMD and a trigger object included in the plurality of physical objects, wherein the trigger object is an object capable of supporting the user in a sitting position; determining, by the artificial reality system, that a user has performed a trigger action on the trigger object, wherein the trigger action is the user sitting on the trigger object; responsive to determining that the user has performed the trigger action, generating artificial reality content by overlaying artificial reality content on an image of the physical environment; presenting, by the artificial reality system, the artificial reality content on a display associated with the HMD; determining, by the artificial reality system, that the user has performed a de-trigger action; and responsive to determining that the user has performed the de-trigger action, discontinuing presentation of the artificial reality content. ### Claim 17 (depends on 16) The method of claim 16, wherein the artificial reality content is triggered artificial reality content, and wherein the method further comprises: presenting, by the artificial reality system and prior to determining that the user has performed a trigger action on the trigger object, idle artificial reality content. ### Claim 18 A non-transitory computer-readable medium comprising instructions for causing processing circuitry of an artificial reality system to perform operations comprising: capturing image data representative of a physical environment including a plurality of physical objects including a head-mounted display (HMD) worn by a user; determining, based on the image data, a map of the physical environment including position information about the HMD and a trigger object included in the plurality of physical objects, wherein the trigger object is an object capable of supporting the user in a sitting position; determining that a user has performed a trigger action on the trigger object, wherein the trigger action is the user sitting on the trigger object; responsive to determining that the user has performed the trigger action, generating artificial reality content by overlaying artificial reality content on an image of the physical environment; presenting the artificial reality content on a display associated with the HMD; determining that the user has performed a de-trigger action; and responsive to determining that the user has performed the de-trigger action, discontinuing presentation of the artificial reality content.