# US 2020/0401804 A1 · Virtual content positioned based on detected object > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0401804 A1 - Document ID: US20200401804A1 - Status: Published application - Kind: A1 - Application number: 16/881,691 - Filed: 2020-05-22 - Published: 2020-12-24 - Earliest claimed: 2019-06-19 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Anselm Grundhoefer; Rahul Nair - Web page: https://patentgrep.com/patent/US20200401804A1 ## CPC - G06T 7/73: using feature-based methods - G06V 20/20: in augmented reality scenes - G02B 27/0172: {characterised by optical features} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/147: using display panels - G06T 7/579: from motion - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06V 40/107: {Static hand or arm} - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - G09G 5/32: with means for controlling the display position {(see provisionally G09G5/42)} - G06T 2207/30196: Human being; Person - G06T 2207/30204: Marker - G09G 2340/0464: Positioning - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00) ## Abstract Various implementations disclosed herein include devices, systems, and methods that use an object as a background for virtual content. Some implementations involve obtaining an image of a physical environment. A location of a surface of an object is detected based on the image. A virtual content location to display virtual content is determined, where the virtual content location corresponds to the location of the surface of the object. Then, a view of the physical environment and virtual content displayed at the virtual content location is provided. ## Claims ### Claim 1 A method, comprising: at an electronic device having a processor: obtaining an image of a physical environment; detecting a location of a surface of an object based on the image, wherein the object is a portion of a body of a user or the object is held by the user; determining a virtual content location to display virtual content, the virtual content location corresponding to the location of the surface of the object; and providing a view of the physical environment and the virtual content positioned at the virtual content location. ### Claim 2 (depends on 1) The method of claim 1, wherein the electronic device is an optical see-through head mounted display. ### Claim 3 (depends on 1) The method of claim 1, wherein providing the view of the physical environment and the virtual content comprises providing a computer-generated-reality (CGR) environment to the user. ### Claim 4 (depends on 1) The method of claim 1, wherein the object is a book or cell phone held by a hand of the user. ### Claim 5 (depends on 1) The method of claim 1, wherein detecting a location of a surface of an object comprises detecting a hand of the user in the image, and wherein providing the view of the physical environment and the virtual content positioned at the virtual content location comprises displaying the virtual content at the virtual content location within an outline of the hand. ### Claim 6 (depends on 5) The method of claim 5, wherein providing the view of the physical environment and the virtual content positioned at the virtual content location comprises displaying the virtual content using a color selected based on a color of the hand, ambient light, or a color of the wearable display. ### Claim 7 (depends on 5) The method of claim 5, further comprising: increasing a size of the virtual content in accordance with a determination that the hand is moved closer to the electronic device; or decreasing the size of the virtual content in accordance with a determination that the hand is moved farther away from the electronic device. ### Claim 8 (depends on 5) The method of claim 5, wherein displaying the virtual content comprises initializing the virtual content display based on detecting a location of a surface of an object. ### Claim 9 (depends on 1) The method of claim 1, wherein detecting a location of a surface of an object comprises: detecting a hand of the user in the image; determining a position of the hand in the physical environment; identifying a plane based on the hand, wherein the plane is substantially orthogonal to a gaze direction toward the hand; and setting the virtual content location on the plane, wherein providing the view of the physical environment and the virtual content positioned at the virtual content location comprises displaying the virtual content at the virtual content location in front of the hand. ### Claim 10 (depends on 1) The method of claim 1, wherein the virtual content location is a portion of the surface of the object satisfying a predetermined criteria. ### Claim 11 (depends on 1) The method of claim 1, further comprising: detecting a first gesture as a first trigger to initiate provision of the virtual content at the virtual content location; and detecting a second gesture is a second trigger to terminate provision of the virtual content at the virtual content location. ### Claim 12 (depends on 1) The method of claim 1, wherein detecting the location of the surface of the object comprises tracking the surface of the object with respect to the electronic device. ### Claim 13 (depends on 12) The method of claim 12, wherein tracking the surface of the object with respect to the electronic device comprises: using sensors to implement simultaneous localization and mapping of the object; using a depth camera and fitting a preset model of the object to the detected depth; using the depth camera to estimate the object shape, orientation, and position; using 2D cameras stereoscopically to estimate the object shape, orientation, and position; using 2D cameras monoscopically to estimate the object shape, orientation, and position; using a structured light camera to estimate the object shape, orientation, and position; using a thermal camera to estimate the object shape, orientation, and position; or using an event camera to estimate the object shape, orientation, and position. ### Claim 14 (depends on 1) The method of claim 1, wherein the virtual content comprises text, image, or video content. ### Claim 15 (depends on 1) The method of claim 1, wherein the virtual content is selected based on the physical environment. ### Claim 16 (depends on 1) The method of claim 1, wherein providing a view of the physical environment and the virtual content positioned at the virtual content location comprises displaying the virtual content using a color selected based on a color of the object, ambient light, or a color of the wearable display. ### Claim 17 (depends on 1) The method of claim 1, wherein the virtual content is augmented content from the image. ### Claim 18 A system comprising: a non-transitory computer-readable storage medium; and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the system to perform operations comprising: obtaining an image of a physical environment; detecting a location of a surface of an object based on the image, wherein the object is a portion of a body of a user or the object is held by the user determining a virtual content location to display virtual content, the virtual content location corresponding to the location of the surface of the object; and providing a view of the physical environment and the virtual content positioned at the virtual content location ### Claim 19 A non-transitory computer-readable storage medium, storing program instructions computer-executable on a computer to perform operations comprising: obtaining an image of a physical environment; detecting a location of a surface of an object based on the image, wherein the object is a portion of a body of a user or the object is held by the user determining a virtual content location to display virtual content, the virtual content location corresponding to the location of the surface of the object; and providing a view of the physical environment and the virtual content positioned at the virtual content location.