# US 2021/0240260 A1 · Method and system for gaze-based control of mixed reality content > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0240260 A1 - Document ID: US20210240260A1 - Status: Published application - Kind: A1 - Application number: 17/048,985 - Filed: 2019-04-12 - Published: 2021-08-05 - Earliest claimed: 2018-04-20 - Assignees: PCMS Holdings, Inc. - Applicants: PCMS Holdings, Inc. - Inventors: Jussi Ronkainen; Jani Mantyjarvi - Web page: https://patentgrep.com/patent/US20210240260A1 ## CPC - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking} - G02B 27/017: {Head mounted} - G02B 27/0172: {characterised by optical features} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/012: {Head tracking input arrangements} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - G02B 2027/014: {comprising information/image processing systems} - G06T 2219/2004: Aligning objects, relative positioning of parts - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Systems and methods are presented for discovering and positioning content into augmented reality space. A method includes forming a three-dimensional (3D) map of surroundings of a user of an augmented reality (AR) head mounted display (HMD); determining a depth-wise location of a gaze point of a user based on eye gaze direction and eye vergence; determining a visual guidance line pathway in the 3D map; guiding an action of the user along the visual guidance line pathway at one or more identified focal points; and rendering a mixed reality (MR) object along the visual guidance line pathway at a location corresponding to a direction of the user's gaze. ## Claims ### Claim 1 A method comprising: forming a three-dimensional (3D) map of surroundings of a user of an augmented reality (AR) head mounted display (HMD); selecting a mixed reality (MR) object for placement or repositioning within the 3D map; displaying a visual guidance line pathway in the 3D map; detecting a gaze of the user at one or more positions along the visual guidance line pathway; and rendering the MR object at rendering positions along the visual guidance line pathway corresponding to the one or more detected positions of the gaze of the user. ### Claim at position 2 2-7. (canceled) ### Claim 8 (depends on 1) The method of claim 1, further comprising: determining the visual guidance line pathway in the 3D map; and determining the gaze of the user at the one or more positions along the visual guidance line pathway, wherein determining the gaze of the user at the one or more positions along the visual guidance line pathway comprises any one of: determining the one or more positions by gaze tracking of the user, determining the one or more positions according to one or more hardware restrictions of the AR HMD, determining the one or more positions based on a number of visually correct focal planes displayable by the AR HMD, determining the one or more positions based on a determined accuracy of gaze tracking available on the AR HMD, determining the one or more positions according to a distance of the MR object, or determining the one or more positions by a movement of the user. ### Claim at position 4 9-13. (canceled) ### Claim 14 (depends on 8) The method of claim 8 wherein determining the visual guidance line pathway in the 3D map comprises any one of: determining the visual guidance line pathway based on the depth-wise location of the gaze point of the user and available space in the 3D map, determining the visual guidance line pathway by forming the visual guidance line pathway to avoid one or more identified objects in the 3D map, determining the visual guidance line pathway by altering the visual guidance line pathway according to pivot points determined by a user gaze, or determining the visual guidance line pathway by altering the visual guidance line pathway according to movements of the user, the movements of the user comprising one or more of a head tilt, a head pitch, a head yaw, and a gesture. ### Claim at position 6 15-28. (canceled) ### Claim 29 A system comprising: a processor; and a non-transitory computer-readable storage medium storing instructions operative, when executed by the processor, to cause the system to: form a three-dimensional (3D) map of surroundings of a user of an augmented reality (AR) head mounted display (HMD); select a mixed reality (MR) object for placement or repositioning within the 3D map; a visual guidance line pathway in the 3D map; detecting a gaze of the user at one or more positions along the visual guidance line pathway; and render the content enhanced MR object along the visual guidance line pathway at a location corresponding to a direction of a gaze of the user. ### Claim 30 (canceled) ### Claim 31 (depends on 29) The system of claim 29 wherein displaying the visual guidance line pathway in the 3D map comprises: displaying the one or more identified focal points as a plurality of focal plane indicators at a plurality of depths within the 3D map. ### Claim at position 10 32-37. (canceled) ### Claim 38 (depends on 1) The method of claim 1, wherein the visual guidance line pathway is a straight line occupying multiple depths relative to the user. ### Claim 39 (depends on 1) The method of claim 1, wherein the visual guidance line pathway is a curved line displayed in 3D space. ### Claim 40 (depends on 1) The method of claim 1, wherein the visual guidance line pathway is displayed with multiple point markers, the point markers identifying the rendering positions at which the virtual object may be rendered. ### Claim 41 (depends on 1) The method of claim 1, wherein the multiple point markers correspond to discrete focal planes supported by an optical system of the HMD. ### Claim 42 (depends on 1) The method of claim 1, further comprising determining the visual guidance line pathway to avoid objects in the 3D map. ### Claim 43 (depends on 1) The method of claim 1, further comprising determining the visual guidance line pathway to avoid additional MR objects displayed for the user by the HMD. ### Claim 44 (depends on 1) The method of claim 1, wherein selecting the MR object comprises: identifying an object in a view of the user for which content enhancement is available; and selecting an available content enhancement object for the identified object to be the selected MR object. ### Claim 45 (depends on 1) The method of claim 1, further comprising: detecting a gaze point of the user, wherein selecting the MR object is based on a detected gaze point of the user. ### Claim 46 (depends on 1) The method of claim 1, wherein rendering the MR object at rendering positions along the visual guidance line changes the depth at which the virtual object is rendered relative to the user. ### Claim 47 (depends on 1) The method of claim 1, further comprising: displaying a visual cue that a content enhancement is available to the user for an object corresponding to the MR object, wherein selecting the MR object comprises activating content enhancement for the object. ### Claim 48 (depends on 1) The method of claim 1, wherein rendering the MR object further comprises: rendering one or more MR objects along the visual guidance line pathway at locations corresponding to a direction of the gaze of the user, while avoiding one or more preexisting objects in the 3D map of the surroundings, wherein the visual guidance line pathway is placed in determined available space within the 3D map of the surroundings, and wherein the one or more preexisting objects comprise at least one of a real-world object or an existing MR object. ### Claim 49 (depends on 29) The system of claim 29, wherein the instructions are further operative, when executed by the processor, to cause the system to: display a visual cue that a content enhancement is available to the user for the object, wherein activating the content enhanced MR object comprises activating the content enhancement for the object. ### Claim 50 (depends on 29) The system of claim 29, wherein the instructions are further operative, when executed by the processor, to cause the system to: prior to activating, display the object in the 3D map along with the visual cue. ### Claim 51 (depends on 29) The system of claims 29, wherein the object comprises one of a virtual object, a pre-existing real world object, or an MR object. ### Claim 52 (depends on 49) The system of claim 49, wherein activating the content enhanced MR object further comprises activating the content enhancement for the object according to the user input to the AR HMD with respect to the visual cue.