# US 12,111,463 B2 · Head-mounted display apparatus and operating method thereof > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,111,463 B2 - Document ID: US12111463B2 - Status: Granted - Kind: B2 - Application number: 18/451,428 - Filed: 2023-08-17 - Published: 2024-06-27 - Granted: 2024-10-08 - Earliest claimed: 2022-12-22 - Assignees: Samsung Electronics Co., Ltd. - Applicants: Samsung Electronics Co., Ltd. - Inventors: Pawel Kies - Web page: https://patentgrep.com/patent/US12111463B2 ## CPC - G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking} - G02B 27/0101: {characterised by optical features (G02B27/0172 takes precedence)} - G02B 27/017: {Head mounted} - G02B 27/0179: {Display position adjusting means not related to the information to be displayed} - G02B 2027/0138: {comprising image capture systems, e.g. camera} - G02B 2027/014: {comprising information/image processing systems} - G02B 2027/0185: {Displaying image at variable distance} - G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye} - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS ## Abstract A head-mounted display apparatus is provided. The head-mounted display apparatus includes at least one camera, at least one sensor, and at least one processor operatively connected to the at least one camera and the at least one sensor. The at least one processor may be configured to acquire information about head pose and head-based gaze direction of a user, based on data input from the at least one sensor and multiple key points input from the at least one camera according to recognizing rotation of a user's head, estimate gaze lines of the user, based on the acquired information about head pose and head-based gaze direction, and estimate a distance to a target point, based on the gaze lines of the user, according to identifying that vestibulo-ocular reflex is activated based on the gaze lines of the user. ## Claims ### Claim 1 A head-mounted display apparatus comprising: at least one camera; at least one sensor; and at least one processor operatively connected to the at least one camera and the at least one sensor, wherein the at least one processor is configured to: acquire information about head pose and head-based gaze direction of a user, based on data input from the at least one sensor and multiple key points input from the at least one camera according to recognizing rotation of a user's head, estimate gaze lines of the user, based on the acquired information about head pose and head-based gaze direction, and estimate a distance to a target point, based on the gaze lines of the user, according to identifying that vestibulo-ocular reflex is activated based on the gaze lines of the user. ### Claim 2 (depends on 1) The head-mounted display apparatus of claim 1, wherein the at least one processor is further configured to: receive the data according to rotation of the user's head from the at least one sensor, and receive the multiple key points according to user gaze tracking from the at least one camera. ### Claim 3 (depends on 1) The head-mounted display apparatus of claim 1, wherein the at least one sensor comprises a magnetic sensor or a gyro sensor. ### Claim 4 (depends on 1) The head-mounted display apparatus of claim 1, wherein the at least one camera comprises a scene camera configured to observe a user's view. ### Claim 5 (depends on 1) The head-mounted display apparatus of claim 1, wherein the at least one processor is further configured to determine that the vestibulo-ocular reflex is activated, in case that at least one of eyelids of the user is open and at least some gaze lines among gaze lines of the user intersect within a predetermined range from an identical point. ### Claim 6 (depends on 1) The head-mounted display apparatus of claim 1, wherein the at least one processor is further configured to: in case that at least one of eyelids of the user is not open and at least some gaze lines among the gaze lines of the user do not intersect within a predetermined range from an identical point, determine that the vestibulo-ocular reflex is not activated, and re-acquire the information about head pose and head-based gaze direction of the user. ### Claim 7 (depends on 1) The head-mounted display apparatus of claim 1, further comprising at least one display or a speaker, wherein the at least one processor is configured to report the estimated distance to the target point to the user by using the at least one display or the speaker. ### Claim 8 (depends on 1) The head-mounted display apparatus of claim 1, wherein the at least one processor is further configured to receive a user input for estimating the distance to the target point through a user interface. ### Claim 9 (depends on 1) The head-mounted display apparatus of claim 1, wherein the at least one processor is further configured to repeatedly estimate the distance to the target point a predetermined number of times, based on the number of times the information about head pose and head-based gaze direction is acquired and calculation performance of the at least one processor. ### Claim 10 (depends on 1) The head-mounted display apparatus of claim 1, wherein the gaze lines of the user correspond to cyclopean gaze lines indicating a center of gaze lines of two eyeballs of the user. ### Claim 11 A method of operating a head-mounted display apparatus comprising at least one camera, at least one sensor, and at least one processor, the method comprising: acquiring information about head pose and head-based gaze direction of a user, based on data input from the at least one sensor and multiple key points input from the at least one camera according to recognizing rotation of a user's head; estimating gaze lines of the user, based on the acquired information about head pose and head-based gaze direction; identifying that vestibulo-ocular reflex is activated based on the gaze lines of the user; and estimating a distance to a target point, based on the gaze lines of the user. ### Claim 12 (depends on 11) The method of claim 11, wherein the acquiring of the head pose and head-based gaze direction information of the user comprises: receiving the data according to rotation of the user's head from the at least one sensor; and receiving the multiple key points according to user gaze tracking from the at least one camera. ### Claim 13 (depends on 11) The method of claim 11, wherein the at least one sensor comprises a magnetic sensor or a gyro sensor. ### Claim 14 (depends on 11) The method of claim 11, wherein the at least one camera comprises a scene camera configured to observe a user's view. ### Claim 15 (depends on 11) The method of claim 11, wherein the identifying that vestibulo-ocular reflex is activated based on the gaze lines of the user comprises determining that the vestibulo-ocular reflex is activated, in case that at least one of eyelids of the user is open, and at least some gaze lines among gaze lines of the user intersect within a predetermined range from an identical point. ### Claim 16 (depends on 11) The method of claim 11, further comprising, in case that at least one of eyelids of the user is not open and at least some gaze lines among the gaze lines of the user do not intersect within a predetermined range from an identical point, determining that the vestibulo-ocular reflex is not activated and re-acquiring the information about head pose and head-based gaze direction of the user. ### Claim 17 (depends on 11) The method of claim 11, further comprising reporting the estimated distance to the target point to the user by using at least one display or speaker. ### Claim 18 (depends on 11) The method of claim 11, further comprising receiving a user input for estimating the distance to the target point through a user interface. ### Claim 19 (depends on 11) The method of claim 11, wherein the estimating of the distance to the target point is repeatedly perform a predetermined number of times, based on the number of times the information about head pose and head-based gaze direction is acquired and calculation performance of the at least one processor. ### Claim 20 (depends on 11) The method of claim 11, wherein the gaze lines of the user correspond to cyclopean gaze lines indicating a center of gaze lines of two eyeballs of the user.