# US 2014/0191927 A1 · Head mount display device providing eye gaze calibration and control method thereof > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2014/0191927 A1 - Document ID: US20140191927A1 - Status: Published application - Kind: A1 - Application number: 13/772,444 - Filed: 2013-02-21 - Published: 2014-07-10 - Earliest claimed: 2013-01-09 - Assignees: LG Electronics Inc. - Applicants: LG Electronics Inc. - Inventors: Eunhyung CHO - Web page: https://patentgrep.com/patent/US20140191927A1 ## CPC - G02B 27/017: {Head mounted} - G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking} - G06F 3/005: {Input arrangements through a video camera} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G02B 2027/0138: {comprising image capture systems, e.g. camera} - G02B 2027/014: {comprising information/image processing systems} - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract An eye gaze calibration method includes sensing an image of surroundings located in front of the HMD device using a camera unit, detecting a reference object from the sensed image of the surroundings using an image processing unit, detecting an eye gaze direction in which a user gazes at the detected reference object using an eye gaze detection unit, and performing eye gaze calibration based on the detected reference object and the detected eye gaze direction using a controller, wherein detecting the reference object includes detecting at least one object contained in the sensed image as a first object, detecting a hand of the user contained in the sensed image as a second object, detecting an event of the second object with respect to the first object, and deciding the first object, on which the event has been detected, as the reference object. ## Claims ### Claim 1 A head mounted display (HMD) device comprising: a camera unit configured to sense an image of surroundings located in front of the HMD device; an image processing unit configured to detect a reference object from the sensed image of the surroundings; an eye gaze detection unit configured to detect an eye gaze direction in which a user gazes at the detected reference object; and a controller configured to perform eye gaze calibration based on the detected reference object and the detected eye gaze direction, wherein the image processing unit is configured to detect at least one object contained in the sensed image of the surroundings as a first object, detect a hand of the user contained in the sensed image of the surroundings as a second object, detect an event of the second object with respect to the first object, and decide the first object, on which the event has been detected, as the reference object. ### Claim 2 (depends on 1) The HMD device according to claim 1, wherein the first object is an external device, the second object is a tip of a finger of the user, and the event of the second object with respect to the first object comprises an operation of controlling the external device. ### Claim 3 (depends on 2) The HMD device according to claim 2, wherein the first object is a keyboard, and the event of the second object with respect to the first object comprises an operation of pressing a key of the keyboard. ### Claim 4 (depends on 1) The HMD device according to claim 1, wherein the controller performs the eye gaze calibration based on a reference point located on the detected reference object, and the reference point is a portion of the first object contacted by the second object to control the first object. ### Claim 5 (depends on 4) The HMD device according to claim 4, wherein in a case in which the reference object is an electronic instrument, the reference point comprises a control button of the electronic instrument, and the event comprises an operation of controlling the control button of the electronic instrument. ### Claim 6 (depends on 5) The HMD device according to claim 5, wherein the control button of the electronic instrument comprises at least one selected from among a power button, volume control buttons, and channel control buttons. ### Claim 7 (depends on 4) The HMD device according to claim 4, wherein in a case in which the reference object is a device comprising a touch sensitive display, the reference point comprises a soft button displayed on the touch sensitive display. ### Claim 8 (depends on 1) The HMD device according to claim 1, wherein in a case in which the first object is a pen, the event of the second object with respect to the first object comprises a writing operation using the pen, and the reference object is a tip of the pen. ### Claim 9 (depends on 1) The HMD device according to claim 1, further comprising: a display unit configured to display a virtual object, wherein the display unit controls a position of the displayed virtual object according to a result of the eye gaze calibration, and the virtual object is a graphical user interface provided to the user by the display unit. ### Claim 10 (depends on 9) The HMD device according to claim 9, wherein the display unit displays the virtual object based on a position of the reference object using an augmented reality method. ### Claim 11 (depends on 9) The HMD device according to claim 9, wherein the display unit is a transparent optical display unit. ### Claim 12 (depends on 9) The HMD device according to claim 9, further comprising: a distance measurement unit configured to measure a first distance from the HMD device to the detected reference object, wherein the HMD device decides depth in which the virtual object is displayed based on the measured first distance. ### Claim 13 (depends on 12) The HMD device according to claim 12, wherein the distance measurement unit measures the first distance using at least one selected from among an ultrasonic sensor, a laser sensor, and an infrared sensor. ### Claim 14 (depends on 12) The HMD device according to claim 12, wherein the display unit provides a left eye and a right eye of the user with a left image and a right image, respectively, and expresses the depth of the virtual object using binocular disparity between the left image and the right image. ### Claim 15 (depends on 1) The HMD device according to claim 1, wherein the controller performs the eye gaze calibration only in a case in which a difference between a direction of the reference object and the eye gaze direction in which the user gazes at the reference object is within an allowable error range. ### Claim 16 (depends on 1) The HMD device according to claim 1, wherein the eye gaze detection unit detects the eye gaze direction based on movement of a pupil of the user. ### Claim 17 (depends on 1) The HMD device according to claim 1, wherein the second object is detected by at least one of a shape and a gesture of the hand of the user contained in the sensed image of the surroundings. ### Claim 18 (depends on 1) The HMD device according to claim 1, wherein the camera unit senses an image having a range corresponding to a visual field of the user with respect to the surroundings. ### Claim 19 An eye gaze calibration method of an HMD device comprising: sensing an image of surroundings located in front of the HMD device using a camera unit; detecting a reference object from the sensed image of the surroundings using an image processing unit; detecting an eye gaze direction in which a user gazes at the detected reference object using an eye gaze detection unit; and performing eye gaze calibration based on the detected reference object and the detected eye gaze direction using a controller, wherein the step of detecting the reference object comprises: detecting at least one object contained in the sensed image of the surroundings as a first object; detecting a hand of the user contained in the sensed image of the surroundings as a second object; detecting an event of the second object with respect to the first object; and deciding the first object, on which the event has been detected, as the reference object. ### Claim 20 An eye gaze calibration method of an HMD device comprising: sensing an image of surroundings located in front of the HMD device using a camera unit; detecting a reference object from the sensed image of the surroundings using an image processing unit; detecting an eye gaze direction in which a user gazes at the detected reference object using an eye gaze detection unit; and performing eye gaze calibration based on the detected reference object and the detected eye gaze direction using a controller, wherein the step of detecting the reference object comprises: detecting a hand of the user contained in the sensed image of the surroundings as a first object; detecting at least one object contained in the sensed image of the surroundings as a second object; detecting an event of the first object with respect to the second object; and deciding the second object, on which the event has been detected, as the reference object. ### Claim 21 An HMD device comprising: a camera unit configured to sense an image of surroundings located in front of the HMD device; an image processing unit configured to detect a reference object from the sensed image of the surroundings; an eye gaze detection unit configured to detect an eye gaze direction in which a user gazes at the detected reference object; and a controller configured to perform eye gaze calibration based on the detected reference object and the detected eye gaze direction, wherein the image processing unit is configured to detect a hand of the user contained in the sensed image of the surroundings, detect a gesture performed by the hand of the user, and decide the detected hand of the user together with the gesture as the reference object.