# US 2019/0377191 A1 · Head-mountable display device and method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0377191 A1 - Document ID: US20190377191A1 - Status: Published application - Kind: A1 - Application number: 16/432,016 - Filed: 2019-06-05 - Published: 2019-12-12 - Earliest claimed: 2018-06-08 - Assignees: Sony Interactive Entertainment Inc. - Applicants: Sony Interactive Entertainment Inc. - Inventors: Colin Jonathan Hughes; Oliver Hume; Patrick John Connor - Web page: https://patentgrep.com/patent/US20190377191A1 ## CPC - G02B 27/017: {Head mounted} - G02B 27/0176: {characterised by mechanical features} - G01B 17/00: Measuring arrangements characterised by the use of infrasonic, sonic or ultrasonic vibrations - G02B 27/0101: {characterised by optical features (G02B27/0172 takes precedence)} - G02B 27/0149: {characterised by mechanical features (G02B27/0176 takes precedence)} - G02B 27/0172: {characterised by optical features} - G06V 40/19: Sensors therefor - G06V 40/197: {Matching; Classification} - G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking} - G02B 2027/0136: {with a single image source for both eyes} - G02B 2027/0161: {characterised by the relative positioning of the constitutive elements} - G02B 2027/0178: {Eyeglass type (eyeglass details G02C)} - G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye} - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A head-mountable display device includes: a display element observable by a user; an image generator operable to generate an image for display by the display element; a plurality of ultrasound transducers, the ultrasound transducers being operable and arranged to emit ultrasound signals towards at least a first eye of the user when the head-mountable display is being worn by the user; one or more sensors operable and arranged to detect reflections of the emitted ultrasound signals; an eye imaging unit operable to generate a representation of the user's eye based on the ultrasound signals received at the one or more sensors; and an eye position detector configured to detect the position of the eye relative to the position of the head-mountable display device based on the representation, the eye position detector being configured to detect whether the position of the eye is offset from a desired position. ## Claims ### Claim 1 A head-mountable display device comprising: a display element observable by a user; an image generator operable to generate an image for display by the display element; a plurality of ultrasound transducers, the ultrasound transducers being operable and arranged to emit ultrasound signals towards at least a first eye of the user when the head-mountable display is being worn by the user; one or more sensors operable and arranged to detect reflections of the emitted ultrasound signals; an eye imaging unit operable to generate a representation of the user's eye based on the ultrasound signals received at the one or more sensors; and an eye position detector configured to detect the position of the eye relative to the position of the head-mountable display device based on the representation, the eye position detector being configured to detect whether the position of the eye is offset from a desired position. ### Claim 2 (depends on 1) A head-mountable display device according to claim 1, comprising: a lens for focussing the image generated by the image generator onto the retina of the user's eye; and wherein the eye position detector is configured to detect whether the eye is offset from the desired position by detecting whether the pupil of the eye is in alignment with the centre of the lens. ### Claim 3 (depends on 1) A head-mountable display device according to claim 1, wherein the eye position detector is configured to detect a gaze-direction of the user, based on the representation generated by the eye imaging unit. ### Claim 4 (depends on 3) A head-mountable display device according to claim 3, wherein the image generator is configured to generate a foveated image; and wherein the image generator is configured to determine a region of the image that is outside the zone gazed by the user's fovea, based on the detected gaze-direction. ### Claim 5 (depends on 1) A head-mountable display device according to claim 1, wherein the head-mountable display device comprises a gaze-direction detector, the gaze direction detector comprising at least one of a camera, hot mirror and a plurality of light sources; and wherein the gaze direction detector is operable to control an orientation of at least one of the camera, hot mirror and light sources, based on the position of the user's eye detected by the eye position detector. ### Claim 6 (depends on 1) A head-mountable display device according to claim 1, wherein at least some of the ultrasound transducers form part of a phased array probe. ### Claim 7 (depends on 1) A head-mountable display device according to claim 1, wherein the sensor comprises the ultrasound transducers. ### Claim 8 (depends on 1) A head-mountable display device according to claim 1, wherein the eye position detector is configured to determine the eye relief of the head-mountable display device, based on the position of the eye detected by the eye position detector. ### Claim 9 (depends on 1) A head-mountable display device according to claim 1, wherein the eye imaging unit is operable to generate a representation of each of the user's eyes; and wherein the head-mountable display device comprises an IPD detector configured to determine the interpupillary distance of the user's eyes based on the representations generated by the eye imaging unit. ### Claim 10 (depends on 1) A head-mountable display device according to claim 1, wherein the image generator is operable to modify an image for display based on the position of the eye detected by the eye position detector. ### Claim 11 (depends on 1) A head-mountable display device according to claim 1, comprising at least one of: i. a frame for affixing the head-mountable display device to the head of the user; and ii. a pad arranged to rest against a top portion of the user's head when the head-mountable display device is being worn by the user. ### Claim 12 (depends on 11) A head-mountable display device according to claim 11, wherein one or more ultrasound transducers are located in at least one of the pad and the frame of the head-mountable display device. ### Claim 13 (depends on 11) A head-mountable display device according to claim 11, comprising: a display portion 50 secured to the frame; a scope adjustor for adjusting the distance of the display portion 50 from an eye of the user when the head-mountable display device is being worn by the user; and wherein at least one of the ultrasound transducers is located within the display portion 50 and is arranged so as to be located in front of the user's eye when the head-mountable display device is being worn by the user. ### Claim 14 (depends on 1) A head-mountable display device according to claim 1, comprising: an audio signal generator operable to generate an audio signal based on the detected position of the user's eye, the audio signal indicating whether the user's eye is offset from the desired position; and one or more speakers for outputting the generated audio signal. ### Claim 15 (depends on 1) A head-mountable display device according to claim 1, wherein the image generator is configured to generate a visual indication for display at the display element, the visual indicator indicating whether the position of the user's eye is offset from the desired position. ### Claim 16 (depends on 1) A head-mountable display device according to claim 1, wherein the eye imaging unit is operable to generate a low resolution representation of a portion of the user's skull that includes the user's eye; and wherein the eye position detector comprises a machine learning classifier trained to determine the position of eyes within low resolution representations of eyes. ### Claim 17 (depends on 1) A head-mountable display device according to claim 1, wherein at least some of the ultrasound transducers are arranged so as to emit ultrasound signals towards the orbit of the user's skull. ### Claim 18 A method of determining whether a head-mountable display is being worn correctly by a user, the method comprising: emitting a plurality of ultrasound signals in the direction of at least a first eye of a user wearing the head-mountable display; detecting reflections of the ultrasound signals at one or more sensors; generating a representation of the user's eye based on the reflections of the ultrasound signals received at the one or more sensors; determining the position of the eye relative to the position of the head-mountable display based on the representation; and determining whether the position of the user's eye is offset from a desired position, the desired position corresponding to the position of the eye when the head-mountable display is being worn correctly. ### Claim 19 (depends on 18) A method according to claim 18, comprising determining a gaze direction of the user, based on the generated representation; and generating a foveated image based on the determined gaze direction of the user. ### Claim 20 (depends on 18) A method according to claim 18, wherein emitting a plurality of ultrasound signals in the direction of an eye of a user comprises controlling a time at which each ultrasound transducer of a phased array probe emits a respective pulsed ultrasound signal. ### Claim 21 (depends on 18) A method according to claim 18, comprising determining an eye relief of the head mountable display based on the determined position of the user's eye relative to the head mountable display. ### Claim 22 (depends on 18) A method according to claim 18, comprising outputting an audio signal at one or more speakers of the HMD, if the position of the user's eye is detected as being offset from the desired position. ### Claim 23 (depends on 18) A method according to claim 18, wherein determining whether the position of the user's eye is offset from a desired position comprises determining whether the position of the user's eye corresponds to a position at which the pupil of the user's eye is aligned with a lens within the head mountable display. ### Claim 24 (depends on 18) A method according to claim 18, comprising modifying an image displayed at the head-mountable display, based on the determined position of the user's eye relative to the head mountable display. ### Claim 25 (depends on 18) A method according to claim 18, comprising training a machine learning algorithm to determine the position of a user's eye based on low resolution three-dimensional images of eyes; wherein generating the representation comprises generating a low resolution representation; and wherein detecting the position of the user's eye relative to head mountable display comprises inputting the low resolution image to the trained machine learning algorithm. ### Claim 26 A non-transitory, computer readable recording medium having computer executable instructions stored thereon, which when executed by a computer system, causes the computer system to determine whether a head-mountable display is being worn correctly by a user, by carrying out actions, comprising: emitting a plurality of ultrasound signals in the direction of at least a first eye of a user wearing the head-mountable display; detecting reflections of the ultrasound signals at one or more sensors; generating a representation of the user's eye based on the reflections of the ultrasound signals received at the one or more sensors; determining the position of the eye relative to the position of the head-mountable display based on the representation; and determining whether the position of the user's eye is offset from a desired position, the desired position corresponding to the position of the eye when the head-mountable display is being worn correctly.