# US 10,311,833 B1 · Head-mounted display device and method of operating a display apparatus tracking an object > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,311,833 B1 - Document ID: US10311833B1 - Status: Granted - Kind: B1 - Application number: 15/937,229 - Filed: 2018-03-27 - Granted: 2019-06-04 - Earliest claimed: 2018-03-27 - Assignees: Seiko Epson Corporation - Applicants: Seiko Epson Corporation - Inventors: Tong Qiu; Yang Yang; Hiu Lok Szeto - Web page: https://patentgrep.com/patent/US10311833B1 ## CPC - G09G 5/363: {Graphics controllers} - G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking} - G02B 27/017: {Head mounted} - G02B 27/0179: {Display position adjusting means not related to the information to be displayed} - G06F 3/147: using display panels - G06T 7/74: {involving reference images or patches} - G06T 7/75: {involving models} - G09G 3/003: {to produce spatial visual effects} - G09G 5/003: {Details of a display terminal, the details relating to the control arrangement of the display terminal and to the interfaces thereto (specific for a CRT G09G1/165; for a flat panel G09G3/2092)} - G09G 5/026: {Control of mixing and/or overlay of colours in general (G09G5/022 and G09G5/024 take precedence)} - G09G 5/14: Display of multiple viewports - G02B 2027/0138: {comprising image capture systems, e.g. camera} - G02B 2027/014: {comprising information/image processing systems} - G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye} - G09G 2340/12: Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels - G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A method tracks a pose of an object in a field of view of a camera; derives movement of the display apparatus or movement in the field of view using at least one of: the sensor data and the image data; determines, using the processor, whether or not the derived movement exceeds a first threshold; if the derived movement exceeds the first threshold, continues tracking the pose of the object; and if the movement does not exceed the first threshold: (1) stops tracking the pose of the object and (2) displays, using the display, the image based on a previously derived pose of the object. ## Claims ### Claim 1 A method for operating a display apparatus having a display, a camera, and an inertial sensor, the method comprising: (a) tracking a pose of an object in a field of view of the camera, the tracking including repeatedly: (1) deriving, using a processor or electronic circuit, the pose of the object relative to the display apparatus using at least one of: image data acquired by the camera and sensor data acquired by the inertial sensor, and (2) displaying, using the display, an image based on the derived pose of the object; (b1) deriving, using the processor or electronic circuit, first movement of the display apparatus using the sensor data; (b2) deriving, using the processor or electronic circuit, second movement of the display apparatus or in the field of view using the image data; (c) determining, using the processor or electronic circuit, whether or not the first movement exceeds a first sensor threshold; (d) if the first movement exceeds the first sensor threshold, continuing tracking the pose of the object in step (a); (e) if the first movement does not exceed the first sensor threshold: (f) determining, using the processor or electronic circuit, whether or not the second movement exceeds an image movement threshold; (g) if the second movement exceeds the image movement threshold, continuing tracking the pose of the object in step (a); and (h) if the second movement does not exceed the image movement threshold: (1) stopping tracking the pose of the object in step (a); and (2) displaying, using the display, the image based on a previously derived pose of the object. ### Claim 2 (depends on 1) The method of claim 1, wherein if the first movement is determined to exceed the first sensor threshold after step (h) has been performed, further comprising restarting tracking the pose of the object in step (a). ### Claim 3 (depends on 1) The method of claim 1, further comprising: (i) determining, using the processor or electronic circuit, whether or not the first movement exceeds a second sensor threshold that is higher than the first sensor threshold; (j) if the first movement does not exceed the second sensor threshold, continuing tracking the pose of the object in step (a); and (k) if the first movement exceeds the second sensor threshold, stopping tracking the pose of the object in step (a). ### Claim 4 (depends on 1) The method of claim 1, wherein displaying the image in steps (a) and (h) includes repeatedly rendering the image using a graphics processor, further comprising, if step (e) is satisfied: (i) measuring, using the processor or electronic circuit, a length of time that the first movement does not exceed the first sensor threshold; (j) determining, using the processor or electronic circuit, whether or not the length of time exceeds a time threshold; (k) if the length of time does not exceed the time threshold, continuing to repeatedly render the image using the graphics processor; and (l) if the length of time exceeds the time threshold, stopping rendering of the image and displaying, using the display, a previous rendering of the image. ### Claim 5 (depends on 1) The method of claim 1, further comprising, if step (e) is satisfied: (i) measuring, using the processor or electronic circuit, a length of time that the first movement does not exceed the first sensor threshold; (j) determining, using the processor or electronic circuit, whether or not the length of time exceeds a time threshold; (k) if the length of time does not exceed the time threshold, continuing displaying, using the display, the image based on the previously derived pose of the object; and (l) if the length of time exceeds the time threshold, stopping displaying the image. ### Claim 6 (depends on 1) The method of claim 1, wherein the object is one of: a three-dimensional object and a planar object. ### Claim 7 (depends on 1) The method of claim 1, wherein the deriving in step (a) is performed using only image data. ### Claim 8 (depends on 1) The method of claim 1, wherein the deriving in step (a) is performed using a combination of image data and sensor data. ### Claim 9 A non-transitory storage medium containing program instructions that when executed by a processor, cause the processor to perform a method for operating a display apparatus having a display, a camera, and an inertial sensor, the method comprising: (a) tracking a pose of an object in a field of view of the camera, the tracking including repeatedly: (1) deriving the pose of the object relative to the display apparatus using at least one of: image data acquired by the camera and sensor data acquired by the inertial sensor, and (2) displaying, using the display, an image based on the derived pose of the object; (b1) deriving first movement of the display apparatus using the sensor data; (b2) deriving second movement of the display apparatus or in the field of view using the image data; (c) determining whether or not the first movement exceeds a first sensor threshold; (d) if the first movement exceeds the first sensor threshold, continuing tracking the pose of the object in step (a); (e) if the first movement does not exceed the first sensor threshold: (f) determining whether or not the second movement exceeds an image movement threshold; (g) if the second movement exceeds the image movement threshold, continuing tracking the pose of the object in step (a); and (h) if the second movement does not exceed the image movement threshold: (1) stopping tracking the pose of the object in step (a); and (2) displaying, using the display, the image based on a previously derived pose of the object. ### Claim 10 (depends on 9) The non-transitory storage medium of claim 9, wherein if the first movement is determined to exceed the first sensor threshold after step (h) has been performed, the method further comprises: (i) restarting tracking the pose of the object in step (a). ### Claim 11 (depends on 9) The non-transitory storage medium of claim 9, wherein the method further comprises: (i) determining whether or not the first movement exceeds a second sensor threshold that is higher than the first sensor threshold; (j) if the first movement does not exceed the second sensor threshold, continuing tracking the pose of the object in step (a); and (k) if the first movement exceeds the second sensor threshold, stopping tracking the pose of the object in step (a). ### Claim 12 (depends on 9) The non-transitory storage medium of claim 9, wherein the method further comprises, if step (e) is satisfied: (i) measuring a length of time that the first movement does not exceed the first sensor threshold; (j) determining whether or not the length of time exceeds a time threshold; (k) if the length of time does not exceed the time threshold, continuing displaying, using the display, the image based on the previously derived pose of the object; and (l) if the length of time exceeds the time threshold, stopping displaying the image. ### Claim 13 (depends on 9) The non-transitory storage medium of claim 9, wherein the object is one of: a three-dimensional object and a planar object. ### Claim 14 (depends on 9) The non-transitory storage medium of claim 9, wherein the deriving in step (a) is performed using only image data. ### Claim 15 (depends on 9) The non-transitory storage medium of claim 9, wherein the deriving in step (a) is performed using a combination of image data and sensor data. ### Claim 16 A system comprising: a head-mounted display device, including: a camera that acquires image data; an inertial sensor that acquires sensor data; and a display; and a processor or electronic circuit configured to: (a) track a pose of an object in a field of view of the camera, the tracking including repeatedly: (1) deriving the pose of the object relative to the head-mounted display device using at least one of: the image data and the sensor data, and (2) displaying, using the display, an image based on the derived pose of the object; (b1) derive first movement of the head-mounted display device using the sensor data; (b2) derive second movement of the head-mounted display device or in the field of view using the image data; (c) determine whether or not the first movement exceeds a first sensor threshold; (d) if the first movement exceeds the first sensor threshold, continue tracking the pose of the object in step (a); (e) if the first movement does not exceed the first sensor threshold: (f) determine whether or not the second movement exceeds an image movement threshold; (g) if the second movement exceeds the image movement threshold, continue tracking the pose of the object in step (a); and (h) if the second movement does not exceed the image movement threshold: (1) stop tracking the pose of the object in step (a); and (2) display, using the display, the image based on a previously derived pose of the object. ### Claim 17 (depends on 16) The system of claim 16, further comprising: a graphics processor configured to repeatedly render the image in steps (a) and (h), and display the image using the display, wherein the processor or electronic circuit is further configured to, if step (e) is satisfied: (i) measure a length of time that the first movement does not exceed the first sensor threshold; (j) determine whether or not the length of time exceeds a time threshold; (k) if the length of time does not exceed the time threshold, instruct the graphics processor to continue to repeatedly render the image; and (l) if the length of time exceeds the time threshold, instruct the graphics processor to: (1) stop rendering the image; and (2) display, using the display, a previous rendering of the image.