# US 12,382,009 B2 · System and method for corrected video-see-through for head mounted displays > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,382,009 B2 - Document ID: US12382009B2 - Status: Granted - Kind: B2 - Application number: 18/131,182 - Filed: 2023-04-05 - Published: 2023-07-27 - Granted: 2025-08-05 - Earliest claimed: 2019-07-09 - Assignees: Interaptix Inc. - Applicants: Interaptix Inc. - Inventors: Dae Hyun Lee - Web page: https://patentgrep.com/patent/US12382009B2 ## CPC - H04N 13/344: with head-mounted left-right displays - H04N 13/117: the virtual viewpoint locations being selected by the viewers or determined by viewer tracking - H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) - H04N 13/383: for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A head mounted display system with video-see-through (VST) is taught. The system and method process video images captured by at least two forward facing video cameras mounted to the headset to produce generated images whose viewpoints correspond to the viewpoint of the user if the user was not wearing the display system. By generating VST images which have viewpoints corresponding to the user's viewpoint, errors in sizing, distances and positions of objects in the VST images are prevented. ## Claims ### Claim 1 A head mounted display system comprising: a display capable of being worn by a user in front of their eyes and displaying images to the user; at least eight video cameras at fixed locations relative to the display and having respective fields of view that are narrower than an expected field of view of the user and that cover a combined field of view angle of at least 180 degrees horizontally and vertically, the at least eight video cameras operable to capture video images from the respective fields of view and configured to detect distances between the display and objects located in the respective fields of view; a computational device operable to: obtain, via at least one of the at least eight video cameras, the distances between the display and the objects located in the respective fields of view; receive user-selected virtual locations of the pupils of the user, the user-selected virtual locations being different to actual locations of the pupils of the user; determine, based on the respective fields of view of the at least eight video cameras and the distances, a transformation to transform captured video images from each of the at least eight video cameras to the user-selected virtual locations of the pupils of the user; and apply the transformation to the captured video images to generate images for providing video-see-through on the display, the generated images corresponding to a virtual viewpoint from the user-selected virtual locations of the pupils of the user to provide the user with a different real-world view from an actual real-world view that would be experienced by the user if not wearing the display, including displaying the objects located in the respective fields of view of the at least eight video cameras at depths corresponding to distances of the objects from the user-selected virtual locations of the pupils of the user, the depths computed based on the obtained distances and the respective fields of view of the at least eight video cameras relative to the user-selected virtual locations of the pupils of the user. ### Claim 2 (depends on 1) The head mounted display system according to claim 1 wherein the computational device is operable to compute the distances between the display and the objects located in the respective fields of view of the at least eight video cameras from the captured video images. ### Claim 3 (depends on 2) The head mounted display system according to claim 2, wherein the computational device is operable to generate an image for each pupil of the user, each generated image corresponding to the virtual viewpoint of the respective pupil of the user and each generated image is displayed to the respective pupil of the user providing the user with a stereoscopic image. ### Claim 4 (depends on 1) The head mounted display system according to claim 1 wherein the computational device is operable to generate an image for each pupil of the user, each generated image corresponding to the virtual viewpoint of the respective pupil of the user and each generated image is displayed to the respective pupil of the user providing the user with a stereoscopic image. ### Claim 5 (depends on 1) The head mounted display system according to claim 1 wherein the computational device is mounted to the display. ### Claim 6 (depends on 1) The head mounted display system according to claim 1, wherein the computational device is further operable to obtain an inter-pupil distance between the pupils of the user and determine the transformation further based on the inter-pupil distance. ### Claim 7 (depends on 1) The head mounted display system according to claim 1 wherein the computational device is connected to the display by a wire tether. ### Claim 8 (depends on 1) The head mounted display system according to claim 1 wherein the computational device is wirelessly connected to the display. ### Claim 9 (depends on 1) The head mounted display system of claim 1, wherein the computational device is operable to determine the fixed respective fields of view of the at least eight video cameras relative to the pupils of the user based on an inter-pupil distance and an eye-to-display distance. ### Claim 10 A method of operating a head mounted display worn by a user in front of their eyes, the head mounted display having at least eight video cameras operable to capture video images, the at least eight video cameras having respective fields of view that are narrower than an expected field of view of the user and that cover a combined field of view angle of at least 180 degrees horizontally and vertically and configured to detect distances between the display and objects, the method comprising the steps of: determining respective fields of view of the at least eight video cameras relative to each pupil of the eyes of the user; obtaining video images captured by the at least eight video cameras; computing distances between the display and objects located in the respective fields of view of the at least eight video cameras based on the video images captured by the at least eight video cameras; receiving user-selected virtual locations of the pupils of the user, the user-selected virtual locations being different to actual locations of the pupils of the user; determining, based on the respective fields of view of each of the at least eight video cameras relative to the pupil of each eye of the user, and the computed distances, a transformation to transform the captured video images from each of the at least eight video cameras to the user-selected virtual locations of the pupils of the user; applying the transformation to the captured video images to render generated images corresponding to a virtual viewpoint from the user-selected virtual locations of the pupils of the user to provide the user with a different real-world view from an actual real-world view that would be experienced by the user if not wearing the display wherein the generated images display the objects at depths corresponding to distances of the objects from the user-selected virtual locations of the pupils of the user, the depths computed based on the computed distances and the respective fields of view of the at least eight video cameras relative to the user-selected virtual locations of the pupils of the user; and providing video-see-through to the user by displaying the generated images to the user on the head mounted display. ### Claim 11 (depends on 10) The method of claim 10, further comprising processing the captured video images to render a respective generated image for each pupil of the user, each respective generated image corresponding to the virtual viewpoint of the respective pupil of the user. ### Claim 12 (depends on 10) The method of claim 10, further comprising obtaining an inter-pupil distance between the pupils of the user and determining the transformation further based on the inter-pupil distance. ### Claim 13 (depends on 1) The head mounted display system of claim 1 wherein the generated images appear enlarged or magnified with respect to the actual real-world view. ### Claim 14 (depends on 1) The head mounted display system of claim 1 wherein the virtual viewpoint defines the pupils of the eyes of the user as being located to one side of, above, or below the user. ### Claim 15 (depends on 10) The method of claim 10 wherein the generated images appear enlarged or magnified with respect to the actual real-world view. ### Claim 16 (depends on 10) The method of claim 10 wherein the virtual viewpoint defines the pupils of the eyes of the user as being located to one side of, above, or below the user.