# US 12,602,803 B2 · Head-mounted display and method for depth prediction > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,803 B2 - Document ID: US12602803B2 - Status: Granted - Kind: B2 - Application number: 18/621,081 - Filed: 2024-03-28 - Published: 2025-10-02 - Granted: 2026-04-14 - Earliest claimed: 2024-03-28 - Assignees: HTC Corporation - Applicants: HTC Corporation - Inventors: Kuan Yu Huang; Chia Ying Lin; Po Shian Liu - Web page: https://patentgrep.com/patent/US12602803B2 ## CPC - G06T 7/55: from multiple images - G02B 27/017: {Head mounted} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/012: {Head tracking input arrangements} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06T 7/10: Segmentation; Edge detection (motion-based segmentation G06T7/215) - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 17/205: {Re-meshing} - G02B 2027/0138: {comprising image capture systems, e.g. camera} - G02B 2027/014: {comprising information/image processing systems} - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20081: Training; Learning - G06T 2207/20092: Interactive image processing based on input by user - G06T 2207/30184: Infrastructure - G06T 2207/30244: Camera pose - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A head-mounted display and a method for depth prediction are provided. The method includes: obtaining location information and a pose of the head-mounted display; obtaining a first street view from a database according to the location information and the pose; performing image processing on a first image captured by the head-mounted display; determining whether the processed first image matches the first street view; generating depth information for an image segment according to the first street view in response to the processed first image matching the first street view; and outputting the depth information. ## Claims ### Claim 1 A head-mounted display for depth prediction, comprising: a storage medium, comprising a plurality of modules; and a processor, coupled to the storage medium, wherein the processor accesses and executes the plurality of modules, wherein the plurality of modules comprise: a geospatial information module, obtaining location information and a pose of the head-mounted display; an image projector, obtaining a first street view from a database according to the location information and the pose; a domain transfer module, performing image processing on a first image captured by the head-mounted display; a ranking module, determining whether the processed first image matches the first street view; and a depth optimizer, generating depth information for an image segment according to the first street view in response to the processed first image matching the first street view, and outputting the depth information. ### Claim 2 (depends on 1) The head-mounted display according to claim 1, wherein the domain transfer module performs the image processing by inputting the first image into a machine learning model. ### Claim 3 (depends on 2) The head-mounted display according to claim 2, wherein the domain transfer module trains the machine learning model by a training data set, wherein the training data set comprises a plurality of historical images corresponding to the same viewpoint and a plurality of timestamps respectively corresponding to the plurality of historical images. ### Claim 4 (depends on 1) The head-mounted display according to claim 1, wherein the plurality of modules further comprise: a mesh retrieve module, obtaining a three-dimensional mesh of an area from the database, and down-sampling a part of the three-dimensional mesh to obtain a simplified three-dimensional mesh, wherein the image projector projects the simplified three-dimensional mesh to a virtual plane to obtain the first street view. ### Claim 5 (depends on 4) The head-mounted display according to claim 4, wherein a distance between the part of the three-dimensional mesh and the head-mounted display is greater than a threshold. ### Claim 6 (depends on 1) The head-mounted display according to claim 1, wherein the depth optimizer performs instance-level semantic segmentation on a second image captured by the head-mounted display to obtain the image segment. ### Claim 7 (depends on 1) The head-mounted display according to claim 1, wherein the depth optimizer obtains, from the database, a plurality of depth values respectively corresponding to a plurality of pixels in the image segment, wherein the depth optimizer performs quantization on the plurality of depth values to generate the depth information. ### Claim 8 (depends on 7) The head-mounted display according to claim 7, wherein the depth optimizer performs the quantization according to a distance between the head-mounted display and an object corresponding to the image segment. ### Claim 9 (depends on 1) The head-mounted display according to claim 1, wherein the image projector obtains a plurality of street views from the database according to the location information and the pose, wherein the ranking module determines the processed first image matches the first street view by performing simultaneous localization and mapping algorithm on the plurality street views. ### Claim 10 (depends on 1) The head-mounted display according to claim 1, further comprising: a human-machine interface, coupled to the processor, wherein the ranking module receives a user command through the human-machine interface and determines the processed first image matches the first street view according to the user command. ### Claim 11 (depends on 10) The head-mounted display according to claim 10, further comprising: a camera, coupled to the processor, wherein the camera captures a plurality of images, wherein the ranking module determines the processed first image, selected from the plurality of images, matches the first street view based on the user command. ### Claim 12 (depends on 10) The head-mounted display according to claim 10, wherein the ranking module displays a plurality of street views through the human-machine interface, wherein the first street view is selected from the plurality of street views according to the user command. ### Claim 13 (depends on 1) The head-mounted display according to claim 1, further comprising: a human-machine interface, coupled to the processor, wherein the geospatial information module receives a user command through the human-machine interface and calibrates the location information or the pose according to the user command. ### Claim 14 (depends on 1) The head-mounted display according to claim 1, further comprising: a transceiver, coupled to the processor, wherein the geospatial information module receives a positioning signal through the transceiver to obtain the location information. ### Claim 15 (depends on 1) The head-mounted display according to claim 1, further comprising: an inertial measurement unit, coupled to the processor, wherein the geospatial information module obtains the pose by the inertial measurement unit. ### Claim 16 (depends on 1) The head-mounted display according to claim 1, further comprising: a human-machine interface, coupled to the processor, wherein the processor outputs, through the human-machine interface, a virtual scene comprising the image segment according to the depth information. ### Claim 17 A method for depth prediction, adapted to a head-mounted display, wherein the method comprises: obtaining location information and a pose of the head-mounted display; obtaining a first street view from a database according to the location information and the pose; performing image processing on a first image captured by the head-mounted display; determining whether the processed first image matches the first street view; generating depth information for an image segment according to the first street view in response to the processed first image matching the first street view; and outputting the depth information.