# US 2014/0192158 A1 · Stereo Image Matching > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2014/0192158 A1 - Document ID: US20140192158A1 - Status: Published application - Kind: A1 - Application number: 13/733,911 - Filed: 2013-01-04 - Published: 2014-07-10 - Earliest claimed: 2013-01-04 - Assignees: Microsoft Corporation - Applicants: Microsoft Corporation - Inventors: Oliver Whyte; Adam G. Kirk; Shahram Izadi; Carsten Rother; Michael Bleyer; Christoph Rhemann - Web page: https://patentgrep.com/patent/US20140192158A1 ## CPC - H04N 13/20: Image signal generators - G06F 18/22: Matching criteria, e.g. proximity measures - G06T 7/593: from stereo images - G06T 2207/10012: Stereo images - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract The description relates to stereo image matching to determine depth of a scene as captured by images. More specifically, the described implementations can involve a two-stage approach where the first stage can compute depth at highly accurate but sparse feature locations. The second stage can compute a dense depth map using the first stage as initialization. This improves accuracy and robustness of the dense depth map. ## Claims ### Claim 1 A system, comprising: a processor configured to receive corresponding images of a scene from a pair of cameras, the corresponding images including features added to the scene at wavelengths of light not visible to the human eye; the processor configured to implement a sparse component configured to employ a sparse location-based matching algorithm to locate the features in the corresponding images and to determine depths of individual features; and, the processor configured to implement a dense component configured to employ a nearest neighbor field (NNF) stereo matching algorithm to the corresponding images utilizing the depths of the individual features to find corresponding pixels in the corresponding images. ### Claim 2 (depends on 1) The system of claim 1, wherein the system includes the pair of cameras. ### Claim 3 (depends on 2) The system of claim 2, wherein the system further includes at least one visible light camera. ### Claim 4 (depends on 1) The system of claim 1, wherein the wavelengths of light not visible to the human eye are infrared (IR) wavelengths and the pair of cameras are infrared (IR) cameras. ### Claim 5 (depends on 4) The system of claim 4, further comprising an IR projector configured to project the features on the scene. ### Claim 6 (depends on 5) The system of claim 5, wherein the IR projector includes a random feature generator, and wherein the features have a width of about 3 to about 5 pixels in the pair of IR cameras. ### Claim 7 (depends on 1) The system of claim 1, wherein the processor, the sparse component, and the dense component are manifest as a system on a chip. ### Claim 8 (depends on 1) The system of claim 1, wherein the processor is manifest as a central processing unit or a graphics processing unit. ### Claim 9 (depends on 1) The system of claim 1, wherein the system is manifest as a single device. ### Claim 10 A method, comprising: determining three-dimensional (3-D) locations of a set of points in a scene with a first technique; initializing a second technique with the 3-D locations of the set of points; and, propagating the second technique to determine 3-D locations of other points in the scene. ### Claim 11 (depends on 10) The method of claim 10, wherein the determining comprises: receiving first and second stereo images; detecting features within the first and second stereo images; computing a disparity map of corresponding pixels that captured the features in the first and second stereo images; and, calculating depths of the features and wherein the features comprise the points. ### Claim 12 (depends on 11) The method of claim 11, wherein the initializing comprises utilizing the depths of the 3-D locations of individual points as a basis for selecting initial minimum and maximum depths for patches of pixels that contain the individual points. ### Claim 13 (depends on 11) The method of claim 11, wherein the first technique comprises a sparse location-based matching technique and the second technique comprises a nearest neighbor field (NNF) stereo matching technique. ### Claim 14 A device, comprising: an infrared (IR) projector configured to project features onto a scene in a random pattern; at least first and second IR cameras configured to capture corresponding images of the scene; a first component configured to determine depths of the features in the corresponding images; and, a second component configured to utilize the determined depths of the features to construct a disparity map between the corresponding images. ### Claim 15 (depends on 14) The device of claim 14, wherein the first component is configured to determine individual pixels on the corresponding images that capture individual features. ### Claim 16 (depends on 14) The device of claim 14, wherein the second component is configured to utilize the determined depths of individual pixels as a basis for selecting potential minimum and maximum depths of patches of pixels in the corresponding images. ### Claim 17 (depends on 14) The device of claim 14, wherein the first component further comprises a feature detector configured to detect an individual feature in the corresponding images and to determine which pixels in the first and second IR cameras captured the individual feature. ### Claim 18 (depends on 14) The device of claim 14, further comprising at least one visible light camera that is synchronized with the at least first and second IR cameras. ### Claim 19 (depends on 18) The device of claim 18, wherein the at least one visible light camera and the at least first and second IR cameras are all video cameras. ### Claim 20 (depends on 14) The device of claim 14, wherein the device is manifest as a smart phone, a pad type computer, a notebook type computer, a set top box, an entertainment console, or a device configured to operate in cooperation with a non-touch-sensitive display device to record user gestures relative to the non-touch-sensitive display device. ### Claim 21 A system, comprising: an infrared (IR) projector configured to project random features on a scene; a pair of IR cameras configured to capture corresponding IR images of the scene and the random features; a pair of visible light cameras configured to capture corresponding visible light images of the scene; a sparse component configured to employ a sparse location-based matching algorithm to locate the features in the corresponding IR images and to determine depths of individual random features; and, a dense component configured to employ a nearest neighbor field (NNF) stereo matching algorithm to the corresponding visible images utilizing the depths of the individual random features to determine depths of pixels in the corresponding visible light images.