# US 10,313,584 B2 · Rear-stitched view panorama for rear-view visualization > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,313,584 B2 - Document ID: US10313584B2 - Status: Granted - Kind: B2 - Application number: 15/661,844 - Filed: 2017-07-27 - Published: 2018-07-05 - Granted: 2019-06-04 - Earliest claimed: 2017-01-04 - Assignees: Texas Instruments Incorporated - Applicants: Texas Instruments Incorporated - Inventors: Janice Shuay-Ann Pan; Vikram VijayanBabu Appia - Web page: https://patentgrep.com/patent/US10313584B2 ## CPC - H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture - B60R 1/26: to the rear of the vehicle - B60R 1/31: providing stereoscopic vision - H04N 7/181: {for receiving images from a plurality of remote sources} - H04N 13/111: Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation - H04N 13/128: Adjusting depth or disparity - H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) - H04N 13/246: Calibration of cameras - H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps - B60R 2300/105: using multiple cameras - B60R 2300/107: using stereoscopic cameras - B60R 2300/303: using joined images, e.g. multiple camera images - B60R 2300/8046: for replacing a rear-view mirror system - B60R 2300/8066: for monitoring rearward traffic - H04N 5/44504: {Circuit details of the additional information generator, e.g. details of the character or graphics signal generator, overlay mixing circuits} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR ## Abstract A rear-stitched view panorama (RSVP) system is provided that includes at least one processor and a memory storing software instructions that, when executed by the least one processor, cause the RSVP system to compute a disparity map for a left center rear image and a right center rear image captured by a stereo camera mounted on a rear of a vehicle, transform a right rear image, a left rear image, a reference center rear image, and the disparity map to a virtual world view, the right rear image and left rear image captured by respective right and left cameras mounted on the vehicle, compute an optimal left seam and an optimal right seam based on the transformed disparity map, and stitch the transformed images based on respective optimal seams to generate a panorama. ## Claims ### Claim 1 A rear-stitched view panorama (RSVP) system comprising: at least one processor; and a memory storing software instructions that, when executed by the at least one processor, cause the RSVP system to compute a disparity map for a left center rear image and a right center rear image, the left center rear image and the right center rear image captured by a stereo camera mounted on a rear of a vehicle; transform a right rear image, a left rear image, a reference center rear image of the left center rear image and the right center rear image, and the disparity map to a virtual world view using virtual camera parameters, the right rear image captured by a right camera mounted on a right side of the vehicle and the left rear image captured by a left camera mounted on a left side of the vehicle; compute an optimal left seam between the transformed left rear image and the transformed reference center rear image based on the transformed disparity map; compute an optimal right seam between the transformed right rear image and the transformed reference center rear image based on the transformed disparity map; and stitch the transformed left rear image and the transformed reference center rear image based on the optimal left seam and the transformed right rear image and the transformed reference center rear image based on the optimal right seam to generate a panorama. ### Claim 2 (depends on 1) The RSVP system of claim 1, wherein to compute an optimal left seam, the software instructions further cause the RSVP system to: compute a world disparity map; compute a world disparity difference map based on the world disparity map and the transformed disparity map; and search an overlap region between the transformed left rear image and the transformed reference center rear image for a first minimum cost seam, wherein costs of candidate seams are computed based on the world disparity difference map. ### Claim 3 (depends on 2) The RSVP system of claim 2, wherein the software instructions further cause the RSVP system to compute a cost of a candidate seam as a sum of disparity values in the world disparity difference map corresponding to pixels in the candidate seam normalized by a number of valid corresponding disparity values in the transformed disparity map. ### Claim 4 (depends on 3) The RSVP system of claim 3, wherein pixels with a disparity value in the transformed disparity map greater than a threshold are not included in the sum. ### Claim 5 (depends on 2) The RSVP system of claim 2, wherein to search an overlap region, the software instructions further cause the RSVP system to: search the overlap region for a minimum cost vertical seam; and search the overlap region for the first minimum cost seam in a neighborhood around the minimum cost vertical seam. ### Claim 6 (depends on 2) The RSVP system of claim 2, wherein to compute an optimal left seam, the software instructions further cause the RSVP system to: compute a ground plane disparity difference map based on a ground plane disparity map and the transformed disparity map; and search the overlap region between the transformed left rear image and the transformed reference center rear image for a second minimum cost seam, wherein costs of candidate seams are computed based on the ground plane disparity difference map; wherein the optimal left seam is a lowest cost seam of the first minimum cost seam and the second minimum cost seam. ### Claim 7 (depends on 1) The RSVP system of claim 1, wherein the software instructions further cause the RSVP system to: apply temporal smoothing to the optimal left seam and the optimal right seam. ### Claim 8 (depends on 7) The RSVP system of claim 7, wherein to apply temporal smoothing, the software instructions further cause the RSVP system to: set the optimal left seam to a previous optimal left seam if all pixels in the optimal left seam are within a minimum distance threshold of the previous optimal left seam; set the optimal right seam to a previous optimal right seam if all pixels in the optimal right seam are within the minimum distance threshold of the previous optimal right seam; scale the optimal left seam such that a largest pixel displacement between the optimal left seam and the previous optimal left seam is no larger than a maximum distance threshold if any pixel in the optimal left seam is greater than the maximum distance threshold number of pixels away from the previous optimal left seam; and scale the optimal right seam such that a largest pixel displacement between the optimal left seam and the previous optimal left seam is no larger than the maximum distance threshold if any pixel in the optimal right seam is greater than the maximum distance threshold away from the previous optimal right seam. ### Claim 9 (depends on 1) The RSVP system of claim 1, wherein to stitch the transformed left rear image, the software instructions further cause the RSVP system to: compute a blending look-up table (LUT) comprising alpha-blending coefficients for combining the transformed left rear image and the transformed reference center rear image; and use the blending LUT to stitch the transformed left rear image and the transformed reference center rear image. ### Claim 10 (depends on 1) The RSVP system of claim 1, wherein the software instructions further cause the RSVP system to: bias the optimal left seam when an object detection modality coupled to the RSVP system indicates presence of an object in a left blind zone, the bias allowing at least some of the object to be seen in the panorama. ### Claim 11 (depends on 1) The RSVP system of claim 1, wherein the software instructions further cause the RSVP system to: bias the optimal right seam when an object detection modality coupled to the RSVP system indicates presence of an object in a right blind zone, the bias allowing at least some of the object to be seen in the panorama. ### Claim 12 A method for generating a rear-stitched view panorama, the method comprising: computing a disparity map for a left center rear image and a right center rear image, the left center rear image captured by a stereo camera mounted on a rear of a vehicle; transforming a right rear image, a left rear image, a reference center rear image of one of the left center rear image and the right center rear image, and the disparity map to a virtual world view using virtual camera parameters, the right rear image captured by a right camera mounted on a right side of the vehicle and the left rear image captured by a left camera mounted on a left side of the vehicle; computing an optimal left seam between the transformed left rear image and the transformed reference center rear image based on the transformed disparity map; computing an optimal right seam between the transformed right rear image and the transformed reference center rear image based on the transformed disparity map; and stitching the transformed left rear image and the transformed reference center rear image based on the optimal left seam and the transformed right rear image and the transformed reference center rear image based on the optimal right seam to generate a panorama. ### Claim 13 (depends on 12) The method of claim 12, wherein computing an optimal left seam further comprises: computing a world disparity map; computing a world disparity difference map based on the world disparity map and the transformed disparity map; and searching an overlap region between the transformed left rear image and the transformed reference center rear image for a first minimum cost seam, wherein costs of candidate seams are computed based on the world disparity difference map. ### Claim 14 (depends on 13) The method of claim 13, wherein a cost of a candidate seam is computed as a sum of disparity values in the world disparity difference map corresponding to pixels in the candidate seam normalized by a number of valid corresponding disparity values in the transformed disparity map. ### Claim 15 (depends on 14) The method of claim 14, wherein pixels with a disparity value in the transformed disparity map greater than a threshold are not included in the sum. ### Claim 16 (depends on 13) The method of claim 13, wherein searching an overlap region further comprises: searching the overlap region for a minimum cost vertical seam; and searching the overlap region for the first minimum cost seam in a neighborhood around the minimum cost vertical seam. ### Claim 17 (depends on 13) The method of claim 13, wherein computing an optimal left seam further comprises: computing a ground plane disparity difference map based on a ground plane disparity map and the transformed disparity map; and searching the overlap region between the transformed left rear image and the transformed reference center rear image for a second minimum cost seam, wherein costs of candidate seams are computed based on the ground plane disparity difference map; wherein the optimal left seam is a lowest cost seam of the first minimum cost seam and the second minimum cost seam. ### Claim 18 (depends on 12) The method of claim 12, further comprising: applying temporal smoothing to the optimal left seam and the optimal right seam. ### Claim 19 (depends on 18) The method of claim 18, wherein applying temporal smoothing further comprises: setting the optimal left seam to a previous optimal left seam if all pixels in the optimal left seam are within a minimum distance threshold of the previous optimal left seam; setting the optimal right seam to a previous optimal right seam if all pixels in the optimal right seam are within the minimum distance threshold of the previous optimal right seam; scaling the optimal left seam such that a largest pixel displacement between the optimal left seam and the previous optimal left seam is no larger than a maximum distance threshold if any pixel in the optimal left seam is greater than the maximum distance threshold away from the previous optimal left seam; and scaling the optimal right seam such that a largest pixel displacement between the optimal left seam and the previous optimal left seam is no larger than the maximum distance threshold if any pixel in the optimal right seam is greater than the maximum distance threshold away from the previous optimal right seam. ### Claim 20 (depends on 12) The method of claim 12, wherein stitching the transformed left rear image further comprises: computing a blending look-up table (LUT) comprising alpha-blending coefficients for combining the transformed left rear image and the transformed reference center rear image; and using the blending LUT to stitch the transformed left rear image and the transformed reference center rear image. ### Claim 21 (depends on 12) The method of claim 12, further comprising: biasing the optimal left seam when an object detection modality indicates presence of an object in a left blind zone, the biasing allowing at least some of the object to be seen in the panorama. ### Claim 22 (depends on 12) The method of claim 12, further comprising biasing the optimal right seam when an object detection modality indicates presence of an object in a right blind zone, the biasing allowing at least some of the object to be seen in the panorama.