# US 8,784,301 B2 · Image capture unit and method with an extended depth of field > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 8,784,301 B2 - Document ID: US8784301B2 - Status: Granted - Kind: B2 - Application number: 13/209,158 - Filed: 2011-08-12 - Published: 2013-02-14 - Granted: 2014-07-22 - Earliest claimed: 2011-08-12 - Assignees: Intuitive Surgical Operations, Inc. - Applicants: Ian McDowall - Inventors: Ian McDowall - Web page: https://patentgrep.com/patent/US8784301B2 ## CPC - A61B 1/00193: {adapted for stereoscopic vision} - A61B 1/000094: {extracting biological structures} - A61B 1/000095: {for image enhancement} - A61B 1/00045: {Display arrangement} - A61B 1/00096: {Optical elements} - A61B 1/051: {Details of CCD assembly} - G02B 23/2415: {Stereoscopic endoscopes} - G02B 23/2469: {using optical fibres} - G02B 27/0075: {with means for altering, e.g. increasing, the depth of field or depth of focus} - G06T 7/571: from focus - G06T 7/593: from stereo images - H04N 13/15: for colour aspects of image signals - H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) - H04N 13/243: using three or more two-dimensional [2D] image sensors - H04N 13/257: Colour aspects - H04N 23/10: for generating image signals from different wavelengths - H04N 23/45: for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images - H04N 23/55: Optical parts specially adapted for electronic image sensors; Mounting thereof - H04N 23/56: provided with illuminating means - H04N 23/60: Control of cameras or camera modules - H04N 23/741: by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors - H04N 23/843: {Demosaicing, e.g. interpolating colour pixel values} - H04N 23/951: by using two or more images to influence resolution, frame rate or aspect ratio - H04N 25/134: {based on three different wavelength filter elements} - G02B 23/2423: {of the distal end} - G02B 23/2453: {of the proximal end} - G06T 2207/10068: Endoscopic image - G06T 2207/30004: Biomedical image processing - H04N 13/117: the virtual viewpoint locations being selected by the viewers or determined by viewer tracking - H04N 13/279: the virtual viewpoint locations being selected by the viewers or determined by tracking - H04N 23/16: Optical arrangements associated therewith, e.g. for beam-splitting or for colour correction - H04N 23/555: {for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes} - H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals} - H04N 2209/046: Colour interpolation to calculate the missing colour values - H04N 2209/048: having several pick-up sensors - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract In a minimally invasive surgical system, an image capture unit includes a prism assembly and sensor assembly. The prism assembly includes a beam splitter, while the sensor assembly includes coplanar image capture sensors. Each of the coplanar image capture sensors has a common front end optical structure, e.g., the optical structure distal to the image capture unit is the same for each of the sensors. A controller enhances images acquired by the coplanar image capture sensors. The enhanced images may include (a) visible images with enhanced feature definition, in which a particular feature in the scene is emphasized to the operator of minimally invasive surgical system; (b) images having increased image apparent resolution; (c) images having increased dynamic range; (d) images displayed in a way based on a pixel color component vector having three or more color components; and (e) images having extended depth of field. ## Claims ### Claim 1 An apparatus comprising: a first image capture sensor comprising a first image capture sensor surface; a second image capture sensor comprising a second image capture sensor surface; a prism assembly, positioned to receive light, comprising: a distal face through which the received light enters the prism assembly; a beam splitter configured to reflect a first portion of the received light, and to transmit a second portion of the received light; and a surface configured to reflect the first portion of the received light onto the first image capture sensor, wherein the surface is separated from the beam splitter; and a reflective unit positioned to receive the second portion of the received light, the reflective unit being configured to direct the second portion of the received light to the second image capture sensor, wherein a first optical path length from the distal face to the first image capture sensor surface is smaller than a second optical path length from the distal face to the second image capture sensor surface, wherein the first image capture sensor captures an image focused at a first object distance, wherein the second image capture sensor captures an image focused at a second object distance; and wherein the first and second image capture sensor surfaces are coplanar. ### Claim 2 (depends on 1) The apparatus of claim 1, wherein the first and second image capture sensors comprise different areas of an image capture sensor chip. ### Claim 3 (depends on 1) The apparatus of claim 1, comprising an endoscope, the endoscope comprising a distal end, the distal end including the first and second image capture sensors, the prism assembly, and the reflective unit. ### Claim 4 (depends on 1) The apparatus of claim 1, further comprising: a stereoscopic endoscope comprising a distal end, a pair of channels, and a plurality of first and second image capture sensors, prism assemblies, and reflective assemblies, wherein the first image capture sensor, the second image capture sensor, the prism assembly, and the reflective unit are included in the plurality, and wherein each channel in the pair of channels includes, in the distal end of the stereoscopic endoscope, a different first image capture sensor of the plurality, a different second image capture sensor of the plurality, a different prism assembly of the plurality, and a different reflective unit of the plurality. ### Claim 5 (depends on 1) The apparatus of claim 1, wherein the prism assembly comprises a pentaprism. ### Claim 6 (depends on 1) The apparatus of claim 1, further comprising a stop positioned distal to the prism assembly and proximal to an opening in the apparatus through which the light enters the apparatus. ### Claim 7 (depends on 1) The apparatus of claim 1, wherein a first portion of the received light is a first percentage of the received light, and the second portion of the received light is a second percentage of the received light. ### Claim 8 (depends on 7) The apparatus of claim 7, wherein the first and second percentages are both about equal. ### Claim 9 (depends on 7) The apparatus of claim 7, wherein one hundred percent minus the first percentage is about the second percentage. ### Claim 10 (depends on 9) The apparatus of claim 9, wherein the first and second percentages are not about equal, and the first percentage is smaller than the second percentage. ### Claim 11 (depends on 10) The apparatus of claim 10, wherein at least one of the first and second image capture sensors comprises a color image capture sensor. ### Claim 12 (depends on 11) The apparatus of claim 11, further comprising: a controller coupled to the first and second image capture sensors to receive first and second images, the controller being configured to automatically generate an in focus output image from the first and second images, the in focus output image having an extended depth of field. ### Claim 13 (depends on 12) The apparatus of claim 12, the controller being configured to sample a first group of pixels in the first image, the controller being configured to sample a corresponding second group of pixels in the second image, and the controller being configured to select one of the first group of pixels and the second group of pixels for the output image. ### Claim 14 (depends on 11) The apparatus of claim 11, further comprising: a controller coupled to the first and second image capture sensors, the controller being configured to retrieve a first image captured by the first image capture sensor and to retrieve a second image captured by the second image capture sensor, and the controller being configured to generate a depth map from the first and second images. ### Claim 15 (depends on 14) The apparatus of claim 14, the controller further being configured to generate a three-dimensional surface of a scene; the controller being configured to project and texture map the first and second images back on the three-dimensional surface to generate a textured image surface; and the controller being configured to generate a new image for a virtual camera point from the depth map and the textured image surface. ### Claim 16 (depends on 11) The apparatus of claim 11, further comprising: a stereoscopic endoscope comprising a distal end, a pair of channels, and a plurality of first and second image capture sensors, prism assemblies, and reflective assemblies, wherein the first image capture sensor, the second image capture sensor, the prism assembly, and the reflective unit are included in the plurality, and wherein each channel in the pair of channels includes, in the distal end of the stereoscopic endoscope, a different first image capture sensor of the plurality, a different second image capture sensor of the plurality, a different prism assembly of the plurality, and a different reflective unit of the plurality. ### Claim 17 (depends on 16) The apparatus of claim 16, further comprising: a controller coupled to the first and second image capture sensors in a first channel of the pair to receive first and second acquired images and coupled to first and second image capture sensors in a second channel of the pair to receive third and fourth acquired images; the controller being configured to generate a first channel depth map for the first channel and to generate a second channel depth map for the second channel; the controller being' configured to generate a three-dimensional surface of a scene based on the first and second channel depth maps; the controller being configured to project and texture the first, second, third and fourth acquired images on the three-dimensional surface to generate a textured image surface; and the controller being configured to generate a new image for a new virtual camera view point from textured image surface. ### Claim 18 In a system comprising an image capture unit and a controller, a method comprising: separating, by a beam splitter of the image capture unit, light received from a common front end optical system into a first portion and a second portion; reflecting, by a surface separated from the beam splitter, the first portion of the received light to a first image capture sensor of the image capture unit; directing, by a reflective unit of the image capture unit, the second portion of the light to a second image capture sensor of the image capture unit; capturing, in the first image capture sensor, a first image from the first portion of light; capturing, in the second image capture sensor of the image capture unit, a second image from the second portion of the light, wherein a first optical path length in the image capture unit to the first image capture sensor is smaller than a second optical path length in the image capture unit to the second image capture sensor, wherein the first image capture sensor captures an image focused at a first object distance, and wherein the second image capture sensor captures an image focused at a second object distance, and wherein surfaces of the first and second image capture sensors are coplanar; retrieving, by the controller, the first image from the first image capture sensor; retrieving, by the controller, the second image from the second image capture sensor; and combining, by the controller, information from the first image and information from the second image to generate an in focus image having an extended depth of field.