# US 2020/0404191 A1 · Image creation device and image creation method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0404191 A1
- Document ID: US20200404191A1
- Status: Published application
- Kind: A1
- Application number: 17/011,763
- Filed: 2020-09-03
- Published: 2020-12-24
- Earliest claimed: 2018-03-05
- Assignees: Denso Corporation
- Applicants: Denso Corporation
- Inventors: Yoshinori OZAKI; Hirohiko YANAGAWA
- Web page: https://patentgrep.com/patent/US20200404191A1

## CPC

- H04N 13/243: using three or more two-dimensional [2D] image sensors
- B60R 1/27: providing all-round vision, e.g. using omnidirectional cameras
- B60R 1/31: providing stereoscopic vision
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 20/64: Three-dimensional [3D] objects
- H04N 5/2628: {Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation}
- H04N 7/18: Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N 7/181: {for receiving images from a plurality of remote sources}
- H04N 13/111: Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
- H04N 13/156: Mixing image signals
- H04N 23/63: by using electronic viewfinders
- H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An aspect of the present disclosure is an image creation device installed in a vehicle. The image creation device includes an image capture section and a bird's-eye view image creation section. The image capture section acquires a captured image within a peripheral range around the vehicle. The bird's-eye view image creation section creates a bird's-eye view image. The bird's-eye view image creation section includes a contour extraction section, a region discrimination section, a first creation section, a second creation section, and an image combining section. The contour extraction section extracts the contour shape of an object in the captured image. The region discrimination section judges whether the contour region is a three-dimensional object region or a road surface region. The first creation section creates a stereoscopic image, and the second creation section creates a planar view image. The image combining section combines the stereoscopic image and the planar view image.

## Claims

### Claim 1

An image creation device for installation in a vehicle, comprising:
an image capture section configured to acquire a captured image, which is an image captured by an image capture device configured to capture images within a predetermined peripheral range around the vehicle, and
a bird's-eye view image creation section configured to create a bird's-eye view image, which is an image as viewed from above the vehicle, by performing a bird's-eye view image conversion, which is a process of converting the image captured by the image capture section into a bird's-eye view image;
wherein the bird's-eye view image creation section comprises:
a contour extraction section configured to extract, from the captured image that is captured by the image capture section, the contour shape of an object that is present in the captured image, based on luminance and chromaticity in the captured image;
a region discrimination section configured to judge, based on the contour shape that is extracted by the contour extraction section, whether the region represented by the contour shape is a three-dimensional object region, which is a region expressing a three-dimensional object, or is a road surface region, which is a region expressing something other than a three-dimensional object;
a first creation section configured to create a stereoscopic image by performing stereoscopic conversion of the three-dimensional object region, with the stereoscopic conversion being processing executing coordinate conversion by projecting the three-dimensional object region onto a three-dimensional projection surface, which has a predetermined three-dimensional curvature and represents a virtual plane which slopes upward with increase of distance from an imaging position of the image capture device;
a second creation section configured to create a planar view image by performing planar view conversion of the road surface region, with the planar view conversion being processing executing viewpoint conversion by projecting the road surface region onto a road surface projection surface, which is a predetermined virtual plane that represents a road surface on which the vehicle exists in real space; and
an image combining section configured to combine the stereoscopic image created by the first creation section and the planar view image created by the second creation section.

### Claim 2 (depends on 1)

The image creation device according to claim 1, wherein
the image capture section is configured to acquire a plurality of captured images from a plurality of image capture devices, installed such as to have an overlap range in which respective imaging ranges of the image capture devices overlap with one other,
the bird's-eye view image creation section is configured to create individual bird's-eye view images by executing the planar view conversion on each of the plurality of captured images acquired by the image capture section, and
the image creation device further comprises an overall combining section configured to create an overall bird's-eye view image by combining the plurality of individual bird's-eye view images;
wherein the region discrimination section comprises an object identification section, a region extraction section, and a selective conversion section;
the object identification section being configured to identify three-dimensional objects that are within the overlap range of the captured images,
the region extraction section being configured such that when the three-dimensional objects that are identified by the object identification section and are present within the overlap range represent the same three-dimensional object, the three-dimensional object region that is present closest to a lower end of the corresponding captured image is extracted from among the plurality of captured images; and
the selective conversion section being configured to create the stereoscopic image by performing the stereoscopic conversion of the extraction region;
wherein the image combining section is configured to combine the stereoscopic image created by the selective conversion section with the planar view image, at a position within the planar view image corresponding to the extraction region.

### Claim 3 (depends on 1)

The image creation device according to claim 1, wherein the image combining section comprises a combining execution section, configured to combine the stereoscopic image with the three-dimensional object region of the planar view image, created by the second creation section, such as to match the position corresponding to the lower end of the three-dimensional object region with the lower end of the stereoscopic image.

### Claim 4 (depends on 1)

The image creation device according to claim 1, wherein the region discrimination section is configured to judge whether an object is a three-dimensional object by using semantic segmentation.

### Claim 5

An image creation method that causes an image creation device for installation in a vehicle to create an image, comprising:
an image capture step configured to acquire a captured image, which is an image captured by an image capture device configured to capture images within a predetermined peripheral range around the vehicle, and
a bird's-eye view image creation step configured to create a bird's-eye view image, which is an image as viewed from above the vehicle, by performing a bird's-eye view image conversion, which is a process of converting the image captured by the image capture step into a bird's-eye view image;
wherein the bird's-eye view image creation step comprises:
a contour extraction step configured to extract, from the captured image that is captured by the image capture step, the contour shape of an object that is present in the captured image, based on luminance and chromaticity in the captured image;
a region discrimination step configured to judge, based on the contour shape that is extracted by the contour extraction step, whether the region represented by the contour shape is a three-dimensional object region, which is a region expressing a three-dimensional object, or is a road surface region, which is a region expressing something other than a three-dimensional object;
a first creation step configured to create a stereoscopic image by performing stereoscopic conversion of the three-dimensional object region, with the stereoscopic conversion being processing executing coordinate conversion by projecting the three-dimensional object region onto a three-dimensional projection surface, which has a predetermined three-dimensional curvature and representing a virtual plane which slopes upward with increase of distance from an imaging position of the image capture device;
a second creation step configured to create a planar view image by performing planar view conversion of the road surface region, with the planar view conversion being processing executing viewpoint conversion by projecting the road surface region onto a road surface projection surface, which is a predetermined virtual plane that represents a road surface on which the vehicle exists in real space; and
an image combining step configured to combine the stereoscopic image created by the first creation step and the planar view image created by the second creation step.