# US 2019/0174060 A1 · Image processing apparatus, image processing method, and program > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0174060 A1 - Document ID: US20190174060A1 - Status: Published application - Kind: A1 - Application number: 16/323,030 - Filed: 2017-07-21 - Published: 2019-06-06 - Earliest claimed: 2016-08-22 - Assignees: Sony Corporation - Applicants: Sony Corporation - Inventors: EIJI OBA; TOMONORI AGAWA; KYOHEI YOSHIMURA - Web page: https://patentgrep.com/patent/US20190174060A1 ## CPC - H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture - B60Q 9/008: {for anti-collision purposes} - B60R 1/26: to the rear of the vehicle - B60R 1/27: providing all-round vision, e.g. using omnidirectional cameras - G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images - H04N 7/18: Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast - H04N 7/188: {Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position (signal generation from motion picture films H04N5/253)} - H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums - H04N 25/61: the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4" - B60R 2300/105: using multiple cameras - B60R 2300/304: using merged images, e.g. merging camera image with stored images - B60R 2300/607: from a bird's eye viewpoint - B60R 2300/70: characterised by an event-triggered choice to display a specific image among a selection of captured images - B60R 2300/806: for aiding parking - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - B60Q: ARRANGEMENT OF SIGNALLING OR LIGHTING DEVICES, THE MOUNTING OR SUPPORTING THEREOF OR CIRCUITS THEREFOR, FOR VEHICLES IN GENERAL - B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract To provide an image processing apparatus that achieves image display with improved visibility of information regarding the surroundings of a vehicle. An image photographed by a camera that photographs the surroundings of the vehicle is input. An overhead image, a vehicle travel direction image, and a vehicle lateral side image corresponding to a vehicle movement direction which is either left or right of the vehicle is generated on the basis of the input image. The generated image is displayed on a display unit. For example, a display image is generated or updated in accordance with a travel direction of the vehicle and direction-of-rotation information of a steering wheel. In a case where the vehicle is backed counter-clockwise, the overhead image, the vehicle rear image, and a vehicle left side image are generated and displayed. In a case where the vehicle is backed clockwise, the overhead image, the vehicle rear image, and a vehicle right side image are generated and displayed. ## Claims ### Claim 1 An image processing apparatus, comprising an image processing unit that generates a display image for a display unit, which is visually recognizable by a vehicle operator, wherein the image processing unit receives an image photographed by a camera that photographs surroundings of a vehicle, generates, on a basis of the received photographed image, a vehicle travel direction image, and a vehicle lateral side image corresponding to a vehicle movement direction which is either left or right of the vehicle, and outputs the generated vehicle travel direction image and the generated vehicle lateral side image to the display unit. ### Claim 2 (depends on 1) The image processing apparatus according to claim 1, wherein the image processing unit further generates, on a basis of the image photographed by the camera, an overhead image which is a virtual image showing a view observed from above the vehicle or obliquely from above the vehicle, and causes the display unit to display the generated overhead image along with the vehicle travel direction image and the vehicle lateral side image at the same time. ### Claim 3 (depends on 1) The image processing apparatus according to claim 1, wherein the image processing unit receives vehicle information including information regarding at least either a travel direction of the vehicle or a direction of rotation of a steering wheel and generates or updates a display image in accordance with the received vehicle information. ### Claim 4 (depends on 3) The image processing apparatus according to claim 3, wherein the image processing unit receives direction-of-rotation information of the steering wheel as the vehicle information and generates or updates a display image in accordance with the received direction-of-rotation information of the steering wheel. ### Claim 5 (depends on 4) The image processing apparatus according to claim 4, wherein the image processing unit causes, in a manner that depends on whether the direction-of-rotation information of the steering wheel indicates a right rotation or a left rotation, a vehicle left side image or a vehicle right side image to be selectively displayed as the vehicle lateral side image. ### Claim 6 (depends on 5) The image processing apparatus according to claim 5, wherein the image processing unit receives, as the vehicle information, the travel direction information of the vehicle and the direction-of-rotation information of the steering wheel, and if it is determined that the vehicle travels backward and counter-clockwise on a basis of the travel direction information of the vehicle and the direction-of-rotation information of the steering wheel, generates a vehicle rear image as the vehicle travel direction image, generates the vehicle left side image as the vehicle lateral side image, and causes the display unit to display the generated vehicle rear image and the generated vehicle left side image. ### Claim 7 (depends on 6) The image processing apparatus according to claim 6, wherein the image processing unit further generates, on a basis of the image photographed by the camera, an overhead image which is a virtual image showing a view observed from above the vehicle, and causes the display unit to display the vehicle left side image while the vehicle left side image is arranged on a left side of the overhead image and on a side opposite to the travel direction of the vehicle. ### Claim 8 (depends on 5) The image processing apparatus according to claim 5, wherein the image processing unit receives, as the vehicle information, the travel direction information of the vehicle and the direction-of-rotation information of the steering wheel, and if it is determined that the vehicle travels backward and clockwise on a basis of the travel direction information of the vehicle and the direction-of-rotation information of the steering wheel, generates a vehicle rear image as the vehicle travel direction image, generates the vehicle right side image as the vehicle lateral side image, and causes the display unit to display the generated vehicle rear image and the generated vehicle right side image. ### Claim 9 (depends on 8) The image processing apparatus according to claim 8, wherein the image processing unit further generates, on a basis of the image photographed by the camera, an overhead image which is a virtual image showing a view observed from above the vehicle, and causes the display unit to display the vehicle right side image while the vehicle right side image is arranged on a right side of the overhead image and on a side opposite to the travel direction of the vehicle. ### Claim 10 (depends on 1) The image processing apparatus according to claim 1, wherein the image processing unit generates the vehicle lateral side image such that a side region image of the vehicle itself is included in the vehicle lateral side image. ### Claim 11 (depends on 1) The image processing apparatus according to claim 1, wherein the image processing unit generates the vehicle lateral side image in which at least any one of a guide line indicating the movement direction of the vehicle, a guide line indicating a sweeping space of an outermost portion of a vehicle body in a road space, or a steering wheel track guide line is superimposed on a generated image based on the image photographed by the camera. ### Claim 12 (depends on 2) The image processing apparatus according to claim 2, wherein the image processing unit superimposes an indication for warning of a hit risk on at least any one of the vehicle travel direction image, the vehicle lateral side image, or the overhead image. ### Claim 13 (depends on 1) The image processing apparatus according to claim 1, wherein the vehicle travel direction image includes an end portion region image of the vehicle itself. ### Claim 14 (depends on 1) The image processing apparatus according to claim 1, wherein the image processing unit further generates, on a basis of the image photographed by the camera, an overhead image which is a virtual image showing a view observed from above the vehicle or obliquely from above the vehicle, and causes the display unit to display the vehicle lateral side image while the vehicle lateral side image is superimposed on the generated overhead image. ### Claim 15 An image processing method to be executed in an image processing apparatus including an image processing unit that generates a display image for a display unit, which is visually recognizable by a vehicle operator, the image processing method comprising: by the image processing unit, receiving an image photographed by a camera that photographs surroundings of a vehicle; generating, on a basis of the received photographed image, a vehicle travel direction image and a vehicle lateral side image corresponding to a vehicle movement direction which is either left or right of the vehicle; and outputting the generated vehicle travel direction image and the generated vehicle lateral side image to the display unit. ### Claim 16 A program that causes image processing to be executed in an image processing apparatus including an image processing unit that generates a display image for a display unit, which is visually recognizable by a vehicle operator, the program causing the image processing unit to execute: processing of receiving an image photographed by a camera that photographs surroundings of a vehicle; processing of generating, on a basis of the received photographed image, a vehicle travel direction image and a vehicle lateral side image corresponding to a vehicle movement direction which is either left or right of the vehicle; and processing of outputting the generated vehicle travel direction image and the generated vehicle lateral side image to the display unit.