# US 10,873,720 B2 · Bird's-eye view video generation device, bird's-eye view video display device, bird's-eye view video generation method, and non-transitory storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,873,720 B2
- Document ID: US10873720B2
- Status: Granted
- Kind: B2
- Application number: 16/374,760
- Filed: 2019-04-04
- Published: 2019-07-25
- Granted: 2020-12-22
- Earliest claimed: 2017-04-03
- Assignees: JVC KENWOOD Corporation
- Applicants: JVC KENWOOD Corporation
- Inventors: Manabu Asayama; Noboru Katsumata; Takuji Teruuchi
- Web page: https://patentgrep.com/patent/US10873720B2

## CPC

- H04N 5/44504: {Circuit details of the additional information generator, e.g. details of the character or graphics signal generator, overlay mixing circuits}
- B60R 1/27: providing all-round vision, e.g. using omnidirectional cameras
- B60R 1/28: with an adjustable field of view
- B62D 15/027: {Parking aids, e.g. instruction means}
- G06T 7/50: Depth or shape recovery
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 20/30: in albums, collections or shared content, e.g. social network photos or video
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- H04N 5/2628: {Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation}
- H04N 5/265: Mixing
- H04N 5/272: Means for inserting a foreground image in a background image, i.e. inlay, outlay
- H04N 7/18: Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N 23/63: by using electronic viewfinders
- H04N 23/633: {for displaying additional information relating to control or operation of the camera}
- H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture
- G06F 3/04817: using icons (graphical or visual programming using iconic symbols G06F8/34)
- G06T 1/00: General purpose image data processing
- G06T 2207/10016: Video; Image sequence
- G06T 2207/30236: Traffic on road, railway or crossing
- G08G 1/168: {Driving aids for parking, e.g. acoustic or visual feedback on parking space}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B62D: MOTOR VEHICLES; TRAILERS (steering, or guiding on a desired track, of agricultural machines or implements A01B69/00; wheels, castors, axles, increasing wheel adhesion B60B; vehicle tyres, tyre inflation or tyre changing B60C; connections between vehicles of a train or the like B60D; vehicles for use on rail and road, amphibious or convertible vehicles B60F; suspension arrangements B60G; heating, cooling, ventilating or other air treating devices B60H; windows, windscreens, non-fixed roofs, doors or similar devices, protective coverings for vehicles not in use B60J; propulsion plant arrangements, auxiliary drives, transmissions, controls, instrumentation or dashboards B60K; electric equipment or propulsion of electrically-propelled vehicles B60L; power supply for electrically-propelled vehicles B60M; passenger accommodation not otherwise provided for B60N; adaptations for load transportation or to carry special loads or objects B60P; arrangement of signalling or lighting devices, the mounting or supporting thereof or circuits therefor, for vehicles in general B60Q; vehicles, vehicle fittings or vehicle parts not otherwise provided for B60R; servicing, cleaning, repairing, supporting, lifting, or manoeuvring, not otherwise provided for, B60S; brake arrangements, brake control systems or parts thereof B60T; air-cushion vehicles B60V; motor-cycles, accessories therefor B62J, B62K; testing of vehicles G01M)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})

## Abstract

A bird's-eye view video generation device includes a video data acquiring unit configured to acquire video data of surroundings video of a vehicle captured by cameras; a bird's-eye view video generator configured to generate a bird's-eye view video displaying a predetermined display area around the vehicle by performing viewpoint conversion and synthesizing processings on the video data of the surroundings videos with a vehicle icon; a reference object detector configured to detect a reference object around the vehicle; an orientation specifying unit configured to specify a relative orientation of the reference object with respect to the vehicle; a superimposing processor configured to generate a superimposed bird's-eye view video by superimposing information representing the relative orientation on the bird's-eye view video based on the vehicle icon or the information representing the relative orientation; and a display controller configured to cause a display panel to display the superimposed bird's-eye view video.

## Claims

### Claim 1

A bird's-eye view video generation device comprising:
a video data acquiring unit configured to acquire video data of surroundings video of a vehicle captured by a video-capturing unit;
a bird's-eye view video generator configured to generate a first bird's-eye view video that displays a predetermined display area around the vehicle by performing a viewpoint conversion processing and a synthesizing processing on the video data of the surroundings videos acquired by the video data acquiring unit with a vehicle icon representing the vehicle;
a reference object detector configured to detect, in the first bird's-eye view video, at least one reference object around the vehicle which is an object with a length equal or more than a predetermined length in the first bird's-eye view video;
an orientation specifying unit configured to specify a relative orientation of the at least one reference object detected by the reference object detector with respect to the vehicle;
a superimposing processor configured to generate a superimposed bird's-eye view video by superimposing a reference orientation icon which is a line that extends along a direction in which the at least one reference object extend and which represents the relative orientation of the at least one reference object with respect to the vehicle on the first bird's-eye view video; and
a display controller configured to cause a display to display the first bird's-eye view video or the superimposed bird's-eye view video generated by the superimposing processor,
wherein,
the display controller is further configured, when the at least one reference object is specified to be present in a display area of the first bird's-eye view video by the orientation specifying unit, to display the first bird's-eye view video on the display, and
when the at least one reference object is specified not to be present in the display area of the first bird's-eye view video by the orientation specifying unit, to display,
in a case in which a value of the relative orientation of the at least one reference object with respect to the vehicle is over a threshold, a superimposed bird's-eye view video which is generated by the superimposing processor by superimposing the reference orientation icon on the first bird's-eye view video and in which the vehicle icon is arranged with its longitudinal direction being parallel with a longitudinal direction of the first bird's-eye view video, and
in a case in which the value of the relative orientation of the at least one reference object with respect to the vehicle is under the threshold, a second superimposed bird's-eye view video which is generated by the superimposing processor by superimposing the reference orientation icon on the first bird's-eye view video and in which the reference orientation icon is arranged in parallel with a lateral direction or a longitudinal direction of the first bird's-eye view video.

### Claim 2 (depends on 1)

The bird's-eye view video generation device according to claim 1, wherein, when the relative orientation of the at least one reference object with respect to the vehicle that are specified by the orientation specifying unit matches an orientation represented by the vehicle icon, the bird's-eye view video generator is further configured to generate the second superimposed bird's-eye view video.

### Claim 3 (depends on 1)

The bird's-eye view video generation device according to claim 1, wherein, while the value of the relative orientation of the at least one reference object with respect to the vehicle that are specified by the orientation specifying unit is under the threshold, the bird's-eye view video generator is further configured to generate the second superimposed bird's-eye view video.

### Claim 4 (depends on 1)

The bird's-eye view video generation device according to claim 1, wherein the superimposing processor is further configured to change a display mode of the reference orientation icon according to the relative orientation of the at least one reference object with respect to the vehicle.

### Claim 5 (depends on 1)

The bird's-eye view video generation device according to claim 1, further comprising a distance detector configured to detect a distance between the at least one reference object that is detected by the reference object detector and the vehicle,
wherein, when the distance between the at least one reference object and the vehicle as the positional relationship between the at least one reference object and the vehicle meets a predetermined condition, the bird's-eye view video generator is further configured to generate the second superimposed bird's-eye view video.

### Claim 6 (depends on 5)

The bird's-eye view video generation device according to claim 5, wherein, when the distance between the at least one reference object and the vehicle which is detected by the distance detector is under the threshold, the bird's-eye view video generator is further configured to generate the second superimposed bird's-eye view video.

### Claim 7 (depends on 6)

The bird's-eye view video generation device according to claim 6, wherein
the distance detector is further configured to detect a closest distance between the at least one reference object detected by the reference object detector and the vehicle, and
when the closest distance between the at least one reference object and the vehicle detected by the distance detector is under the threshold, the bird's-eye view video generator is further configured to generate the second superimposed bird's-eye view video.

### Claim 8 (depends on 6)

The bird's-eye view video generation device according to claim 6, wherein
the distance detector is further configured to detect a distance between the at least one reference object detected by the reference object detector and a reference position in an opposed part which is opposed to the at least one reference object in the vehicle, and
when the distance between the at least one reference object and the reference position in the opposed part which is opposed to the at least one reference object in the vehicle is under a predetermined threshold value, the bird's-eye view video generator is further configured to generate the second superimposed bird's-eye view video.

### Claim 9 (depends on 6)

The bird's-eye view video generation device according to claim 6, wherein, while the distance between the at least one reference object and the vehicle that is detected by the distance detector is under the threshold, the bird's-eye view video generator is further configured to generate the second superimposed bird's-eye view video.

### Claim 10 (depends on 1)

A bird's-eye view video display device comprising:
the bird's-eye view video generation device according to claim 1; and
at least one of the video-capturing unit configured to supply the video data of the surroundings videos to the video data acquiring unit and the display that is caused by the display controller to display the superimposed bird's-eye view video.

### Claim 11

A bird's-eye view video generation method comprising:
acquiring video data of surroundings video of a vehicle captured by a video-capturing unit;
generating a first bird's-eye view video that displays a predetermined display area around the vehicle by performing a viewpoint conversion processing and synthesizing processing on the acquired video data of the surroundings with a vehicle icon representing the vehicle;
detecting, in the first bird's-eye view video, at least one reference object around the vehicle which is an object with a length equal or more than a predetermined length in the first bird's-eye view video;
specifying a relative orientation of the detected at least one reference object with respect to the vehicle;
generating a superimposed bird's-eye view video by superimposing a reference orientation icon which is a line that extends along a direction in which the at least one reference object extends and which represents the relative orientation of the at least one reference object with respect to the vehicle on the generated first bird's-eye view video icon; and
causing a display to display the generated first bird's-eye view video or the generated superimposed bird's-eye view video,
wherein, when the at least one reference object is specified to be present in a display area of the first bird's-eye view video, causing the display to display the first bird's-eye view video, and
when the at least one reference object is specified not to be present in the display area of the first bird's-eye view video, causing the display to display,
in a case in which a value of the relative orientation of the at least one reference object with respect to the vehicle is equal to or larger than a threshold, a superimposed bird's-eye view video which is generated by superimposing the reference orientation icon on the first bird's-eye view video and in which the vehicle icon is arranged with its longitudinal direction being parallel with a longitudinal direction of the first bird's-eye view video, and
in a case in which the value of the relative orientation of the at least one reference object with respect to the vehicle is smaller than the threshold, a second superimposed bird's-eye view video which is generated by superimposing the reference orientation icon on the first bird's-eye view video and in which the reference orientation icon is arranged in parallel with a lateral direction or a longitudinal direction of the first bird's-eye view video.

### Claim 12

A non-transitory storage medium that stores a program for causing a computer that operates as a bird's-eye view video generation device to execute a process comprising:
acquiring video data of surroundings video of a vehicle captured by a video-capturing unit;
generating a first bird's-eye view video that displays a predetermined display area around the vehicle by performing a viewpoint conversion processing and synthesizing processing on the acquired video data of the surroundings with a vehicle icon representing the vehicle;
detecting, in the first bird's-eye view video at least one reference object around the vehicle which is an object with a length equal or more than a predetermined length in the first bird's-eye view video;
specifying a relative orientation of the detected at least one reference object with respect to the vehicle;
generating a superimposed bird's-eye view video by superimposing a reference orientation icon which is a line that extends along a direction in which the at least one reference object extends and which represents the relative orientation of the at least one reference object with respect to the vehicle on the generated first bird's-eye view video; and
causing a display to display the generated first bird's-eye view video or the generated superimposed bird's-eye view video,
wherein, when the at least one reference object is specified to be present in a display area of the first bird's-eye view video, causing the display to display the first bird's-eye view video, and
when the at least one reference object is specified not to be present in the display area of the first bird's-eye view video, causing the display to display,
in a case in which a value of the relative orientation of the at least one reference object with respect to the vehicle is equal to or larger than a threshold, a superimposed bird's-eye view video which is generated by superimposing the reference orientation icon on the first bird's-eye view video and in which the vehicle icon is arranged with its longitudinal direction being parallel with a longitudinal direction of the first bird's-eye view video, and
in a case in which the value of the relative orientation of the at least one reference object with respect to the vehicle is smaller than the threshold, a second superimposed bird's-eye view video which is generated by superimposing the reference orientation icon on the first bird's-eye view video and in which the reference orientation icon is arranged in parallel with a lateral direction or a longitudinal direction of the first bird's-eye view video.