# US 12,223,739 B2 · Obstacle detection system, agricultural work vehicle, obstacle detection program, recording medium on which obstacle detection program is recorded, and obstacle detection method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,223,739 B2 - Document ID: US12223739B2 - Status: Granted - Kind: B2 - Application number: 17/609,544 - Filed: 2020-05-20 - Published: 2022-07-21 - Granted: 2025-02-11 - Earliest claimed: 2019-06-27 - Assignees: Kubota Corporation - Applicants: Kubota Corporation - Inventors: Shunsuke Edo; Kenichi Iwami; Shunsuke Miyashita - Web page: https://patentgrep.com/patent/US12223739B2 ## CPC - G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - A01B 69/001: {Steering by means of optical assistance, e.g. television cameras (steering devices for road marking vehicles E01C23/163)} - A01B 69/008: {automatic} - A01B 76/00: Parts, details or accessories of agricultural machines or implements, not provided for in groups A01B51/00 - A01B75/00 - A01D 41/127: Control or measuring arrangements specially adapted for combines - A01D 75/185: {Avoiding collisions with obstacles} - A01D 75/20: Devices for protecting men or animals - G01S 17/931: of land vehicles - G05D 1/0214: {in accordance with safety or protection criteria, e.g. avoiding hazardous areas (monitoring the location of vehicles within a certain area, e.g. forbidden or allowed areas, in traffic control systems for road vehicles G08G1/13)} - G05D 1/0238: {using obstacle or wall sensors (G05D1/0246 and G05D1/0289 take precedence; lidar systems designed for anti-collision purposes G01S17/93)} - G05D 1/0248: {in combination with a laser (lasers per se H01S)} - G05D 1/0255: {using acoustic signals, e.g. ultra-sonic singals (sonar systems designed for anti-collision purposes G01S15/93)} - G05D 1/0259: {using magnetic or electromagnetic means} - G05D 1/247: using signals provided by artificial sources external to the vehicle, e.g. navigation beacons - G05D 1/617: Safety or protection, e.g. defining protection zones around obstacles or avoiding hazards (arrangements for controlling the position or course of two or more vehicles for avoiding collisions therebetween G05D1/693; arrangements for reacting to or preventing system or operator failure G05D1/80) - G05D 1/628: following the obstacle profile, e.g. a wall or undulated terrain - G06T 11/00: Two-dimensional [2D] image generation - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G06V 10/82: using neural networks - A01D 41/1274: {for drives} - G01S 2013/9323: {Alternative operation using light waves} - G06T 2210/22: Cropping - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - A01B: SOIL WORKING IN AGRICULTURE OR FORESTRY; PARTS, DETAILS, OR ACCESSORIES OF AGRICULTURAL MACHINES OR IMPLEMENTS, IN GENERAL (making or covering furrows or holes for sowing, planting or manuring A01C5/00; mowers convertible to soil working apparatus or capable of soil working A01D42/04; mowers combined with soil working implements A01D43/12; soil working for engineering purposes E01, E02, E21) - A01D: HARVESTING; MOWING - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract An obstacle detection system for an agricultural work vehicle includes: an obstacle estimation unit that estimates a region in a field based on a detection signal from an obstacle sensor in which region an obstacle is present, and outputs obstacle present region information; an image capturing unit that outputs a captured image of the field; an obstacle detection unit that detects the obstacle from an input image and outputs obstacle detection information; and an image preprocessing unit that generates, as an image to be input to the obstacle detection unit, a trimmed image obtained by trimming the captured image so as to include the region in which the obstacle is present, based on the obstacle present region information and shooting-angle-of-view information regarding the image capturing unit. ## Claims ### Claim 1 An obstacle detection system for an agricultural work vehicle, comprising: an obstacle sensor configured to detect an obstacle in a field; an obstacle estimation unit configured to estimate a region in the field based on a detection signal from the obstacle sensor in which region the obstacle is present and output obstacle present region information; an image capturing unit configured to capture an image of the field and output the captured image; an image preprocessing unit configured to generate a trimmed image based on the obstacle present region information and shooting-angle-of-view information regarding the image capturing unit which trimmed image is obtained by trimming the captured image so as to include the region in which the obstacle is present; and an obstacle detection unit configured to receive the trimmed image as an input image and output obstacle detection information that includes a result of the detection of the obstacle, wherein the obstacle sensor is a scanning sensor that uses ultrasonic beams, optical beams, or electromagnetic wave beams, and outputs the detection signal based on reflection beams that return to the obstacle sensor as a result of transmitted beams reflecting off a reflection body, wherein the obstacle estimation unit calculates a three-dimensional position of the obstacle from the detection signal with use of the reflection body as the obstacle, and outputs the obstacle present region information including the three-dimensional position, wherein the obstacle detection unit is configured as a machine-learned neural network, and wherein the obstacle detection information output based on the input image includes a type of the obstacle. ### Claim 2 (depends on 1) The obstacle detection system according to claim 1, wherein: travel control of the agricultural work vehicle is changed to obstacle avoidance control based on the obstacle detection information output from the obstacle detection unit. ### Claim 3 An agricultural work vehicle including an obstacle detection system, comprising: an obstacle sensor configured to detect an obstacle in a surrounding region of the agricultural work vehicle; an obstacle estimation unit configured to estimate a region based on a detection signal from the obstacle sensor in a region which the obstacle is present and output obstacle present region information; an image capturing unit configured to capture an image of a surrounding region of the agricultural work vehicle and output the captured image; an image preprocessing unit configured to generate a trimmed image based on the obstacle present region information and shooting-angle-of-view information regarding the image capturing unit which trimmed image is obtained by trimming the captured image so as to include the region in which the obstacle is present; and an obstacle detection unit configured to receive the trimmed image as an input image and output obstacle detection information that includes a result of the detection of the obstacle, wherein the obstacle sensor is a scanning sensor that uses ultrasonic beams, optical beams, or electromagnetic wave beams, and outputs the detection signal based on reflection beams that return to the obstacle sensor as a result of transmitted beams reflecting off a reflection body, wherein the obstacle estimation unit calculates a three-dimensional position of the obstacle from the detection signal with use of the reflection body as the obstacle, and outputs the obstacle present region information including the three-dimensional position, wherein the obstacle detection unit is configured as a machine-learned neural network, and wherein the obstacle detection information output based on the input image includes a type of the obstacle. ### Claim 4 A non-transitory computer readable storage medium comprising an obstacle detection program for an agricultural work vehicle that, when executed by a computer comprising at least one processor, causes the computer to perform: an obstacle estimation function of, based on a detection signal from an obstacle sensor that detects an obstacle in a surrounding region of the agricultural work vehicle, estimating a region in which the obstacle is present, and outputting obstacle present region information; an image preprocessing function of generating a trimmed image, which is obtained by trimming a captured image of a surrounding region of the agricultural work vehicle so as to include the region in which the obstacle is present, based on the captured image, shooting-angle-of-view information regarding an image capturing unit that captured the captured image, and the obstacle present region information; and an obstacle detection function of receiving the trimmed image as an input image and outputting obstacle detection information that includes a result of the detection of the obstacle, wherein the obstacle sensor is a scanning sensor that uses ultrasonic beams, optical beams, or electromagnetic wave beams, and outputs the detection signal based on reflection beams that return to the obstacle sensor as a result of transmitted beams reflecting off a reflection body, wherein an obstacle estimation unit calculates a three-dimensional position of the obstacle from the detection signal with use of the reflection body as the obstacle, and outputs the obstacle present region information including the three-dimensional position, wherein the obstacle detection function is configured as a machine-learned neural network, and wherein the obstacle detection information output based on the input image includes a type of the obstacle. ### Claim 5 A non-transitory computer readable recording medium comprising an obstacle detection program for an agricultural work vehicle, the obstacle detection program, when executed by a computer comprising at least one processor, causes the computer to perform: an obstacle estimation function of estimating a region based on a detection signal from an obstacle sensor that detects an obstacle in a surrounding region of the agricultural work vehicle in which region the obstacle is present, and outputting obstacle present region information; an image preprocessing function of generating a trimmed image, which is obtained by trimming a captured image of a surrounding region of the agricultural work vehicle so as to include the region in which the obstacle is present, based on the captured image, shooting-angle-of-view information regarding an image capturing unit that captured the captured image, and the obstacle present region information; and an obstacle detection function of receiving the trimmed image as an input image and outputting obstacle detection information that includes a result of the detection of the obstacle, wherein the obstacle sensor is a scanning sensor that uses ultrasonic beams, optical beams, or electromagnetic wave beams, and outputs the detection signal based on reflection beams that return to the obstacle sensor as a result of transmitted beams reflecting off a reflection body, wherein an obstacle estimation unit calculates a three-dimensional position of the obstacle from the detection signal with use of the reflection body as the obstacle, and outputs the obstacle present region information including the three-dimensional position, wherein the obstacle detection function is configured as a machine-learned neural network, and wherein the obstacle detection information output based on the input image includes a type of the obstacle. ### Claim 6 An obstacle detection method for an agricultural work vehicle, comprising: estimating a region based on a detection signal from an obstacle sensor that detects an obstacle in a surrounding region of the agricultural work vehicle in which region the obstacle is present, and outputting obstacle present region information; generating a trimmed image, which is obtained by trimming a captured image of a surrounding region of the agricultural work vehicle so as to include the region in which the obstacle is present, based on the captured image, shooting-angle-of-view information regarding an image capturing unit that captured the captured image, and the obstacle present region information; and receiving the trimmed image as an input image and outputting obstacle detection information that includes a result of the detection of the obstacle, wherein the obstacle sensor is a scanning sensor that uses ultrasonic beams, optical beams, or electromagnetic wave beams, and outputs the detection signal based on reflection beams that return to the obstacle sensor as a result of transmitted beams reflecting off a reflection body, wherein an obstacle estimation unit calculates a three-dimensional position of the obstacle from the detection signal with use of the reflection body as the obstacle, and outputs the obstacle present region information including the three-dimensional position, wherein a machine-learned neural network is used for the receiving of the trimmed image and the outputting of the obstacle detection information, and wherein the obstacle detection information output based on the input image includes a type of the obstacle.