# US 12,223,743 B2 · Road surface detection device, object detection device, object detection system, mobile object, and object detection method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,223,743 B2 - Document ID: US12223743B2 - Status: Granted - Kind: B2 - Application number: 17/753,611 - Filed: 2020-09-11 - Published: 2022-10-13 - Granted: 2025-02-11 - Earliest claimed: 2019-09-12 - Assignees: KYOCERA Corporation - Applicants: KYOCERA Corporation - Inventors: Junya Kishimoto; Kenji Kono - Web page: https://patentgrep.com/patent/US12223743B2 ## CPC - G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road} - G08G 1/166: {for active traffic, e.g. moving vehicles, pedestrians, bikes} - B60W 40/06: Road conditions - G06T 7/12: Edge-based segmentation - G06T 7/593: from stereo images - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - H04N 13/128: Adjusting depth or disparity - H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) - B60W 2420/403: Image sensing, e.g. optical camera - B60W 2554/00: Input parameters relating to objects - G06T 2207/10012: Stereo images - G06T 2207/20021: Dividing image into blocks, subimages or windows - G06T 2207/30256: Lane; Road marking - H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals} - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93}) - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract An object detection device includes a processor. The processor acquires or generates a first disparity image generated on the basis of an output of a stereo camera mounted in a mobile object. In the first disparity image, pixels representing disparities are arranged on a two-dimensional plane formed by a first direction corresponding to a base-length direction of the stereo camera and a second direction intersecting the first direction. The processor approximates a relationship between a coordinate, in the second direction, of a road surface in a direction of travel of the mobile object and a disparity representing the road surface with two straight lines, the relationship being included in the first disparity image. ## Claims ### Claim 1 A road surface detection device comprising: a processor configured to: acquire or generate a first disparity map, the first disparity map being: generated based on an output of a stereo camera that captures an image including a road surface, and a map in which a disparity obtained from the output of the stereo camera is associated with two-dimensional coordinates formed by a first direction corresponding to a horizontal direction of the image captured by the stereo camera and a second direction intersecting the first direction, and using two straight lines, approximate a relationship between a coordinate of the road surface in the second direction and a disparity representing the road surface, the relationship being included in the first disparity map, wherein the two straight lines include a first straight line and a second straight line, the processor approximates a short-distance side in a real space with respect to a side on which the stereo camera is located using the first straight line and approximates a long-distance side farther than the first straight line using the second straight line, the processor is configured to determine coordinates of an approximation start point in accordance with the first disparity map, the approximation start point being a point at which approximation using the second straight line is started on the long-distance side relative to the first straight line, and the processor is configured to determine the coordinates of the approximation start point based on an error between the first straight line and the disparity representing the road surface. ### Claim 2 (depends on 1) The road surface detection device according to claim 1, wherein the processor determines the first straight line by approximating the relationship between the coordinate of the road surface in the second direction and the disparity representing the road surface from the side on which the stereo camera is located to a coordinate in the second direction corresponding to a predetermined distance using a straight line. ### Claim 3 (depends on 2) The road surface detection device according to claim 2, wherein the processor sets the predetermined distance to a distance that is one half a distance range for detecting an object. ### Claim 4 (depends on 2) The road surface detection device according to claim 2, wherein the processor determines a coordinate of the approximation start point in the second direction to be a coordinate corresponding to a side closer to the stereo camera in the real space than the predetermined distance. ### Claim 5 (depends on 1) The road surface detection device according to claim 1, wherein the processor is configured to select the second straight line from a plurality of candidate straight lines having an angle difference within a range relative to the first straight line, the plurality of candidate straight lines are candidates of the second straight line having the approximation start point as a starting point and having angles whose angle differences from the first straight line are selected from a predetermined angle range, and the processor is configured to select a candidate having a minimum error from the disparity representing the road surface, and to set the selected candidate as the second straight line. ### Claim 6 (depends on 1) An object detection device comprising the road surface detection device according to claim 1, wherein the processor is configured to detect an object using the disparity representing the road surface approximated using the straight lines. ### Claim 7 (depends on 6) The object detection device according to claim 6, wherein the processor detects the object based on a second disparity map obtained by removing the disparity representing the road surface from the first disparity map. ### Claim 8 An object detection system comprising: a stereo camera that captures a plurality of images having a disparity therebetween; and an object detection device including at least one processor, wherein the processor is configured to: acquire or generate a first disparity map, the first disparity map being: generated based on an output of the stereo camera that captures an image including a road surface, and a map in which a disparity obtained from the output of the stereo camera is associated with two-dimensional coordinates formed by a first direction corresponding to a horizontal direction of the image captured by the stereo camera and a second direction intersecting the first direction, using two straight lines, approximate a relationship between a coordinate of the road surface in the second direction and a disparity representing the road surface, the relationship being included in the first disparity map, and detect an object using the disparity representing the road surface approximated using the straight lines, wherein the two straight lines include a first straight line and a second straight line, the processor approximates a short-distance side in a real space with respect to a side on which the stereo camera is located using the first straight line and approximates a long-distance side farther than the first straight line using the second straight line, the processor is configured to determine coordinates of an approximation start point in accordance with the first disparity map, the approximation start point being a point at which approximation using the second straight line is started on the long-distance side relative to the first straight line, and the processor is configured to determine the coordinates of the approximation start point based on an error between the first straight line and the disparity representing the road surface. ### Claim 9 (depends on 8) A mobile object comprising the object detection system according to claim 8. ### Claim 10 An object detection method comprising: acquiring or generating a first disparity map, the first disparity map being: generated based on an output of a stereo camera that captures an image including a road surface, and a map in which a disparity obtained from the output of the stereo camera is associated with two-dimensional coordinates formed by a first direction corresponding to a horizontal direction of the image captured by the stereo camera and a second direction intersecting the first direction; using two straight lines, approximating a relationship between a coordinate of the road surface in the second direction and a disparity representing the road surface, the relationship being included in the first disparity map; and detecting an object using the disparity representing the road surface approximated using the straight lines, wherein the two straight lines include a first straight line and a second straight line, the method further comprising: approximating a short-distance side in a real space with respect to a side on which the stereo camera is located using the first straight line and approximating a long-distance side farther than the first straight line using the second straight line, and determining coordinates of an approximation start point: in accordance with the first disparity map, the approximation start point being a point at which approximation using the second straight line is started on the long-distance side relative to the first straight line, and based on an error between the first straight line and the disparity representing the road surface.