# US 11,079,768 B2 · Use of a reference image to detect a road obstacle > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,079,768 B2 - Document ID: US11079768B2 - Status: Granted - Kind: B2 - Application number: 16/896,203 - Filed: 2020-06-08 - Published: 2020-09-24 - Granted: 2021-08-03 - Earliest claimed: 2012-09-13 - Assignees: Waymo LLC - Applicants: Waymo LLC - Inventors: David Ian Ferguson; Jiajun Zhu - Web page: https://patentgrep.com/patent/US11079768B2 ## CPC - G05D 1/0253: {extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow (determining position or orientation from images G06T7/70)} - G06T 7/74: {involving reference images or patches} - G05D 1/0248: {in combination with a laser (lasers per se H01S)} - G05D 1/0257: {using a radar (radar systems designed for anti-collision purposes between land vehicles or between land vehicle and fixed obstacles G01S13/931)} - G06T 7/001: {using an image reference approach} - G06T 7/254: involving subtraction of images - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G06T 2207/30261: Obstacle - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Methods and systems for use of a reference image to detect a road obstacle are described. A computing device configured to control a vehicle, may be configured to receive, from an image-capture device, an image of a road on which the vehicle is travelling. The computing device may be configured to compare the image to a reference image; and identify a difference between the image and the reference image. Further, the computing device may be configured to determine a level of confidence for identification of the difference. Based on the difference and the level of confidence, the computing device may be configured to modify a control strategy associated with a driving behavior of the vehicle; and control the vehicle based on the modified control strategy. ## Claims ### Claim 1 A method, comprising: obtaining a plurality of source images associated with a road on which a vehicle is traveling; comparing respective source images of the plurality of source images; extracting one or more features common to the plurality of source images; identifying, based on the one or more common features, one or more background objects associated with the road; combining the plurality of source images to generate a reference image depicting the one or more background objects; assigning an accuracy level to the reference image, wherein the accuracy level is indicative of a degree of accuracy of the reference image in depicting the one or more background objects associated with the road; receiving, at a computing device, from a camera coupled to the vehicle, an image of the road; comparing the image to the reference image to identify a difference between the image and the reference image; identifying, based on the difference, a foreground object in the image; and controlling, using the computing device, the vehicle based on (i) the accuracy level assigned to the reference image and (ii) the identified foreground object. ### Claim 2 (depends on 1) The method of claim 1, wherein merging the plurality of source images further comprises capturing the source images from a plurality of camera poses. ### Claim 3 (depends on 1) The method of claim 1, wherein extracting the one or more features common to the plurality of source images comprises assigning an accuracy level to each feature common to the plurality of source images. ### Claim 4 (depends on 1) The method of claim 1, further comprising identifying the reference image based on location information indicative of a current location of the vehicle received from a global positioning system module and orientation information indicative of a current orientation of the vehicle received from an internal measurement unit. ### Claim 5 (depends on 4) The method of claim 4, wherein identifying the reference image is further based on a tag associated with the reference image that indicates a speed of the vehicle at the time the reference image was captured. ### Claim 6 (depends on 1) The method of claim 1, further comprising determining a level of confidence for identification of the foreground object. ### Claim 7 (depends on 6) The method of claim 6, wherein the level of confidence is based on the accuracy level assigned to the reference image. ### Claim 8 (depends on 6) The method of claim 6, further comprising determining an object type for the foreground object, wherein controlling the vehicle comprises: selecting at least one action of a plurality of actions ranked based on the object type of the foreground object and the level of confidence; modifying a control strategy associated with a driving behaviour of the vehicle based on the selected at least one action; and controlling the vehicle based on the modified control strategy. ### Claim 9 (depends on 6) The method of claim 6, further comprising determining an object type for the foreground object, wherein determining the object type of the foreground object comprises identifying the foreground object as a vehicle, the method further comprising: comparing the level of confidence to a predetermined threshold; determining that the level of confidence meets or exceeds the predetermined threshold; and based on the determination that the level of confidence meets or exceeds the predetermined threshold, using sensor information to track the identified vehicle, wherein controlling the vehicle comprises navigating the vehicle based on the sensor information used to track the identified vehicle. ### Claim 10 (depends on 6) The method of claim 6, wherein the image is captured from a first camera pose, and the reference image is captured from a second camera pose, and wherein the level of confidence is determined based on a closeness of the first camera pose to the second camera pose and the accuracy level of reference image. ### Claim 11 (depends on 10) The method of claim 10, wherein comparing the image to the reference image comprises: determining a transform based on the first camera pose and the second camera pose; and applying the transform to align the reference image to the image. ### Claim 12 (depends on 1) The method of claim 1, wherein identifying the foreground object comprises identifying one or more of (i) a moving object, (ii) an obstacle, and (iii) a road change. ### Claim 13 (depends on 1) The method of claim 1, further comprising receiving a plurality of images of the road on which the vehicle is travelling, and estimating a relative speed of the foreground object based on a speed of the vehicle and a comparison of respective positions of the foreground object in at least two images of the plurality of images. ### Claim 14 (depends on 1) The method of claim 1, further comprising estimating a relative speed of the foreground object based on a speed of the vehicle and information received from at least one of a radio detection and ranging (RADAR) unit or a light detection and ranging (LIDAR) unit. ### Claim 15 A non-transitory computer readable medium having stored thereon instructions executable by a computing device to cause the computing device to perform functions comprising: obtaining a plurality of source images associated with a road on which a vehicle is traveling; comparing respective source images of the plurality of source images; extracting one or more features common to the plurality of source images; identifying, based on the one or more common features, one or more background objects associated with the road; combining the plurality of source images to generate a reference image depicting the one or more background objects; assigning an accuracy level to the reference image, wherein the accuracy level is indicative of a degree of accuracy of the reference image in depicting the one or more background objects associated with the road; receiving, from a camera coupled to the vehicle, an image of the road; comparing the image to the reference image to identify a difference between the image and the reference image; identifying, based on the difference, a foreground object in the image; and controlling the vehicle based on (i) the accuracy level assigned to the reference image and (ii) the identified foreground object. ### Claim 16 (depends on 15) The non-transitory computer readable medium of claim 15, wherein extracting the one or more features common to the plurality of source images comprises assigning an accuracy level to each feature common to the plurality of source images. ### Claim 17 (depends on 15) The non-transitory computer readable medium of claim 15, the functions further comprising determining a level of confidence for identification of the foreground object, wherein the level of confidence is based on the accuracy level assigned to the reference image. ### Claim 18 (depends on 15) The non-transitory computer readable medium of claim 15, the functions further comprising: determining a level of confidence for identification of the foreground object; and determining an object type for the foreground object, wherein controlling the vehicle comprises: selecting at least one action of a plurality of actions ranked based on the object type of the foreground object and the level of confidence; modifying a control strategy associated with a driving behaviour of the vehicle based on the selected at least one action; and controlling the vehicle based on the modified control strategy. ### Claim 19 (depends on 15) The non-transitory computer readable medium of claim 15, the functions further comprising: identifying an object type of the foreground object; determining a level of confidence for identification of the foreground object comparing the level of confidence to a predetermined threshold; determining that the level of confidence meets or exceeds the predetermined threshold; and based on the determination that the level of confidence meets or exceeds the predetermined threshold, using sensor information to track the foreground object, wherein controlling the vehicle comprises navigating the vehicle based on the sensor information used to track the identified foreground object. ### Claim 20 A control system for a vehicle, comprising: an image-capture device; and a computing device in communication with the image-capture device and configured to: obtain a plurality of source images associated with a road on which a vehicle is traveling; compare respective source images of the plurality of source images; extract one or more features common to the plurality of source images; identify, based on the one or more common features, one or more background objects associated with the road; combine the plurality of source images to generate a reference image depicting the one or more background objects; assign an accuracy level to the reference image, wherein the accuracy level is indicative of a degree of accuracy of the reference image in depicting the one or more background objects associated with the road; receive, from a camera coupled to the vehicle, an image of the road; compare the image to the reference image to identify a difference between the image and the reference image; identify, based on the difference, a foreground object in the image; and control the vehicle based on (i) the accuracy level assigned to the reference image and (ii) the identified foreground object.