# US 2023/0054914 A1 · Vehicle localization > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2023/0054914 A1 - Document ID: US20230054914A1 - Status: Published application - Kind: A1 - Application number: 17/955,161 - Filed: 2022-09-28 - Published: 2023-02-23 - Earliest claimed: 2018-03-14 - Assignees: Five Al Limited - Applicants: Five Al Limited - Inventors: Lars Mennen; John Redford - Web page: https://patentgrep.com/patent/US20230054914A1 ## CPC - G01C 21/30: Map- or contour-matching - G06T 7/579: from motion - B60W 60/001: {Planning or execution of driving tasks} - G01C 21/3602: {Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera} - G01C 21/3819: {Road shape data, e.g. outline of a route} - G01C 21/3822: {Road feature data, e.g. slope data} - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06T 7/344: {involving models} - G06T 7/50: Depth or shape recovery - G06T 7/75: {involving models} - G06V 10/82: using neural networks - G06V 10/945: {User interactive design; Environments; Toolboxes} - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road} - B60W 2420/403: Image sensing, e.g. optical camera - B60W 2552/20: Road profile, i.e. the change in elevation or curvature of a plurality of continuous road segments - G06T 2207/10004: Still image; Photographic image - G06T 2207/10016: Video; Image sequence - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20076: Probabilistic image processing - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/20221: Image fusion; Image merging - G06T 2207/30244: Camera pose - G06T 2207/30256: Lane; Road marking - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract In one aspect, a vehicle localization system implements the following steps: receiving a predetermined road map; receiving at least one captured image from an image capture device of a vehicle; processing, by a road detection component, the at least one captured image, to identify therein road structure for matching with corresponding structure of the predetermined road map, and determine a location of the vehicle relative to the identified road structure; and using the determined location of the vehicle relative to the identified road structure to determine a location of the vehicle on the road map, by matching the road structure identified in the at least one captured image with the corresponding road structure of the predetermined road map. ## Claims ### Claim at position 1 -45. (canceled) ### Claim 46 A vehicle localization method implemented in a computer system, the method comprising: receiving a predetermined road map; receiving at least one road image for determining a vehicle location; processing, by a road detection component, the at least one road image, to identify therein road structure for matching with corresponding structure of the predetermined road map, and determine the vehicle location relative to the identified road structure; and using the determined vehicle location relative to the identified road structure to determine a vehicle location on the road map, by matching the road structure identified in the at least one road image with the corresponding road structure of the predetermined road map; wherein the road structure identified in the at least one road image comprises: a centre line for matching with a corresponding centre line of the road map, wherein determining the vehicle location relative thereto comprises determining a lateral separation between the vehicle and the centre line in a direction perpendicular to the centre line; and/or a junction region for matching with a corresponding junction region of the predetermined road map, wherein determining the vehicle location relative thereto comprises determining a longitudinal separation between the vehicle and the junction region in a direction along a road being travelled by the vehicle. ### Claim 47 (depends on 46) A method according to claim 46, wherein the road structure identified in the at least one road image comprises a centre line for matching with a corresponding centre line of the road map and a junction region for matching with a corresponding junction region of the predetermined road map; wherein determining the vehicle location relative thereto comprises determining a lateral separation between the vehicle and the centre line in a direction perpendicular to the centre line and a longitudinal separation between the vehicle and the junction region in a direction along a road being travelled by the vehicle. ### Claim 48 (depends on 46) A method according to claim 46, comprising: using the determined vehicle location on the predetermined road map to determine a location, relative to the vehicle location, of expected road structure indicated by the predetermined road map; and merging the road structure identified in the at least one road image with the expected road structure indicated by the predetermined road map, to determine merged road structure and a location of the merged road structure relative to the vehicle location on the predetermined road map. ### Claim 49 (depends on 46) A method according to claim 46, wherein the road detection component identifies the road structure in the at least one road image and the vehicle location relative to the identified road structure by assigning, to each of a plurality of spatial points within the image, at least one road structure classification value, and determining a location of those spatial points in a vehicle frame of reference. ### Claim 50 (depends on 49) A method according to claim 49, comprising: using the determined vehicle location on the predetermined road map to determine a location, relative to the vehicle location, of expected road structure indicated by the predetermined road map; and merging the road structure identified in the at least one road image with the expected road structure indicated by the predetermined road map, to determine merged road structure and a location of the merged road structure relative to the vehicle location on the predetermined road map; wherein the merging comprises merging the road structure classification value assigned to each of those spatial points with a corresponding road structure value determined from the predetermined road map for a corresponding spatial point on the predetermined road map. ### Claim 51 (depends on 46) A method according to claim 46, comprising: determining an approximate vehicle location on the road map and using the approximate vehicle location to determine a target area of the map containing the corresponding road structure for matching with the road structure identified in the at least one road image, wherein the vehicle location on the predetermined road map that is determined by matching those structures has a greater accuracy than the approximate vehicle location. ### Claim 52 (depends on 46) A method according to claim 46, wherein the road image comprises 3D image data and the vehicle location relative to the identified road structure is determined using depth information of the 3D image data. ### Claim 53 (depends on 52) A method according to claim 52, wherein the predetermined road map is a two dimensional road map and the method comprises a step of using the depth information to geometrically project the identified road structure onto a plane of the two dimensional road map for matching with the corresponding road structure of the two dimensional road map. ### Claim 54 (depends on 46) A method according to claim 46, wherein the road map is a three dimensional road map, the vehicle location on the predetermined road map being a three dimensional location in a frame of reference of the predetermined road map. ### Claim 55 (depends on 46) A method according to claim 46, comprising: determining an error estimate for the determined vehicle location on the predetermined road map, based on the matching of the visually identified road structure with the corresponding road structure of the road map. ### Claim 56 (depends on 55) A method according to claim 55, comprising: receiving one or more further vehicle location estimates on the road map, each with an associated indication of error; and applying a filter to: (i) the vehicle location on the road map as determined from the structure matching and the error estimate determined therefor, and (ii) the one or more further vehicle location estimates and the indication(s) of error received therewith, in order to determine an overall vehicle location estimate on the road map. ### Claim 57 (depends on 55) A method according to claim 55, comprising: using the determined vehicle location on the predetermined road map to determine a location, relative to the vehicle location, of expected road structure indicated by the predetermined road map; wherein determining the location of the expected road structure comprises determining, based on the road map and the error estimate, a plurality of expected road structure confidence values for a plurality of spatial points in a vehicle frame of reference. ### Claim 58 (depends on 57) A method according to claim 57, comprising merging the road structure identified in the at least one road image with the expected road structure indicated by the predetermined road map, to determine merged road structure and a location of the merged road structure relative to the vehicle location on the predetermined road map, wherein the merging is performed in dependence on the expected road structure confidence values for those spatial points. ### Claim 59 (depends on 46) A method according to claim 46, wherein the road detection component comprises a convolutional neural network, the road structure being identified by applying the convolutional neural network to the at least one road image. ### Claim 60 (depends on 59) A method according to claim 59, wherein the road structure identified in the at least one road image comprises a road shape identified by applying a first convolutional neural network to the at least one road image, and a junction region for matching with a corresponding junction region of the predetermined road map, the junction region identified by applying a second convolutional neural network to the at least one road image. ### Claim 61 (depends on 46) A method according to claim 46, wherein the matching is performed by determining an approximate vehicle location on the road map, determining a region of the road map corresponding to the at least one image based on the approximate location, computing an error between the captured at least one image and the corresponding region of the road map, and adapting the approximate location using an optimization algorithm to minimize the computed error, and thereby determining the said vehicle location on the road map. ### Claim 62 (depends on 46) A method according to claim 46, comprising determining an error estimate for the determined vehicle location on the predetermined road map, based on the matching of the visually identified road structure with the corresponding road structure of the road map, wherein the determined error estimate comprises or is derived from the error between the road image and the corresponding region of the road map as computed upon completion of the optimization algorithm. ### Claim 63 (depends on 46) A method according to claim 46, wherein the road structure identified in the at least one road image is matched with the corresponding road structure of the predetermined road map by matching a shape of the identified road structure with a shape of the corresponding road structure. ### Claim 64 A computer system, comprising: a map input configured to receive a predetermined road map; an image input configured to receive at least one road image for determining a vehicle location; and one or more hardware processors configured to: process the at least one road image, to identify therein road structure for matching with corresponding structure of the predetermined road map, and determine the vehicle location relative to the identified road structure; and use the determined vehicle location relative to the identified road structure to determine a vehicle location on the road map, by matching the road structure identified in the at least one road image with the corresponding road structure of the predetermined road map; wherein the road structure identified in the at least one road image comprises: a centre line for matching with a corresponding centre line of the road map, wherein determining the vehicle location relative thereto comprises determining a lateral separation between the vehicle and the centre line in a direction perpendicular to the centre line; and/or a junction region for matching with a corresponding region of the predetermined road map, wherein determining the vehicle location relative thereto comprises determining a separation between the vehicle and the junction region in a direction along a road being travelled by the vehicle. ### Claim 65 A computer program comprising executable instructions stored on a non-transitory computer-readable storage medium and configured, when executed on one or more processors, to implement operations comprising: receiving a predetermined road map; receiving at least one road image for determining a vehicle location; processing, by a road detection component, the at least one road image, to identify therein road structure for matching with corresponding structure of the predetermined road map, and determine the vehicle location relative to the identified road structure; and using the determined vehicle location relative to the identified road structure to determine a vehicle location on the road map, by matching the road structure identified in the at least one road image with the corresponding road structure of the predetermined road map; wherein the road structure identified in the at least one road image comprises: a centre line for matching with a corresponding centre line of the road map, wherein determining the vehicle location relative thereto comprises determining a lateral separation between the vehicle and the centre line in a direction perpendicular to the centre line; and/or a junction region for matching with a corresponding junction region of the predetermined road map, wherein determining the vehicle location relative thereto comprises determining a longitudinal separation between the vehicle and the junction region in a direction along a road being travelled by the vehicle.