# US 2023/0419678 A1 · Joint Detection and Grouping of Road Objects Using Machine Learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0419678 A1
- Document ID: US20230419678A1
- Status: Published application
- Kind: A1
- Application number: 17/808,244
- Filed: 2022-06-22
- Published: 2023-12-28
- Earliest claimed: 2022-06-22
- Assignees: Waymo LLC
- Applicants: Waymo LLC
- Inventors: Maya Kabkab; Yu-Han Chen; Zhibo Yang; Xia Chen
- Web page: https://patentgrep.com/patent/US20230419678A1

## CPC

- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/82: using neural networks
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G08G 1/096725: {where the received information generates an automatic action on the vehicle control}
- G08G 1/096741: {where the source of the transmitted information selects which information to transmit to each vehicle}
- G08G 1/096775: {where the origin of the information is a central station}
- G08G 1/165: {for passive traffic, e.g. including static obstacles, trees}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})

## Abstract

A method includes obtaining an image representing road objects belonging to a particular class, and generating, based on the image, feature maps that represent visual features of the image. The method also includes determining, based on the feature maps and for each respective road object of the plurality of road objects, a corresponding location at which the respective road object has been detected within the image and a corresponding tag value associated with the respective road object. The method additionally includes determining groups of the road objects based on the tag value of each respective road object, and identifying, for each respective group, a corresponding road condition based on the corresponding locations of the road objects in the respective group and the particular class. The method further includes generating an output that represents the corresponding road condition of each respective group.

## Claims

### Claim 1

A computer-implemented method comprising:
obtaining an image representing a plurality of road objects belonging to a particular class;
generating, using an encoder model and based on the image, a plurality of feature maps that represent visual features of the image;
determining, based on the plurality of feature maps and for each respective road object of the plurality of road objects, a corresponding location at which the respective road object has been detected within the image;
determining, based on the plurality of feature maps and for each respective road object of the plurality of road objects, a corresponding tag value associated with the respective road object;
determining one or more groups of the plurality of road objects based on the tag value of each respective road object of the plurality of road objects, wherein each respective group of the one or more groups comprises two or more corresponding road objects;
identifying, for each respective group of the one or more groups, a corresponding road condition based on (i) the corresponding locations of the two or more corresponding road objects in the respective group and (ii) the particular class of the plurality of road objects; and
generating an output that represents the corresponding road condition of each respective group of the one or more groups.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein determining the corresponding location at which each respective road object has been detected within the image comprises:
generating, based on the plurality of feature maps and using a keypoint machine learning model, a keypoint map comprising a plurality of pixels, wherein a corresponding value of each respective pixel of the plurality of pixels indicates a likelihood that the respective pixel represents a predetermined portion of a corresponding road object of the plurality of road objects; and
identifying, within the keypoint map, two or more pixels associated with corresponding values that exceed a threshold value, wherein respective positions of the two or more pixels within the keypoint map represent the corresponding locations of the plurality of road objects.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein the predetermined portion of the corresponding road object comprises one or more of: (i) a point along a vertical center line of the corresponding road object, (ii) a point along a horizontal center line of the corresponding road object, (iii) a point on a bounding box of the corresponding road object, or (iv) a center point of the bounding box of the corresponding road object.

### Claim 4 (depends on 2)

The computer-implemented method of claim 2, wherein determining the corresponding location at which the respective road object has been detected within the image comprises:
determining, using an offset machine learning model and based on the plurality of feature maps, an offset map that represents, for each respective pixel of the plurality of pixels of the keypoint map, a corresponding offset between (i) a location of the respective pixel within the keypoint map and (ii) a location within the image of a feature represented by the respective pixel; and
determining, for each respective road object of the plurality of road objects, a corresponding location of the respective road object within the image by adding (i) the corresponding offset to (ii) a corresponding location of the respective road object within the keypoint map.

### Claim 5 (depends on 2)

The computer-implemented method of claim 2, wherein determining the corresponding tag value comprises:
generating, using a tag machine learning model, based on the plurality of feature maps, and for each respective pixel of the plurality of pixels of the keypoint map, a corresponding vector comprising a plurality of values and configured to indicate a group to which an object represented by the respective pixel belongs.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, wherein determining the one or more groups comprises:
determining, for each respective road object of the plurality of road objects, one or more corresponding distance metrics between (i) the corresponding vector associated with a pixel in the keypoint map that represents the predetermined portion of the respective road object and (ii) one or more vectors associated with one or more other pixels in the keypoint map that represent respective predetermined portions of one or more other road objects of the plurality of road objects; and
partitioning the plurality of road objects into the one or more groups based on the one or more corresponding distance metrics of each respective road object of the plurality of road objects.

### Claim 7 (depends on 2)

The computer-implemented method of claim 2, wherein determining the corresponding tag value comprises:
generating, using a tag machine learning model, based on the plurality of feature maps, and for each respective pixel of the plurality of pixels of the keypoint map, a corresponding scalar value associated with the respective pixel and configured to indicate a group to which an object represented by the respective pixel belongs.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, wherein determining the one or more groups comprises:
determining, for each respective road object of the plurality of road objects, the corresponding scalar value associated with a pixel in the keypoint map that represents the predetermined portion of the respective road object; and
partitioning the plurality of road objects into the one or more groups based on the corresponding scalar value of each respective road object of the plurality of road objects.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the corresponding tag value is determined by a tag machine learning model that comprises a semantic embedding model and a relation reasoning machine learning (ML) model, and wherein determining the corresponding tag value comprises:
generating, using the semantic embedding model, based on the plurality of feature maps, and for each respective road object of the plurality of road objects, a corresponding semantic embedding vector;
generating, for each respective road object of the plurality of road objects, a corresponding position-aware semantic embedding vector by combining (i) the corresponding semantic embedding vector of the respective road object with (ii) a positional embedding vector of the respective road object; and
generating, using the relation reasoning ML model and for each respective road object of the plurality of road objects, the corresponding tag value based on the corresponding position-aware semantic embedding vector.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, wherein the relation reasoning ML model comprises a self-attention model configured to determine, for each respective road object of the plurality of road objects, a corresponding plurality of attention scores between (i) the corresponding position-aware embedding vector of the respective road object and (ii) the corresponding position-aware embedding vectors of other road objects of the plurality of road objects, wherein the corresponding tag value of the respective road object is based on the corresponding plurality of attention scores.

### Claim 11 (depends on 9)

The computer-implemented method of claim 9, wherein the plurality of road objects are represented as nodes of a graph, and wherein the relation reasoning ML model comprises a graph neural network model configured to determine, for each respective road object of the plurality of road objects, the corresponding tag value based on one or more corresponding position-aware embedding vectors of one or more other road objects of the plurality of road objects, wherein each of the one or more other road objects is linked to the respective road object by a corresponding edge of the graph.

### Claim 12 (depends on 1)

The computer-implemented method of claim 1, wherein the plurality of road objects are configured to indicate construction zones, wherein the corresponding road condition represents a corresponding construction zone in a region of environment indicated by the corresponding locations of the two or more corresponding road objects in the respective group.

### Claim 13 (depends on 1)

The computer-implemented method of claim 1, wherein the corresponding location of each respective road object is determined using a keypoint machine learning model that has been trained by a training process comprising:
obtaining a training sample comprising (i) a training image representing a plurality of training road objects belonging to the particular class and (ii), for each respective training road object of the plurality of training road objects, a corresponding ground-truth location of the respective training road object within the training image;
generating, using the encoder model and based on the training image, a plurality of training feature maps that represent visual features of the training image;
determining, using the keypoint machine learning model, based on the plurality of training feature maps, and for each respective training road object of the plurality of training road objects, a corresponding training location at which the respective training road object has been detected within the training image;
determining a focal loss value by comparing, for each respective training road object of the plurality of training road objects and using a focal loss function, the corresponding ground-truth location of the respective training road object to the corresponding training location of the respective training road object; and
adjusting one or more parameters of the keypoint machine learning model based on the focal loss value.

### Claim 14 (depends on 1)

The computer-implemented method of claim 1, wherein the corresponding tag value associated with each respective road object is determined using a tag machine learning model that has been trained by a training process comprising:
obtaining a training sample comprising (i) a training image representing a plurality of training road objects belonging to the particular class and (ii) one or more ground-truth groups of the plurality of road objects;
generating, using the encoder model and based on the training image, a plurality of training feature maps that represent visual features of the training image;
determining, using the tag machine learning model, based on the plurality of training feature maps, and for each respective training road object of the plurality of training road objects, a corresponding training tag value associated with the respective training road object;
determining one or more training groups of the plurality of training road objects based on the corresponding training tag value of each respective training road object of the plurality of training road objects;
determining a grouping loss value by comparing, using a grouping loss function, the one or more ground-truth groups to the one or more training groups, wherein the grouping loss function is configured to, for each respective training road object of the plurality of training road objects, cause the corresponding training tag value to (i) converge to first corresponding tag values of a first set of training road objects within a ground-truth group of the respective training road object and (ii) diverge from second corresponding tag values of a second set of training road objects outside of the ground-truth group; and
adjusting one or more parameters of the tag machine learning model based on the grouping loss value.

### Claim 15 (depends on 1)

The computer-implemented method of claim 1, wherein identifying the corresponding road condition comprises:
identifying, for each respective group of the one or more groups and based on (i) the corresponding locations of the two or more corresponding road objects in the respective group and (ii) the particular class of the plurality of road objects, a corresponding environment region in which the corresponding road condition is present, wherein the output represents the corresponding environment region of each respective group of the one or more groups.

### Claim 16 (depends on 1)

The computer-implemented method of claim 1, wherein generating the output comprises:
generating, based on the image, an annotated image that represents the corresponding road condition indicated by each respective group of the one or more groups.

### Claim 17 (depends on 1)

The computer-implemented method of claim 1, further comprising:
controlling operation of a vehicle based on the output.

### Claim 18

A system comprising:
a camera;
a processor; and
a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations comprising:
obtaining, from the camera, an image representing a plurality of road objects belonging to a particular class;
generating, using an encoder model and based on the image, a plurality of feature maps that represent visual features of the image;
determining, based on the plurality of feature maps and for each respective road object of the plurality of road objects, a corresponding location at which the respective road object has been detected within the image;
determining, based on the plurality of feature maps and for each respective road object of the plurality of road objects, a corresponding tag value associated with the respective road object;
determining one or more groups of the plurality of road objects based on the tag value of each respective road object of the plurality of road objects, wherein each respective group of the one or more groups comprises two or more corresponding road objects;
identifying, for each respective group of the one or more groups, a corresponding road condition based on (i) the corresponding locations of the two or more corresponding road objects in the respective group and (ii) the particular class of the plurality of road objects; and
generating an output that represents the corresponding road condition of each respective group of the one or more groups.

### Claim 19 (depends on 18)

The system of claim 18, wherein the corresponding tag value is determined by a tag machine learning model that comprises a semantic embedding model and a relation reasoning machine learning (ML) model, and wherein determining the corresponding tag value comprises:
generating, using the semantic embedding model, based on the plurality of feature maps, and for each respective road object of the plurality of road objects, a corresponding semantic embedding vector;
generating, for each respective road object of the plurality of road objects, a corresponding position-aware semantic embedding vector by combining (i) the corresponding semantic embedding vector of the respective road object with (ii) a positional embedding vector of the respective road object; and
generating, using the relation reasoning ML model and for each respective road object of the plurality of road objects, the corresponding tag value based on the corresponding position-aware semantic embedding vector.

### Claim 20

A non-transitory computer-readable medium having stored thereon instructions that, when executed by a computing device, cause the computing device to perform operations comprising:
obtaining, from the camera, an image representing a plurality of road objects belonging to a particular class;
generating, using an encoder model and based on the image, a plurality of feature maps that represent visual features of the image;
determining, based on the plurality of feature maps and for each respective road object of the plurality of road objects, a corresponding location at which the respective road object has been detected within the image;
determining, based on the plurality of feature maps and for each respective road object of the plurality of road objects, a corresponding tag value associated with the respective road object;
determining one or more groups of the plurality of road objects based on the tag value of each respective road object of the plurality of road objects, wherein each respective group of the one or more groups comprises two or more corresponding road objects;
identifying, for each respective group of the one or more groups, a corresponding road condition based on (i) the corresponding locations of the two or more corresponding road objects in the respective group and (ii) the particular class of the plurality of road objects; and
generating an output that represents the corresponding road condition of each respective group of the one or more groups.
