# US 12,594,954 B2 · Perception-based sign detection and interpretation for autonomous machine systems and applications

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,594,954 B2
- Document ID: US12594954B2
- Status: Granted
- Kind: B2
- Application number: 17/827,280
- Filed: 2022-05-27
- Published: 2022-12-01
- Granted: 2026-04-07
- Earliest claimed: 2021-05-28
- Assignees: NVIDIA Corporation
- Applicants: NVIDIA Corporation
- Inventors: Berta Rodriguez Hervas; Hang Dou; Hsin-I Chen; Kexuan Zou; Nizar Gandy Assaf; Minwoo Park
- Web page: https://patentgrep.com/patent/US12594954B2

## CPC

- B60W 60/001: {Planning or execution of driving tasks}
- B60W 30/12: Lane keeping
- B60W 40/06: Road conditions
- G06V 10/762: using clustering, e.g. of similar faces in social networks
- G06V 10/82: using neural networks
- G06V 20/582: {of traffic signs}
- G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road}
- B60W 2420/403: Image sensing, e.g. optical camera
- B60W 2552/05: Type of road, e.g. motorways, local streets, paved or unpaved roads
- B60W 2552/30: Road curve radius
- B60W 2552/53: Road markings, e.g. lane marker or crosswalk
- B60W 2555/60: Traffic rules, e.g. speed limits or right of way
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

In various examples, lanes may be grouped and a sign may be assigned to a lane in a group, then propagated to another lane in the group to associate semantic meaning corresponding to the sign with the lanes. The sign may be assigned to the most similar lane as quantified by a matching score subject to the lane meeting any hard constraints. Propagation of an assignment of the sign to a different lane may be based on lane attributes and/or sign attributes. Lane attributes may be evaluated and assignments of signs may occur for a lane as a whole, and/or for particular segments of a lane (e.g., of multiple segments perceived by the system). A sign may be a compound sign that is identified as individual signs, which are associated with one another. Attributes of the compound sign may provide semantic meaning used to operate a machine.

## Claims

### Claim 1

A method comprising:
detecting, based at least on sensor data obtained using one or more sensors, at least one lane and at least one sign in an environment;
assigning one or more semantic attributes of the at least one sign to the at least one lane based at least on one or more sign attributes associated with the at least one sign and one or more lane attributes associated with the at least one lane;
computing, using first lane attributes of the at least one lane and second lane attributes of a lane, a matching score indicating a level of similarity between the first lane attributes and the second lane attributes;
based at least on the assignment of the one or more semantic attributes to the at least one lane and the matching score exceeding a threshold value, evaluating the lane using one or more propagation constraints that correspond to the at least one sign;
based at least on the evaluating indicating the lane satisfies the one or more propagation constraints, propagating the one or more semantic attributes of the at least one sign from the at least one lane to the lane; and
based at least on the one or more semantic attributes being propagated to the lane, controlling one or more of autonomous or semi-autonomous braking, steering, or acceleration of a machine based at least on the one or more semantic attributes.

### Claim 2 (depends on 1)

The method of claim 1, further comprising determining the one or more semantic attributes based at least on one or more sign types and a geographic region of the at least one sign, and the evaluating of the one or more propagation constraints includes determining that a lane type of the lane is compatible with the one or more sign types of the at least one sign using a set of criteria encoded in and selected based at least one the one or more sign types.

### Claim 3 (depends on 1)

The method of claim 1, further comprising determining the one or more semantic attributes, at least in part, by:
identifying one or more regulatory rules associated with the at least one sign based at least on a type of the at least one sign; and
parsing the one or more regulatory rules to extract the one or more semantic attributes, wherein the one or more propagation constraints include a semantic compatibility check that prevents propagation of the one or more semantic attributes to the lane if a semantic meaning of the at least one sign is incompatible with a lane type of the lane, the semantic compatibility check being derived from the one or more regulatory rules.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
classifying, using one or more machine learning models (MLMs), the at least one lane as corresponding to one or more lane types; and
identifying an association between one or more sign types of the at least one sign and the one or more lane types, wherein the assigning the at least one sign to the at least one lane is based at least on the association being identified.

### Claim 5 (depends on 1)

The method of claim 1, wherein the at least one sign includes a first sign and a second sign, and the assigning comprises assigning the first sign and the second sign to the at least one lane based at least on classifying the first sign as a primary sign and the second sign as a supplemental sign corresponding to the first sign.

### Claim 6 (depends on 1)

The method of claim 1, wherein the one or more lane attributes correspond to:
one or more roadmarks associated with the at least one lane; or
one or more lane types assigned to the at least one lane.

### Claim 7 (depends on 1)

The method of claim 1,
wherein the computing includes:
computing pair-wise matching scores between sets of lanes; and
based at least on the pair-wise matching scores, grouping the lanes into a plurality of groups, wherein the at least one lane is included in at least two groups of the plurality of groups and the evaluating is performed based at least on the lane being included in one or more of the at least two groups with the at least one lane.

### Claim 8 (depends on 1)

The method of claim 1, wherein the one or more propagation constraints include one or more hard constraints and one or more soft constraints on propagation of the one or more semantic attributes of the at least one sign to the lane.

### Claim 9 (depends on 1)

The method of claim 1, wherein the assigning of the at least one sign to the at least one lane comprises:
determining, in an image in which the at least one lane, the at least one sign, and the lane were detected using image analysis, a location associated with a sign of the at least one sign along a lane of the at least one lane;
defining, in the image, a segment of the lane using the location; and
assigning the sign to the segment.

### Claim 10 (depends on 1)

The method of claim 1, wherein the one or more lane attributes correspond to one or more lane types that include one or more of:
an exit lane;
a turn lane;
an entry lane;
a carpool lane;
a forking lane; or
a merging lane.

### Claim 11 (depends on 1)

The method of claim 1, wherein the one or more propagation constraints vary based at least on a type of the at least one sign, such that a first set of the one or more propagation constraints applies to a first type of sign and a second set of the one or more propagation constraints applies to a second type of sign different from the first type of sign.

### Claim 12 (depends on 1)

The method of claim 1, further comprising, based at least on the matching score, grouping a plurality of lanes having different categories of lane type into a common group, and the one or more propagation constraints correspond to one or more constraints on one or more categories of lane type that can be propagated to using the at least one sign.

### Claim 13

A system comprising:
one or more processing units to execute operations comprising:
computing, using lane attributes indicating geometry information and semantic information corresponding to a plurality of lanes in an environment of a machine, matching scores indicating levels of similarity between the lane attributes;
grouping one or more first lanes of the plurality of lanes with one or more second lanes of the plurality of lanes into a group based at least on the matching scores between the one or more first lanes and the one or more second lanes exceeding a threshold value;
assigning one or more semantic attributes of at least one sign detected in the environment to at least one first lane of the one or more first lanes based at least on one or more distances between the at least one sign and the at least one first lane;
based at least on one or more lanes being in the group with the at least one first lane and the assignment of the one or more semantic attributes to the at least one first lane, evaluating the one or more lanes using one or more propagation constraints that correspond to the at least one sign;
based at least on the evaluating indicating the one or more lanes satisfy the one or more propagation constraints, propagating the one or more semantic attributes of the at least one sign from the at least one first lane to the one or more lanes; and
controlling one or more of braking, steering, or acceleration of the machine based at least on the one or more semantic attributes being propagated to the one or more lanes.

### Claim 14 (depends on 13)

The system of claim 13, wherein the grouping is further based at least on evaluating one or more similarities between at least one of:
one or more roadmarks associated with the plurality of lanes; or
one or more lane types assign to the plurality of lanes.

### Claim 15 (depends on 13)

The system of claim 13, wherein the lane attributes, the plurality of lanes, the at least one sign, the one or more distances, are each extracted from an image of the environment, and the assignment and the propagating in with respect to the image.

### Claim 16 (depends on 13)

The system of claim 13, wherein the assigning comprises:
determining a location associated with a sign of the at least one sign along a first lane of the one or more first lanes;
defining a segment of the first lane using the location; and
assigning the sign to the segment.

### Claim 17 (depends on 13)

The system of claim 13, wherein the assigning comprises assigning a first sign detected in the environment and a second sign detected in the environment to the one or more first lanes based at least on classifying the first sign as a primary sign and the second sign as a supplemental sign corresponding to the first sign.

### Claim 18 (depends on 13)

The system of claim 13, wherein the geometry associated with the plurality of lanes is determined based at least on applying sensor data generated using one or more sensors of the machine to one or more machine learning models.

### Claim 19 (depends on 13)

The system of claim 13, wherein the system comprises or is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system for generating synthetic data;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.

### Claim 20

At least one processor comprising:
one or more circuits to control one or more of braking, steering, or acceleration of a machine based at least on a propagation of one or more semantic attributes of a sign to one or more first lanes from one or more second lanes, wherein the propagation is based at least on:
generating an assignment of the one or more semantic attributes of the sign to the one or more second lanes,
computing, using first lane attributes of the one or more second lanes and second lane attributes of the one or more first lanes, one or more matching scores indicating one or more levels of similarity between the first lane attributes and the second lane attributes, and
based at least on the assignment of the one or more semantic attributes to the one or more second lanes and the one or more matching scores exceeding a threshold value, evaluating the one or more first lanes using one or more propagation constraints that correspond to the sign.

### Claim 21 (depends on 20)

The at least one processor of claim 20, wherein the first lane attributes comprise one or more locations associated with the one or more second lanes, and the assignment of the sign to the one or more second lanes is based at least on one or more distances between the one or more locations and the sign.

### Claim 22 (depends on 20)

The at least one processor of claim 20, wherein the first lane attributes represent at least one of:
geometry associated with the one or more first lanes and the one or more second lanes;
one or more lane types associated with the one or more first lanes and the one or more second lanes; or
one or more roadmarks associated with the one or more first lanes and the one or more second lanes.

### Claim 23 (depends on 20)

The at least one processor of claim 20, wherein the at least one processor is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system for generating synthetic data;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.