# US 12,183,090 B2 · Driving scenario understanding > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,183,090 B2 - Document ID: US12183090B2 - Status: Granted - Kind: B2 - Application number: 17/855,745 - Filed: 2022-06-30 - Published: 2023-05-18 - Granted: 2024-12-31 - Earliest claimed: 2021-11-16 - Assignees: Honda Motor Co., Ltd. - Applicants: Honda Motor Co., Ltd. - Inventors: Nakul Agarwal; Yi-Ting Chen - Web page: https://patentgrep.com/patent/US12183090B2 ## CPC - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road} - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/764: using classification, e.g. of video objects - G06V 10/806: {of extracted features} - G06V 10/82: using neural networks - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract According to one aspect, intersection scenario description may be implemented by receiving a video stream of a surrounding environment of an ego-vehicle, extracting tracklets and appearance features associated with dynamic objects from the surrounding environment, extracting motion features associated with dynamic objects from the surrounding environment based on the corresponding tracklets, passing the appearance features through an appearance neural network to generate an appearance model, passing the motion features through a motion neural network to generate a motion model, passing the appearance model and the motion model through a fusion network to generate a fusion output, passing the fusion output through a classifier to generate a classifier output, and passing the classifier output through a loss function to generate a multi-label classification output associated with the ego-vehicle, dynamic objects, and corresponding motion paths. ## Claims ### Claim 1 A system for intersection scenario description, comprising: a processor; and a memory storing instructions which, when executed by the processor cause the processor to perform: receiving a video stream of a surrounding environment of an ego-vehicle; extracting tracklets and appearance features associated with one or more dynamic objects from the surrounding environment; extracting motion features associated with one or more dynamic objects from the surrounding environment based on the corresponding tracklets; passing the appearance features through an appearance neural network to generate an appearance model; passing the motion features through a motion neural network to generate a motion model; passing the appearance model and the motion model through a fusion network to generate a fusion output; passing the fusion output through a classifier to generate a classifier output; and passing the classifier output through a loss function to generate a multi-label classification output associated with the ego-vehicle, one or more of the dynamic objects, and one or more corresponding motion paths. ### Claim 2 (depends on 1) The system for intersection scenario description of claim 1, wherein weakly supervised phrase grounding is performed by converting a last fully connected layer of the fusion network and a last fully connected layer of the classifier as a set of weights between a number of nodes from the appearance neural network and the motion neural network and a total number of classes from the loss function. ### Claim 3 (depends on 1) The system for intersection scenario description of claim 1, wherein the loss function is a binary cross entropy (BCE) loss function. ### Claim 4 (depends on 1) The system for intersection scenario description of claim 1, wherein the appearance features are extracted using an Inception v3 convolutional neural network (CNN) and region of interest align (RoIAlign). ### Claim 5 (depends on 1) The system for intersection scenario description of claim 1, wherein each node of the appearance neural network and the motion neural network represents one of the one or more dynamic objects. ### Claim 6 (depends on 1) The system for intersection scenario description of claim 1, wherein the multi-label classification output includes annotations for the ego-vehicle, dynamic objects, and corresponding motion paths using action units that describe an intersection scenario, wherein the action units include action unit nouns that describe the ego-vehicle and the dynamic objects and action unit verbs that describe respective motion paths of the ego-vehicle and the dynamic objects within the intersection scene. ### Claim 7 (depends on 6) The system for intersection scenario description of claim 6, wherein the processor retrieves one or more intersection scenarios based on a query of an electronic dataset that stores a combination of action units. ### Claim 8 (depends on 7) The system for intersection scenario description of claim 7, wherein the processor retrieves one or more of the intersection scenarios based on a hamming distance between predicted binary labels for the intersection scenario and ground truth labels. ### Claim 9 (depends on 8) The system for intersection scenario description of claim 8, wherein the processor converts predicted binary labels for the intersection scenario into dictionaries. ### Claim 10 (depends on 7) The system for intersection scenario description of claim 7, wherein the processor builds a database for training neural networks based on parameters associated the query. ### Claim 11 A computer-implemented method for intersection scenario description, comprising: receiving a video stream of a surrounding environment of an ego-vehicle; extracting tracklets and appearance features associated with one or more dynamic objects from the surrounding environment; extracting motion features associated with one or more dynamic objects from the surrounding environment based on the corresponding tracklets; passing the appearance features through an appearance neural network to generate an appearance model; passing the motion features through a motion neural network to generate a motion model; passing the appearance model and the motion model through a fusion network to generate a fusion output; passing the fusion output through a classifier to generate a classifier output; and passing the classifier output through a loss function to generate a multi-label classification output indicative of the ego-vehicle, one or more of the dynamic objects, and one or more corresponding motion paths. ### Claim 12 (depends on 11) The computer-implemented for intersection scenario description of claim 11, comprising performing weakly supervised phrase grounding by converting a last fully connected layer of the fusion network and a last fully connected layer of the classifier as a set of weights between a number of nodes from the appearance neural network and the motion neural network and a total number of classes from the loss function. ### Claim 13 (depends on 11) The computer-implemented for intersection scenario description of claim 11, wherein the loss function is a binary cross entropy (BCE) loss function. ### Claim 14 (depends on 11) The computer-implemented for intersection scenario description of claim 11, wherein the appearance features are extracted using an Inception v3 convolutional neural network (CNN) and region of interest align (RoIAlign). ### Claim 15 (depends on 11) The computer-implemented for intersection scenario description of claim 11, wherein each node of the appearance neural network and the motion neural network represents one of the one or more dynamic objects. ### Claim 16 (depends on 11) The computer-implemented for intersection scenario description of claim 11, wherein the multi-label classification output includes annotations for the ego-vehicle, dynamic objects, and corresponding motion paths using action units that describe an intersection scenario, wherein the action units include action unit nouns that describe the ego-vehicle and the dynamic objects and action unit verbs that describe respective motion paths of the ego-vehicle and the dynamic objects within the intersection scene. ### Claim 17 A system for intersection scenario description, comprising: a processor; and a memory storing instructions which, when executed by the processor cause the processor to perform: receiving a video stream of a surrounding environment of an ego-vehicle; extracting tracklets and appearance features associated with one or more dynamic objects from the surrounding environment; extracting motion features associated with one or more dynamic objects from the surrounding environment based on the corresponding tracklets; passing the appearance features through an appearance neural network to generate an appearance model; passing the motion features through a motion neural network to generate a motion model; passing the appearance model and the motion model through a fusion network to generate a fusion output; passing the fusion output through a classifier to generate a classifier output; and passing the classifier output through a loss function to generate a multi-label classification output indicative of the ego-vehicle, one or more of the dynamic objects, and one or more corresponding motion paths, wherein during weakly supervised phrase grounding, a last fully connected layer of the fusion network and a last fully connected layer of the classifier include a set of weights between a number of nodes from the appearance neural network and the motion neural network and a total number of classes from the loss function. ### Claim 18 (depends on 17) The system for intersection scenario description of claim 17, wherein the loss function is a binary cross entropy (BCE) loss function. ### Claim 19 (depends on 17) The system for intersection scenario description of claim 17, wherein the appearance features are extracted using an Inception v3 convolutional neural network (CNN) and region of interest align (RoIAlign). ### Claim 20 (depends on 17) The system for intersection scenario description of claim 17, wherein each node of the appearance neural network and the motion neural network represents one of the one or more dynamic objects.