# US 12,118,787 B2 · Localization of narrations in image data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,787 B2
- Document ID: US12118787B2
- Status: Granted
- Kind: B2
- Application number: 17/499,193
- Filed: 2021-10-12
- Published: 2023-04-13
- Granted: 2024-10-15
- Earliest claimed: 2021-10-12
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Hailin Jin; Bryan Russell; Reuben Xin Hong Tan
- Web page: https://patentgrep.com/patent/US12118787B2

## CPC

- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 18/251: {of input or preprocessed data}
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/82: using neural networks
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 15/16: using artificial neural networks
- G10L 15/19: Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 25/54: for retrieval
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Methods, system, and computer storage media are provided for multi-modal localization. Input data comprising two modalities, such as image data and corresponding text or audio data, may be received. A phrase may be extracted from the text or audio data, and a neural network system may be utilized to spatially and temporally localize the phrase within the image data. The neural network system may include a plurality of cross-modal attention layers that each compare features across the first and second modalities without comparing features of the same modality. Using the cross-modal attention layers, a region or subset of pixels within one or more frames of the image data may be identified as corresponding to the phrase, and a localization indicator may be presented for display with the image data. Embodiments may also include unsupervised training of the neural network system.

## Claims

### Claim 1

A computer-implemented method for localizing language in image data, the computer-implemented method comprising:
receiving input data comprising a video partitioned into a first modality and a second modality, the first modality being image data comprising a plurality of frames and the second modality being one of audio data or text data corresponding to the image data;
extracting a first phrase from the second modality;
identifying, via a neural network system, a first spatiotemporal region in the first modality corresponding to the first phrase, the first spatiotemporal region comprising a first frame from the plurality of frames and a portion of pixels within the first frame, wherein the first spatiotemporal region is identified utilizing a plurality of cross-modal attention layers in the neural network system to compare features from the first modality with features from the second modality; and
causing presentation, on a graphic user interface, of an indicator of the first spatiotemporal region.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the plurality of cross-modal attention layers alternate with at least one self-attention layer.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein a full attention layer is absent from the neural network system.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein identifying, via the neural network system, the first spatiotemporal region in the first modality corresponding to the first phrase extracted from the second modality comprises:
extracting a set of spatiotemporal features from the first modality and extracting a set of word features from the second modality; and
computing similarity scores between at least the first phrase within the second modality and regions within the first modality using the set of spatiotemporal features and the set of word features, wherein the first spatiotemporal region identified in the first modality as corresponding to the first phrase has the highest similarity score relative to other regions within the first modality.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein each cross-modal attention layer is bidirectional in that, at each cross-modal attention layer, the set of spatiotemporal features are utilized for a query with the set of word features are utilized for a key and the set of spatiotemporal features are utilized for a key with the set of word features are utilized for a query.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the second modality comprises audio data, and wherein extracting the first phrase comprises extracting text from the audio data utilizing automatic speech recognition and performing natural language processing to identify the first phrase.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the indicator on the graphic user interface comprises a bounding box around the first spatiotemporal region.

### Claim 8

A computerized system for localizing language across other modalities, the computerized system comprising:
at least one processor; and
one or more computer storage media storing computer-usable instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
receiving input data partitioned into a first modality and a second modality, the first modality being image data comprising a plurality of frames and the second modality being one of audio data or text data corresponding to the image data;
extracting a first phrase from the second modality;
identifying, via a neural network system, a first spatiotemporal region in the first modality corresponding to the first phrase, the first spatiotemporal region comprising a first frame from the plurality of frames and a portion of pixels within the first frame, wherein the first spatiotemporal region is identified utilizing a plurality of cross-modal attention layers in the neural network system to compare features from the first modality with features from the second modality; and
causing presentation, on a graphic user interface, of an indicator of the first spatiotemporal region.

### Claim 9 (depends on 8)

The computerized system of claim 8, wherein the first phrase narrates an interaction between two objects.

### Claim 10 (depends on 8)

The computerized system of claim 8, wherein identifying, via the neural network system, the first spatiotemporal region in the first modality corresponding to the first phrase extracted from the second modality comprises:
extracting a set of spatiotemporal features from the first modality and extracting a set of word features from the second modality; and
computing similarity scores between at least the first phrase within the second modality and multiple regions within the first modality using the set of spatiotemporal features and the set of word features, wherein the first spatiotemporal region identified in the first modality as corresponding to the first phrase has the highest similarity score relative to other regions within the first modality.

### Claim 11 (depends on 10)

The computerized system of claim 10, wherein the operations further comprise, based on similarities scores for regions within the first modality, generating a heat map for the first modality.

### Claim 12 (depends on 11)

The computerized system of claim 11, wherein the indicator of the first spatiotemporal region comprises at least a portion of the heat map corresponding to the first spatiotemporal region.

### Claim 13 (depends on 8)

The computerized system of claim 8, wherein the plurality of cross-modal attention layers alternate with at least one self-attention layer.

### Claim 14 (depends on 8)

The computerized system of claim 8, wherein a full attention layer is absent from the neural network system.

### Claim 15 (depends on 8)

The computerized system of claim 8, wherein the indicator on the graphic user interface comprises a change in one or more of a brightness level and a color channel level for the portion of pixels within the first frame corresponding to the first spatiotemporal region, and wherein the change does not occur for pixels of the first frame outside the first spatiotemporal region.

### Claim 16

A computer-implemented method for unsupervised training of neural network system, the computer-implemented method comprising:
receiving training data, the training data comprising a first modality and a second modality, wherein the first modality comprises image data having a plurality of frames and the second modality comprises one of audio data or text data corresponding to the image data;
extracting a set of spatiotemporal features from different spatial and temporal portions of the image data of the first modality;
extracting a set of word features from the second modality;
determining a spatiotemporal feature representation and a word feature representation at least by passing the set of spatiotemporal features and the set of word features through a plurality of cross-modal attention layers within the neural network system;
computing a contrastive loss value from the spatiotemporal feature representation and the word feature representation; and
adjusting weights within the neural network system based on the contrastive loss value.

### Claim 17 (depends on 16)

The computer-implemented method of claim 16, wherein the plurality of cross-modal attention layers alternate with at least one self-attention layer within the neural network system.

### Claim 18 (depends on 16)

The computer-implemented method of claim 16, a full attention layer is absent from the neural network system.

### Claim 19 (depends on 16)

The computer-implemented method of claim 16, wherein the word feature representation comprises a sentence-level representation corresponding to a plurality of words, wherein the contrastive loss value comprises a sentence-level loss.