# US 2026/0267914 A1 · Retrieval augmented video understanding with compositional reasoning over graph

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0267914 A1
- Document ID: US20260267914A1
- Status: Published application
- Kind: A1
- Application number: 19/555,048
- Filed: 2026-03-03
- Published: 2026-09-10
- Earliest claimed: 2025-03-07
- Assignees: Fujitsu Limited
- Applicants: Fujitsu Limited
- Inventors: Sameer MALIK; Moyuru YAMADA
- Web page: https://patentgrep.com/patent/US20260267914A1

## CPC

- G06F 16/732: Query formulation
- G06F 16/735: Filtering based on additional data, e.g. user or group profiles
- G06F 40/289: Phrasal analysis, e.g. finite state techniques or chunking
- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/042: Knowledge-based neural networks; Logical representations of neural networks
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 5/022: Knowledge engineering; Knowledge acquisition
- G06V 10/82: using neural networks
- G06V 10/86: using syntactic or structural representations of the image or video pattern, e.g. symbolic string recognition; using graph matching
- G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44)
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

According to an aspect, operations include acquiring a sequence of video frames associated with video data and a query. The operations further include acquiring a temporal graph associated with the sequence of video frames. The temporal graph includes a set of subgraphs corresponding to the sequence of video frames, with a set of nodes and edges in each subgraph representing the set of entities and relationships between the set of nodes, respectively. The operations further include converting the query into a set of retrieval functions and applying the set of retrieval functions on the temporal graph to retrieve a query response including at least one video frame of the set of video frames.

## Claims

### Claim 1

A method, executable by a system, the method comprising:
acquiring a sequence of video frames associated with video data;
receiving a query associated with the video data;
acquiring a temporal graph associated with the sequence of video frames,
the temporal graph including a set of subgraphs corresponding to the sequence of video frames, with a set of nodes in each subgraph of the set of subgraphs representing a set of entities and edges in each subgraph of the set of subgraphs representing relationships between the set of nodes in respective video frame;
converting the query into a set of retrieval functions; and
applying the set of retrieval functions on the temporal graph to retrieve a query response including at least one video frame of the sequence of video frames.

### Claim 2 (depends on 1)

The method according to claim 1, further comprising:
generating a plurality of textual descriptions corresponding to a plurality of nodes in the temporal graph,
wherein the plurality of nodes includes the set of nodes in each subgraph of the set of subgraphs, and
wherein each textual description of the plurality of textual descriptions includes edge information associated with a respective node of the plurality of nodes and attributes associated with the respective node of the plurality of nodes; and
generating a plurality of node embeddings by applying a text embedding model on the plurality of text descriptions.

### Claim 3 (depends on 2)

The method according to claim 2, further comprising grouping the plurality of textual descriptions to obtain a set of entity-specific descriptions corresponding to each entity of the set of entities.

### Claim 4 (depends on 3)

The method according to claim 3, further comprising:
phrasing the query into a sequence of phrases by prompting a neural language model with the query; and
selecting the set of retrieval functions from a defined set of retrieval functions; and
assigning each phrase of the sequence of phrases as an input parameter to a respective retrieval function of the set of retrieval functions.

### Claim 5 (depends on 4)

The method according to claim 4, wherein the set of retrieval functions include a node localization function, an entity event analysis function, and a frame extraction function.

### Claim 6 (depends on 4)

The method according to claim 4, wherein a first retrieval function of the set of retrieval functions is applied by:
selecting a first phrase as a grounding phrase from the sequence of phrases;
computing a first phrase embedding by applying the text embedding model on the first phrase;
computing a plurality of similarity scores between the first phrase embedding and each node embedding of the plurality of node embeddings; and
identifying, from the plurality of nodes, a relevant node that matches the first phrase based on the plurality of similarity scores.

### Claim 7 (depends on 6)

The method according to claim 6, wherein a second retrieval function of the set of retrieval functions is applied by:
selecting a second phrase from the sequence of phrases;
acquiring the set of entity-specific descriptions corresponding to an entity associated with the relevant node,
wherein the entity is in the set of entities, and each entity-specific description of the set of entity-specific descriptions corresponds to a respective video frame of the sequence of video frames;
determining, from the set of entity-specific descriptions, an entity-specific description that matches a context of the second phrase, by prompting the neural language model with an instruction including the second phrase and the set of entity-specific descriptions; and
determining a time index of the entity-specific description.

### Claim 8 (depends on 7)

The method according to claim 7, wherein a third retrieval function of the set of retrieval functions is applied by:
selecting a third phrase from the sequence of phrases; and
extracting the at least one video frame from the sequence of video frames based on analysis of the third phrase and the time index.

### Claim 9 (depends on 1)

The method according to claim 1, further comprising:
receiving a first query from a user device of a user; and
rephrasing, based on application of a neural language model on the first query, the first query into the query.

### Claim 10

One or more non-transitory computer-readable storage media storing instructions that, in response to being executed, cause a system to perform operations, the operations comprising:
acquiring a sequence of video frames associated with video data;
receiving a query associated with the video data;
acquiring a temporal graph associated with the sequence of video frames;
the temporal graph including a set of subgraphs corresponding to the sequence of video frames, with a set of nodes in each subgraph of the set of subgraphs representing a set of entities and edges in each subgraph of the set of subgraphs representing relationships between the set of nodes in respective video frame;
converting the query into a set of retrieval functions; and
applying the set of retrieval functions on the temporal graph to retrieve a query response including at least one video frame of the set of video frames.

### Claim 11 (depends on 10)

The one or more non-transitory computer-readable storage media according to claim 10, further comprising:
generating a plurality of textual descriptions corresponding to a plurality of nodes in the temporal graph,
wherein the plurality of nodes includes the set of nodes in each subgraph of the set of subgraphs, and
wherein each textual description of the plurality of textual descriptions includes edge information associated with a respective node of the plurality of nodes and attributes associated with the respective node of the plurality of nodes; and
generating a plurality of node embeddings by applying a text embedding model on the plurality of text descriptions.

### Claim 12 (depends on 11)

The one or more non-transitory computer-readable storage media according to claim 11, further comprising grouping the plurality of textual descriptions to obtain a set of entity-specific descriptions corresponding to each entity of the set of entities.

### Claim 13 (depends on 12)

The one or more non-transitory computer-readable storage media according to claim 12, further comprising:
phrasing the query into a sequence of phrases by prompting a neural language model with the query; and
selecting the set of retrieval functions from a defined set of retrieval functions; and
assigning each phrase of the sequence of phrases as an input parameter to a respective retrieval function of the set of retrieval functions.

### Claim 14 (depends on 13)

The one or more non-transitory computer-readable storage media according to claim 13, wherein the set of retrieval functions include a node localization function, an entity event analysis function, and a frame extraction function.

### Claim 15 (depends on 13)

The one or more non-transitory computer-readable storage media according to claim 13, wherein a first retrieval function of the set of retrieval functions is applied by:
selecting a first phrase as a grounding phrase from the sequence of phrases;
computing a first phrase embedding by applying the text embedding model on the first phrase;
computing a plurality of similarity scores between the first phrase embedding and each node embedding of the plurality of node embeddings; and
identifying, from the plurality of nodes, a relevant node that matches the first phrase based on the plurality of similarity scores.

### Claim 16 (depends on 15)

The one or more non-transitory computer-readable storage media according to claim 15, wherein a second retrieval function of the set of retrieval functions is applied by:
selecting a second phrase from the sequence of phrases;
acquiring the set of entity-specific descriptions corresponding to an entity associated with the relevant node,
wherein the entity is in the set of entities, and each entity-specific description of the set of entity-specific descriptions corresponds to a respective video frame of the sequence of video frames;
determining, from the set of entity-specific descriptions, an entity-specific description that matches a context of the second phrase, by prompting the neural language model with an instruction including the second phrase and the set of entity-specific descriptions; and
determining a time index of the entity-specific description.

### Claim 17 (depends on 16)

The one or more non-transitory computer-readable storage media according to claim 16, wherein a third retrieval function of the set of retrieval functions is applied by:
selecting a third phrase from the sequence of phrases; and
extracting the at least one video frame from the sequence of video frames based on analysis of the third phrase and the time index.

### Claim 18 (depends on 10)

The one or more non-transitory computer-readable storage media according to claim 10, further comprising:
receiving a first query from a user device of a user; and
rephrasing, based on application of a neural language model on the first query, the first query into the query.

### Claim 19

A system, comprising:
one or more memory devices storing instructions, and
one or more processors, coupled to the one or more memory devices, executing the stored instructions to perform a process comprising:
acquiring a sequence of video frames associated with video data;
receiving a query associated with the video data;
acquiring a temporal graph associated with the sequence of video frames,
the temporal graph including a set of subgraphs corresponding to the sequence of video frames, with a set of nodes in each subgraph of the set of subgraphs representing a set of entities and edges in each subgraph of the set of subgraphs representing relationships between the set of nodes;
converting the query into a set of retrieval functions; and
applying the set of retrieval functions on the temporal graph to retrieve a query response including at least one video frame of the set of video frames.

### Claim 20 (depends on 19)

The system according to claim 19, wherein the process further comprises:
generating a plurality of textual descriptions corresponding to a plurality of nodes in the temporal graph,
wherein the plurality of nodes includes the set of nodes in each subgraph of the set of subgraphs, and
wherein each textual description of the plurality of textual descriptions includes edge information associated with a respective node of the plurality of nodes and attributes associated with the respective node of the plurality of nodes; and
generating a plurality of node embeddings by applying a text embedding model on the plurality of text descriptions.
