# US 2025/0157353 A1 · Knowledge-enhanced procedure planning of instructional videos using knowledge graph and large language models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0157353 A1
- Document ID: US20250157353A1
- Status: Published application
- Kind: A1
- Application number: 18/944,762
- Filed: 2024-11-12
- Published: 2025-05-15
- Earliest claimed: 2023-11-13
- Assignees: NEC Laboratories America, Inc.
- Applicants: NEC Laboratories America, Inc.
- Inventors: Honglu Zhou; Renqiang Min
- Web page: https://patentgrep.com/patent/US20250157353A1

## CPC

- G09B 5/065: {Combinations of audio and video presentations, e.g. videotapes, videodiscs, television systems}
- G09B 23/28: for medicine
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)

## Abstract

Systems and methods include predicting a first action step and a last action step based on an initial visual observation and a goal visual state and retrieving multiple procedural plans from a procedural knowledge graph (PKG), trained using a set of training instructional videos, which start with the first action step and end with the last action step. A procedure plan is generated using the retrieved multiple procedural plans. An instructional video is generated based on the procedure plan.

## Claims

### Claim 1

A computer-implemented method, comprising:
predicting a first action step and a last action step based on an initial visual observation and a goal visual state;
retrieving multiple procedural plans from a procedural knowledge graph (PKG), trained using a set of training instructional videos, which start with the first action step and end with the last action step;
generating a procedure plan using the multiple procedural plans; and
generating an instructional video based on the procedure plan.

### Claim 2 (depends on 1)

The method of claim 1, wherein the PKG includes action steps represented as nodes; and edges representing transitions between action steps.

### Claim 3 (depends on 2)

The method of claim 2, wherein the edges include transition probabilities between action steps.

### Claim 4 (depends on 1)

The method of claim 1, wherein predicting the first action step and the last action step includes using a step recognition model trained on the set of training instructional videos.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
prompting a large language model (LLM) with the first action step and last action step to generate additional procedural plans; and
incorporating the additional procedural plans from the LLM in generating the procedure plan.

### Claim 6 (depends on 1)

The method of claim 1, wherein generating the procedure plan includes using a procedure planning model that takes as input the initial visual observation, the goal visual state, and the multiple procedural plans.

### Claim 7 (depends on 6)

The method of claim 6, further comprising training the procedure planning model using annotated action steps from the training of instructional videos as supervision signals.

### Claim 8 (depends on 1)

The method of claim 1, wherein the instructional video includes a sequence of action steps for transitioning from the initial visual observation to the goal visual state.

### Claim 9 (depends on 1)

The method of claim 1, wherein the instructional video includes a sequence of action steps for transitioning from the initial visual observation to the goal visual state.

### Claim 10 (depends on 1)

The method of claim 1, wherein generating the procedure plan includes using artificial intelligence techniques to optimize a sequence of action steps.

### Claim 11 (depends on 1)

The method of claim 1, wherein the instructional video is generated for a medical procedure in a healthcare setting.

### Claim 12

A system, comprising:
a memory storing instructions; and
a processor configured to execute the instructions to:
access a procedural knowledge graph (PKG) constructed using a set of training instructional videos;
predict a first action step and a last action step based on an initial visual observation and a goal visual state;
retrieve multiple procedural plans from the PKG that start with the first action step and end with the last action step;
generate a procedure plan using the multiple procedural plans retrieved from the PKG; and
generate an instructional video based on the procedure plan.

### Claim 13 (depends on 12)

The system of claim 12, wherein the PKG includes:
nodes representing action steps extracted from training instructional videos; and
edges connecting the nodes, the edges representing transitions between action steps and including transition probabilities between action steps.

### Claim 14 (depends on 12)

The system of claim 12, wherein the processor is further configured to execute the instructions to predict the first action step and the last action step using a step recognition model trained on a set of training instructional videos.

### Claim 15 (depends on 12)

The system of claim 12, wherein the processor is further configured to execute the instructions to:
prompt a large language model (LLM) with the first action step and last action step to generate additional procedural plans; and
incorporate the additional procedural plans from the LLM in generating the procedure plan.

### Claim 16 (depends on 12)

The system of claim 12, further comprising: a procedure planning model that takes as input the initial visual observation, the goal visual state, and the multiple procedural plans.

### Claim 17 (depends on 16)

The system of claim 16, wherein the processor is further configured to execute the instructions to train the procedure planning model using annotated action steps from training instructional videos as supervision signals.

### Claim 18 (depends on 12)

The system of claim 12, wherein the instructional video includes a sequence of action steps for transitioning from the initial visual observation to the goal visual state.

### Claim 19 (depends on 12)

The system of claim 12, wherein the procedure plan is generated using artificial intelligence techniques to optimize a sequence of action steps.

### Claim 20 (depends on 12)

The system of claim 12, wherein the instructional video is generated for a medical procedure in a healthcare setting.
