Back to results
US 2025/0157353 A1Published application
Knowledge-enhanced procedure planning of instructional videos using knowledge graph and large language models
AssigneeNEC Laboratories America, Inc.
InventorsHonglu Zhou, Renqiang Min
Life of this patent
Estimating the term…
Abstract
Systems and methods include predicting a first action step and a last action step based on an initial visual observation and a goal visual state and retrieving multiple procedural plans from a procedural knowledge graph (PKG), trained using a set of training instructional videos, which start with the first action step and end with the last action step. A procedure plan is generated using the retrieved multiple procedural plans. An instructional video is generated based on the procedure plan.
Claims20
A computer-implemented method, comprising:
- predicting a first action step and a last action step based on an initial visual observation and a goal visual state;
- retrieving multiple procedural plans from a procedural knowledge graph (PKG), trained using a set of training instructional videos, which start with the first action step and end with the last action step;
- generating a procedure plan using the multiple procedural plans;
- and
- generating an instructional video based on the procedure plan.
2 more elements
A system, comprising:
- a memory storing instructions;
- and
- a processor configured to execute the instructions to:
- access a procedural knowledge graph (PKG) constructed using a set of training instructional videos;
- predict a first action step and a last action step based on an initial visual observation and a goal visual state;
- retrieve multiple procedural plans from the PKG that start with the first action step and end with the last action step;
- generate a procedure plan using the multiple procedural plans retrieved from the PKG;
- and
- generate an instructional video based on the procedure plan.
6 more elements
Dependencies are inferred from the claim text.