# US 11,445,244 B1 · Context-aware question-answer for interactive media experiences

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,445,244 B1
- Document ID: US11445244B1
- Status: Granted
- Kind: B1
- Application number: 17/338,360
- Filed: 2021-06-03
- Granted: 2022-09-13
- Earliest claimed: 2021-06-03
- Assignees: Tcl Research America Inc.
- Applicants: Tcl Research America Inc.
- Inventors: Kyle Otto Jorgensen; Zhiqun Zhao; Haohong Wang; Mea Wang
- Web page: https://patentgrep.com/patent/US11445244B1

## CPC

- G06F 40/35: Discourse or dialogue representation
- H04N 21/44008: {involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 40/216: using statistical methods
- G06F 40/253: Grammatical analysis; Style critique
- G06F 40/30: Semantic analysis
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- H04N 21/439: Processing of audio elementary streams
- H04N 21/4394: {involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams (arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58)}
- H04N 21/6125: {involving transmission via Internet (transmission by internet of broadcast information H04H60/82)}
- H04N 21/643: Communication protocols {(network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP], H04L65/65)}
- H04N 21/84: Generation or processing of descriptive data, e.g. content descriptors {(systems specially adapted for using meta-information in broadcast systems H04H60/73)}
- H04N 21/8456: {by decomposing the content in the time domain, e.g. in time segments}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A context-aware method for answering a question about a video includes: receiving the question about the video that is paused at a pausing position; obtaining and analyzing context information at the pausing position of the video, the context information including supplementary materials of the video; and automatically searching an answer to the question based on the context information at the pausing position of the video.

## Claims

### Claim 1

A context-aware method for answering a question about a video, comprising:
receiving the question about the video that is paused at a pausing position;
obtaining and analyzing context information at the pausing position of the video, the context information including supplementary materials of the video; and
automatically searching an answer to the question based on the context information at the pausing position of the video,
wherein obtaining and analyzing the context information at the pausing position of the video includes:
analyzing verbs of the question to determine a tense of the question;
in response to the tense of the question being a past tense, determining a scene base of the question to include a current scene and (N−1) scenes preceding the current scene, N being a size of the scene base for the question in the past tense or in a future tense;
in response to the tense of the question being a present tense, determining the scene base to include the current scene;
in response to the tense of the question being the future tense, determining the scene base to include the current scene and (N−1) scenes succeeding the current scene; and
obtaining and analyzing the supplementary materials of the video for the scene base according to the question.

### Claim 2 (depends on 1)

The method according to claim 1, further comprising:
performing a language smoothing process on the answer to obtain a language smoothed answer; and
outputting the language smoothed answer.

### Claim 3 (depends on 1)

The method according to claim 1, wherein analyzing the supplementary materials of the video for the scene base includes:
organizing the supplementary materials into a plurality of passages; and
ranking relevancy of each of the plurality of passages with the question, the relevancy being a passage score.

### Claim 4 (depends on 1)

The method according to claim 1, wherein:
N is an integer greater than 1; or
N=3.

### Claim 5 (depends on 3)

The method according to claim 3, wherein organizing the supplementary materials into the plurality of passages includes:
organizing the supplementary materials for each scene into three levels, wherein a first level includes a passage of a summary of each scene, a second level includes a passage of a screenplay of each scene, and a third level includes a passage of closed caption of each scene or a passage of transcription of an audio track of each scene; and
indexing a passage describing a character under a character name of the character, wherein a passage describing multiple characters is indexed multiple times under each character name of the multiple characters.

### Claim 6 (depends on 3)

The method according to claim 3, wherein ranking relevancy of each of the plurality of passages with the question includes:
using a document ranker to evaluate a probability that a passage is related to the question to generate the passage score for each of the plurality of passages, wherein the passage score is a negative floating-point number.

### Claim 7 (depends on 1)

The method according to claim 1, wherein searching the answer to the question based on the context information at the pausing position of the video includes:
searching the plurality of passages to find a passage matching the question based on the passage score;
in response to finding a passage matching the question, generating the answer based on the passage matching the question; and
in response to no passage matching the question being found, generating the answer to be a notification that no answer is found.

### Claim 8 (depends on 7)

The method according to claim 7, wherein searching the plurality of passages to find the passage matching the question based on the passage score includes:
performing a location search on the plurality of passages for the passage matching the question based on the passage score;
in response to no passage matching the question being found in the location search, performing a character search on the plurality of passages for the passage matching the question based on the passage score; and
in response to no passage matching the question being found in the character search, performing an iterative search on the plurality of passages for the passage matching the question based on the passage score.

### Claim 9 (depends on 8)

The method according to claim 8, wherein:
a passage is determined to match the question when the passage score is greater than or equal to a preset threshold corresponding to a current search, the current search being one of the location search, the character search, and the iterative search; and
a passage is determined not to match the question when the passage score is smaller than the preset threshold corresponding to the current search.

### Claim 10 (depends on 9)

The method according to claim 9, wherein:
the preset threshold for the location search and the character search is higher than the preset threshold for the iterative search.

### Claim 11 (depends on 1)

The method according to claim 1, wherein:
the supplementary materials of the video include one or more of a starting timestamp and an ending timestamp of a scene, metadata of the scene, a screenplay of the video, a list of characters, closed caption of the video, and an audio track of the video.

### Claim 12 (depends on 2)

The method according to claim 2, wherein performing the language smoothing process on the answer includes:
performing a grammar error correction process on the answer to obtain the language smoothed answer.

### Claim 13

A context-aware device for answer a question about a video, comprising:
a memory storing program instructions; and
a processor coupled with the memory and configured to execute the program instructions to:
receive the question about the video that is paused at a pausing position;
obtain and analyze context information at the pausing position of the video, the context information including supplementary materials of the video; and
automatically search an answer to the question based on the context information at the pausing position of the video,
wherein obtaining and analyzing the context information at the pausing position of the video includes:
analyzing verbs of the question to determine a tense of the question;
in response to the tense of the question being a past tense, determining a scene base of the question to include a current scene and (N−1) scenes preceding the current scene, N being a size of the scene base for the question in the past tense or in a future tense;
in response to the tense of the question being a present tense, determining the scene base to include the current scene;
in response to the tense of the question being the future tense, determining the scene base to include the current scene and (N−1) scenes succeeding the current scene; and
obtaining and analyzing the supplementary materials of the video for the scene base according to the question.

### Claim 14 (depends on 13)

The device according to claim 13, wherein the processor is further configured to:
perform a language smoothing process on the answer to obtain a language smoothed answer; and
output the language smoothed answer.

### Claim 15 (depends on 12)

The device according to claim 12, wherein the processor is further configured to:
organize the supplementary materials into a plurality of passages; and
rank relevancy of each of the plurality of passages with the question, the relevancy being a passage score.

### Claim 16 (depends on 15)

The device according to claim 15, wherein the processor is further configured to:
organize the supplementary materials for each scene into three levels, wherein a first level includes a passage of a summary of each scene, a second level includes a passage of a screenplay of each scene, and a third level includes a passage of the closed caption of each scene or a passage of transcription of the audio track of each scene; and
index a passage describing each character by a character name.

### Claim 17

A context-aware system for answering a question about a video, comprising:
a terminal device configured to:
receive the question about the video that is paused at a pausing position;
send the question to a cloud server;
receive a language smoothed answer from the cloud server; and
output the answer; and
the cloud server configured to
receive the question from the terminal device;
obtain and analyze context information at the pausing position of the video, the context information including supplementary materials of the video;
automatically search the answer to the question based on the context information at the pausing position of the video;
perform a language smoothing process on the answer to obtain the language smoothed answer; and
send the language smoothed answer to the terminal device,
wherein obtaining and analyzing the context information at the pausing position of the video includes:
analyzing verbs of the question to determine a tense of the question;
in response to the tense of the question being a past tense, determining a scene base of the question to include a current scene and (N−1) scenes preceding the current scene, N being a size of the scene base for the question in the past tense or in a future tense;
in response to the tense of the question being a present tense, determining the scene base to include the current scene;
in response to the tense of the question being the future tense, determining the scene base to include the current scene and (N−1) scenes succeeding the current scene; and
obtaining and analyzing the supplementary materials of the video for the scene base according to the question.

### Claim 18 (depends on 17)

The system according to claim 17, wherein:
the terminal device communicates with the cloud server through HTTP API.