# US 2001/0025375 A1 · Browser for use in navigating a body of information, with particular application to browsing information represented by audiovisual data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2001/0025375 A1
- Document ID: US20010025375A1
- Status: Published application
- Kind: A1
- Application number: 09/866,956
- Filed: 2001-05-29
- Published: 2001-09-27
- Earliest claimed: 1996-12-05
- Inventors: Subutai Ahmad; Neal A. Bhadkamkar; Steve B. Cousins; Emanuel E. Farber; Paul A. Freiberger; Christopher D. Horner; Philippe P. Piernot; Brygg A. Ullmer
- Web page: https://patentgrep.com/patent/US20010025375A1

## CPC

- H04N 21/4622: {Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet (web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859)}
- G06F 16/738: Presentation of query results
- G06F 16/739: {in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames}
- G06F 16/7834: {using audio features}
- G06F 16/7844: {using original textual content or text extracted from visual content or transcript of audio data}
- G06F 16/785: {using colour or luminescence}
- G06F 16/951: Indexing; Web crawling techniques
- H04N 21/4782: Web browsing {, e.g. WebTV}
- H04N 21/8586: {by using a URL (processing chained hypermedia data for information retrieval G06F16/94; information retrieval from the Internet by using URLs G06F16/955; URL in broadcast information H04H20/93; Web-based protocols H04L67/02)}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

The invention facilitates and enhances review of a body of information (that can be represented by a set of audio data, video data, text data or some combination of the three), enabling the body of information to be quickly reviewed to obtain an overview of the content of the body of information and allowing flexibility in the manner in which the body of information is reviewed. In a particular application of the invention, the content of audiovisual news programs is acquired from a first set of one or more information sources (e.g., television news programs) and text news stories are acquired from a second set of one or more information sources (e.g., online news services or news wire services). In such a particular application, the invention can enable the user to access the news stories of audiovisual news programs in a random manner so that the user can move quickly among news stories or news programs. The invention can also enable the user to quickly locate news stories pertaining to a particular subject. Additionally, when the user is observing a particular news story in a news program, the invention can identify and display related news stories. The invention can also enable the user to control the display of the news programs by, for example, speeding up the display, causing a summary of one or more news stories to be displayed, or pausing the display of the news stories. Additionally, the invention can indicate to the user which news story is currently being viewed, as well as which news stories have previously been viewed.

## Claims

### Claim 1

A system for acquiring and reviewing a body of information, wherein the body of information includes a plurality of segments, each segment representing a defined set of information in the body of information, the system comprising:
means for acquiring data representing the body of information;
means for storing the acquired data;
first display means for generating a display of a first segment of the body of information from data that is part of the stored data;
means for comparing data representing a segment of the body of information to data representing a different segment of the body of information to determine whether, according to one or more predetermined criteria, the compared segments are related; and
second display means for generating a display of a portion of, or a representation of, a second segment of the body of information from data that is part of the stored data, wherein the second display means displays the portion or representation of the second segment in response to the display by the first display means of a first segment to which the second segment is related.

### Claim 2 (depends on 1)

A system as in claim 1, wherein the second display means displays the portion or representation of the second segment substantially coextensive in time with the display of the related first segment by the first display means.

### Claim 3 (depends on 1)

A system as in claim 1, wherein:
at least a portion of the body of information is represented by audiovisual data;
the first segment is represented by audiovisual data;
the first display means displays an audiovisual display of the first segment; and
the second segment is represented by audiovisual data.

### Claim 4 (depends on 3)

A system as in claim 3, further comprising means for selecting a segment for which a portion or representation is displayed by the second display means, wherein selection of such segment causes the first display means to display an audiovisual display of the selected segment.

### Claim 5 (depends on 1)

A system as in claim 1, wherein:
at least a portion of the body of information is represented by audiovisual data;
the first display means displays an audiovisual display of the first segment; and
the second display means displays a text display of a portion or representation of the second segment.

### Claim 6 (depends on 1)

A system as in claim 1, wherein:
the first display means is an analog display device; and
the second display means is a digital display device.

### Claim 7 (depends on 1)

A system as in claim 1, wherein:
the first display means is a television; and
the second display means is a computer display monitor.

### Claim 8 (depends on 1)

A system as in claim 1, further comprising means for identifying the subject matter content of a segment of the body of information, wherein the means for comparing further comprises means for determining the similarity of the subject matter content of a segment to the subject matter content of a different segment, the predetermined criteria including a predefined degree of similarity with respect to which the relatedness of the compared segments is determined.

### Claim 9 (depends on 8)

A system as in claim 8, wherein the means for determining the similarity of the subject matter of segments further comprises means for performing a relevance feedback method.

### Claim 10 (depends on 1)

A system as in claim 1, wherein the means for acquiring data further comprises means for acquiring television broadcast signals.

### Claim 11 (depends on 1)

A system as in claim 1, wherein the means for acquiring data further comprises means for acquiring radio broadcast signals.

### Claim 12 (depends on 1)

A system as in claim 1, wherein the means for acquiring data further comprises means for acquiring computer-readable data files over a computer network from an information providing site that is part of that network.

### Claim 13 (depends on 1)

A system as in claim 1, wherein the means for acquiring data further comprises:
means for acquiring television broadcast signals; and
means for acquiring computer-readable data files over a computer network from an information providing site that is part of that network.

### Claim 14 (depends on 13)

A system as in claim 13, wherein:
the first segment is represented by data produced from the television broadcast signals; and
the second segment is represented by data from the computer-readable data files.

### Claim 15 (depends on 1)

A system as in claim 1, further comprising means for identifying an instruction from a user to begin displaying at least some of the body of information, wherein the first display means begins displaying a segment in response to the user instruction.

### Claim 16 (depends on 1)

A system as in claim 1, wherein the first and second display means are physically separate.

### Claim 17 (depends on 1)

A system as in claim 1, wherein the means for storing the acquired data, the first display means and the second display means are interconnected to a conventional computer bus that enables the devices to communicate with each other such that the devices do not require wire communication over network communication lines to communicate with each other.

### Claim 18

A system for reviewing a body of audiovisual information that can vary with time, the system comprising:
means for displaying the audiovisual information; and
means for controlling operation of the system, the means for controlling being physically separate from the means for displaying, the means for controlling including a graphical user interface for enabling specification of control instructions.

### Claim 19 (depends on 18)

A system as in claim 18, wherein the means for controlling is portable.

### Claim 20 (depends on 19)

A system as in claim 19, further comprising means for 2-way wireless communication between the means for displaying and the means for controlling.

### Claim 21 (depends on 18)

A system as in claim 18, wherein the graphical user interface includes a playback control region for enabling specification of control instructions that control the manner in which the audiovisual information is displayed by the means for displaying.

### Claim 22 (depends on 21)

A system as in claim 21, wherein:
the audiovisual information includes a plurality of segments, each segment representing a defined set of information in the audiovisual information;
the playback control region includes an interface that enables selection of one of a plurality of subject matter categories; and
the means for controlling further comprises:
means for identifying the subject matter category of a segment; and
means for controlling the system to display each of the segments that correspond to a selected subject matter category.

### Claim 23 (depends on 21)

A system as in claim 21, wherein:
the playback control region includes an interface that enables variation of the apparent display rate at which the audiovisual information is displayed; and
the means for controlling further comprises means for controlling the means for displaying to cause the audiovisual information to be displayed at an apparent display rate other than a normal display rate.

### Claim 24 (depends on 21)

A system as in claim 21, wherein:
the audiovisual information includes a plurality of segments, each segment representing a defined set of information in the audiovisual information;
the playback control region includes an interface that enables specification of the display of a summary of a segment of the audiovisual information; and
the means for controlling further comprises:
means for summarizing a segment of the audiovisual information; and
means for controlling the means for displaying to cause the summary of the segment to be displayed.

### Claim 25 (depends on 21)

A system as in claim 21, wherein:
the playback control region includes:
an interface that enables specification of a pause instruction; and
an interface that enables specification of a resume instruction; and
the means for controlling further comprises:
means for identifying a pause instruction from a user;
means for controlling the means for displaying to stop the display of the audiovisual information in response to identification of the pause instruction;
means for identifying a resume instruction from a user; and
means for controlling the means for displaying to restart the display of the audiovisual information in response to identification of the resume instruction, wherein the audiovisual information is displayed at an accelerated rate that is greater than the rate at which the audiovisual information was previously displayed, such accelerated rate continuing until the display of the audiovisual information coincides with the display that would have appeared had the display not been paused.

### Claim 26 (depends on 21)

A system as in claim 21, wherein:
the audiovisual information includes a plurality of segments, each segment representing a defined set of information in the audiovisual information;
the playback control region includes an interface that enables specification of a termination instruction; and
the means for controlling further comprises:
means for identifying a termination instruction from a user; and
means for terminating display of the segment currently being displayed and beginning display of a new segment in response to identification of a termination instruction.

### Claim 27 (depends on 21)

A system as in claim 21, wherein:
the audiovisual information includes a plurality of segments, each segment representing a defined set of information in the audiovisual information;
the playback control region includes an interface that enables specification of a repeat instruction; and
the means for controlling further comprises:
means for identifying a repeat instruction from a user; and
means for repeating the display of the segment currently being displayed in response to identification of a repeat instruction.

### Claim 28 (depends on 18)

A system as in claim 18, wherein the graphical user interface includes a map region for providing a description of the subject matter content of the audiovisual information and for enabling specification of control instructions that enable navigation within the audiovisual information.

### Claim 29 (depends on 28)

A system as in claim 28, wherein:
the audiovisual information includes a plurality of segments, each segment representing a defined set of information in the audiovisual information; and
the map region further identifies a segment of the audiovisual information that is currently being displayed.

### Claim 30 (depends on 28)

A system as in claim 28, wherein:
the audiovisual information includes a plurality of segments, each segment representing a defined set of information in the audiovisual information; and
the map region further identifies each segment of the audiovisual information that has previously been displayed.

### Claim 31 (depends on 18)

A system as in claim 18, wherein:
the audiovisual information includes a plurality of segments, each segment representing a defined set of information in the audiovisual information; and
the graphical user interface includes a related information region for displaying a portion of, or a representation of, a segment that is related to a segment being displayed by the means for displaying.

### Claim 32 (depends on 18)

A system as in claim 18, wherein:
the audiovisual information includes a plurality of segments, each segment representing a defined set of information in the audiovisual information; and
the graphical user interface includes a secondary information display region for displaying a secondary information segment that is related to a segment of the audiovisual information that is being displayed by the means for displaying.

### Claim 33 (depends on 18)

A system as in claim 18, wherein the audiovisual information further comprises the content from one or more news programs.

### Claim 34

A system for reviewing a body of information, the body of information including a first portion that is represented by audiovisual data that can vary with time and a second portion that is represented by text data, comprising:
a first display device for displaying the first portion of information, the first display device particularly adapted for generation of a display from time-varying audiovisual data; and
a second display device for displaying the second portion of information, the second display device particularly adapted for generation of a display from text data.

### Claim 35

A method for acquiring and reviewing a body of information, wherein the body of information includes a plurality of segments, each segment representing a defined set of information in the body of information, the method comprising the steps of:
acquiring data representing the body of information;
storing the acquired data;
generating a display of a first segment of the body of information from data that is part of the stored data;
comparing data representing a segment of the body of information to data representing a different segment of the body of information to determine whether, according to one or more predetermined criteria, the compared segments are related; and
generating a display of a portion of, or a representation of, a second segment of the body of information from data that is part of the stored data, wherein the display of the portion or representation of the second segment is generated in response to the display of a first segment to which the second segment is related.

### Claim 36

A method for categorizing according to subject matter an uncategorized segment of a body of information that includes a plurality of segments, each segment representing a defined set of information in the body of information, one or more segments of the body of information having previously been categorized by identifying each of the one or more segments with one or more subject matter categories, the method comprising the steps of:
determining the degree of similarity between the subject matter content of the uncategorized segment and the subject matter content of each of the previously categorized segments;
identifying one or more of the previously categorized segments as relevant to the uncategorized segment based upon the determined degrees of similarity of subject matter content between the uncategorized segment and the previously categorized segments; and
selecting one or more subject matter categories with which to identify the uncategorized segment based upon the subject matter categories used to identify the relevant previously categorized segments.

### Claim 37 (depends on 36)

A method as in claim 36, wherein the step of determining the degree of similarity is accomplished using a relevance feedback method.

### Claim 38 (depends on 36)

A method as in claim 36, wherein the step of identifying one or more of the previously categorized segments as relevant to the uncategorized segment further comprises the steps of:
identifying a plurality of the previously categorized segments that are the most similar to the uncategorized segment;
determining the degree of similarity between each of the plurality of previously categorized segments and each other of the plurality of previously categorized segments;
for each pair of previously categorized segments of the plurality of previously categorized segments having greater than a predefined degree of similarity, eliminating one of the pair of previously categorized segments from the plurality of previously categorized segments, wherein the previously categorized segment or segments remaining after the step of eliminating are similar and distinct previously categorized segments; and
identifying one or more of the similar and distinct previously categorized segments as relevant previously categorized segments.

### Claim 39 (depends on 36)

A method as in claim 36, wherein the step of selecting one or more subject matter categories further comprises selecting the most frequently occurring subject matter category or categories associated with the relevant previously categorized segments.

### Claim 40 (depends on 36)

A system as in claim 36, wherein the uncategorized segment has been acquired from a first data source and the previously categorized segment or segments have been acquired from a second data source that is different than the first data source.

### Claim 41 (depends on 40)

A system as in claim 40, wherein:
the data acquired from the first data source are television or radio broadcast signals; and
the data acquired from the second data source are computer-readable data files.

### Claim 42

A method for determining whether a first set of information represented by a set of data of a first type is relevant to a second set of information represented by a set of data of a second type, the first and second sets of information being different from each other, the method comprising the steps of:
deriving a set of data of the second type from the set of data of the first type, the derived set of data of the second type also being representative of the first set of information;
determining the degree of similarity between the set of data of the second type representing the second set of information and the derived set of data of the second type representing the first set of information; and
determining whether the first set of information is relevant to the second set of information based upon the degree of similarity between the set of data of the second type representing the second set of information and the derived set of data of the second type representing the first set of information.

### Claim 43 (depends on 42)

A method as in claim 42, wherein the first type of data is audiovisual data and the second type of data is text data.

### Claim 44 (depends on 43)

A method as in claim 43, wherein the step of determining the degree of similarity is accomplished using a relevance feedback method.

### Claim 45 (depends on 42)

A method as in claim 42, wherein a plurality of sets of information, each different from the other sets of the plurality of sets of information, are each represented by an associated set of data of the second type, the method enabling determination of which, if any, of the plurality of sets of information represented by a set of data of the second type are relevant to the first set of information represented by the set of data of the first type, the method further comprising the steps of:
determining the degree of similarity between each set of data of the second type representing one of the plurality of sets of information and the derived set of data of the second type representing the first set of information;
identifying which, if any, of the sets of data of the second type representing one of the plurality of sets of information have greater than a predefined degree of similarity to the derived set of data of the second type representing the first set of information, the sets of data of the second type so identified being termed similar sets of data of the second type;
determining the degree of similarity between each similar set of data of the second type and each other similar set of data of the second type;
for each pair of similar sets of data of the second type having greater than a predefined degree of similarity, eliminating one of the pair of similar sets of data of the second type from the set of similar sets of data of the second type, wherein the set or sets of similar data of the second type remaining after the step of eliminating are similar and distinct sets of data of the second type; and
identifying the set or sets of information corresponding to one or more of the similar and distinct sets of data of the second type as relevant to the second set of information.

### Claim 46 (depends on 45)

A method as in claim 45, wherein the step of identifying the relevant set or sets of information further comprises identifying no more than a predetermined number of relevant sets of information, the predetermined number of relevant sets of information corresponding to the sets of data of the second type having the greatest degree of similarity to the derived set of data of the second type.

### Claim 47

A method for identifying the boundaries of segments in a body of information, each segment comprising a contiguous related set of information in the body of information, wherein the body of information is represented by at least a set of text data and a set of video data, the method comprising the steps of:
performing a coarse partitioning method, the coarse partitioning method further comprising the steps of:
identifying time-stamped markers in the set of text data; and
determining approximate segment boundaries within the body of information as the times of occurrence of the time-stamp markers;
for each approximate segment boundary, specifying a range of time that includes the time of occurrence of the approximate segment boundary;
extracting subsets of video data from the set of video data that occur during the specified ranges of time;
performing a fine partitioning method to identify one or more breaks in the set of video data; and
selecting the best break that occurs in each subset of video data, the time of occurrence of the best break in each subset being designated as a boundary of a segment in the body of information.

### Claim 48 (depends on 47)

A method as in claim 47, wherein the step of performing a fine partitioning method further comprises identifying the best breaks using a process that includes scene break identification.

### Claim 49 (depends on 47)

A method as in claim 47, wherein the step of fine partitioning is performed on the entire set of video data to identify all of the breaks in the set of video data.

### Claim 50 (depends on 47)

A method as in claim 47, wherein the step of fine partitioning is performed only on the subsets of video data to identify only breaks that occur in the subsets.

### Claim 51 (depends on 47)

A method as in claim 47, wherein the best break of each subset is determined according to the criteria of the fine partitioning method used.

### Claim 52 (depends on 47)

A method as in claim 47, wherein the best break of each subset is the break occurring closest in time to the time of occurrence of the segment boundary in the text data that corresponds to that subset.

### Claim 53 (depends on 47)

A method as in claim 47, wherein the body of information is represented by a set of text data, a set of audio data and a set of video data, the method further comprising the steps of:
ascertaining a synchronization of the audio data and the video data; and
determining the location of the segment boundaries in the set of audio data using the previously determined location of the segment boundaries in the set of video data and the synchronization of the audio data and video data.

### Claim 54

A method for identifying the boundaries of segments in a body of information, each segment comprising a contiguous related set of information in the body of information, wherein the body of information is represented by a set of text data, a set of video data, and a set of audio data the method comprising the steps of:
performing a coarse partitioning method, the coarse partitioning method further comprising the steps of:
identifying time-stamped markers in the set of text data; and
determining approximate segment boundaries within the body of information as the times of occurrence of the time-stamp markers;
for each approximate segment boundary, specifying a range of time that includes the time of occurrence of the approximate segment boundary;
extracting subsets of audio data from the set of audio data that occur during the specified ranges of time;
performing a fine partitioning method to identify one or more breaks in the set of audio data;
selecting the best break that occurs in each subset of audio data, the time of occurrence of the best break in each subset being designated as a boundary of a segment in the body of information;
ascertaining a synchronization of the audio data and the video data; and
determining the location of the segment boundaries in the set of video data using the previously determined location of the segment boundaries in the set of audio data and the synchronization of the audio data and video data.

### Claim 55 (depends on 54)

A method as in claim 54, wherein the step of performing fine partitioning further comprises identifying the best breaks using a process that includes pause recognition.

### Claim 56 (depends on 54)

A method as in claim 54, wherein the step of performing fine partitioning further comprises identifying the best breaks using a process that includes voice recognition.

### Claim 57 (depends on 54)

A method as in claim 54, wherein the step of performing fine partitioning further comprises identifying the best breaks using a process that includes word recognition.

### Claim 58 (depends on 54)

A method as in claim 54, wherein the step of performing fine partitioning further comprises identifying the best breaks using a process that includes music recognition.

### Claim 59

A computer readable medium encoded with one or more computer programs for enabling acquisition and review of a body of information, wherein the body of information includes a plurality of segments, each segment representing a defined set of information in the body of information, comprising:
instructions for acquiring data representing the body of information;
instructions for storing the acquired data;
instructions for generating a display of a first segment of the body of information from data that is part of the stored data;
instructions for comparing data representing a segment of the body of information to data representing a different segment of the body of information to determine whether, according to one or more predetermined criteria, the compared segments are related; and
instructions for generating a display of a portion of, or a representation of, a second segment of the body of information from data that is part of the stored data, wherein the display of the portion or representation of the second segment is generated in response to the display of a first segment to which the second segment is related.

### Claim 60

A computer readable medium encoded with one or more computer programs for enabling categorization according to subject matter of an uncategorized segment of a body of information that includes a plurality of segments, each segment representing a defined set of information in the body of information, one or more segments having previously been categorized by identifying each of the one or more segments with one or more subject matter categories, comprising:
instructions for determining the degree of similarity between the subject matter content of the uncategorized segment and the subject matter content of each of the previously categorized segments;
instructions for identifying one or more of the previously categorized segments as relevant to the uncategorized segment based upon the determined degrees of similarity of subject matter content between the uncategorized segment and the previously categorized segments; and
instructions for selecting one or more subject matter categories with which to identify the uncategorized segment based upon the subject matter categories used to identify the relevant previously categorized segments.

### Claim 61

A computer readable medium encoded with one or more computer programs for enabling determination of whether a first set of information represented by a set of data of a first type is relevant to a second set of information represented by a set of data of a second type, the first and second sets of information being different from each other, comprising:
instructions for deriving a set of data of the second type from the set of data of the first type, the derived set of data of the second type also being representative of the first set of information;
instructions for determining the degree of similarity between the set of data of the second type representing the second set of information and the derived set of data of the second type representing the first set of information; and
instructions for determining whether the first set of information is relevant to the second set of information based upon the degree of similarity between the set of data of the second type representing the second set of information and the derived set of data of the second type representing the first set of information.

### Claim 62

A computer readable medium encoded with one or more computer programs for enabling identification of the boundaries of segments in a body of information, each segment comprising a contiguous related set of information in the body of information, wherein the body of information is represented by at least a set of text data and a set of video data, comprising:
instructions for performing a coarse partitioning method, the coarse partitioning instructions further comprising:
instructions for identifying time-stamped markers in the set of text data; and
instructions for determining approximate segment boundaries within the body of information as the times of occurrence of the time-stamp markers;
instructions for specifying, for each approximate segment boundary, a range of time that includes the time of occurrence of the approximate segment boundary;
instructions for extracting subsets of video data from the set of video data that occur during the specified ranges of time;
instructions for performing a fine partitioning method to identify one or more breaks in the set of video data; and
instructions for selecting the best break that occurs in each subset of video data, the time of occurrence of the best break in each subset being designated as a boundary of a segment in the body of information.
