# US 7,610,546 B1 · Document processing apparatus having capability of controlling video data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 7,610,546 B1
- Document ID: US7610546B1
- Status: Granted
- Kind: B1
- Application number: 09/624,580
- Filed: 2000-07-24
- Granted: 2009-10-27
- Earliest claimed: 1999-08-02
- Assignees: Sony Corporation
- Inventors: Katashi Nagao; Kazuyuki Marukawa
- Web page: https://patentgrep.com/patent/US7610546B1

## CPC

- H04N 5/76: Television signal recording
- G06F 16/4393: {Multimedia presentations, e.g. slide shows, multimedia albums}
- G06F 16/78: Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F 40/117: Tagging; Marking up (details of markup languages G06F40/143); Designating a block; Setting of attributes (style sheets, e.g. eXtensible Stylesheet Language Transformation [XSLT], G06F40/154)
- G06F 40/14: Tree-structured documents (parsing G06F40/205; validation G06F40/226)
- G06F 40/169: Annotation, e.g. comment data or footnotes
- H04N 5/445: for displaying additional information (H04N5/50 takes precedence)
- H04N 21/43074: {of additional data with content streams on the same device, e.g. of EPG data or interactive icon with a TV programme}
- H04N 21/4316: {for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window}
- H04N 21/440236: {by media transcoding, e.g. video is transformed into a slideshow of still pictures, audio is converted into text}
- H04N 21/47217: {for controlling playback functions for recorded or on-demand content, e.g. using progress bars, mode or play-point indicators or bookmarks (specific graphical features in visual interfaces H04N21/4312)}
- H04N 21/4886: {for displaying a ticker, e.g. scrolling banner for news, stock exchange, weather data}
- H04N 21/8543: using a description language, e.g. Multimedia and Hypermedia information coding Expert Group [MHEG], eXtensible Markup Language [XML] {(information retrieval of semistructured data, the underlying structure being taken into account, e.g. mark-up language structure data G06F16/80)}
- H04N 9/8205: {involving the multiplexing of an additional signal and the colour video signal}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A document processing apparatus is disclosed that has a capability of outputting video data related to document data. When electronic document data including information (video tag) specifying video data is input, video data related to that electronic document data is detected. The video data related to the electronic document data is output in synchronization with or independently of the output of the electronic document data thereby presenting to a user not only the document but also the video data related to the document.

## Claims

### Claim 1

A document processing apparatus comprising:
an input device;
a display device;
a processor;
a memory device which stores a plurality of instructions, which when executed by the processor, cause the processor to operate with the display device and the input device to:
(a) detect video data designation information attached to electronic document data, the electronic document data including: (i) a first element having a first central activation value used to generate an index; (ii) a second element having a second central activation value used to generate said index; and (iii) read out audio attribute information;
(b) generate a summary of said electronic document data, wherein said generation of said summary includes spreading said first central activation value to said second central activation value;
(c) select video data in accordance with said detected video data designation information;
(d) store a categorization model, the categorization model including a plurality of data categories;
(e) create an automatic categorization based on any one of said video data and electronic document data in accordance with the categorization model;
(f) update the categorization model with the automatic categorization;
(g) control an output of said summary of said electronic document data such that said summary of said electronic data being output is automatically progressed based on at least one of a size of a display area and a length of time displayed;
(h) control an output of said selected video data in correspondence with the output of said summary of said electronic document data such that said selected video data being output is output in synchronization with said progress of the said operation of outputting said summary of said electronic data;
(i) control an output of a read out audio based on read out audio attribute information in said electronic document to synthesize said read out audio; and
(j) automatically terminate the output of said video data upon completion of the outputting of said summary of said electronic document data regardless of whether an end of the video data has been reached.

### Claim 2 (depends on 1)

The document processing apparatus of claim 1, wherein when executed by the processor, said plurality of instructions cause the processor to operate with the display device to control the outputting of said video data, in synchronization with the progress of the operation of reading aloud said electronic data.

### Claim 3 (depends on 1)

The document processing apparatus of claim 1, wherein said video data designation information is attached in the form of tag data to said electronic document.

### Claim 4 (depends on 1)

The document processing apparatus of claim 1, wherein said video data is motion image data.

### Claim 5 (depends on 1)

The document processing apparatus of claim 1, wherein said video data is still image data.

### Claim 6 (depends on 1)

The document processing apparatus of claim 1, wherein before a portion of said electronic document data being output is progressed to not be perceivable, a corresponding portion of said video data is output.

### Claim 7 (depends on 1)

The document processing apparatus of claim 1, wherein a portion of said video data corresponding to a portion of said electronic document data is not output before said portion of said electronic document data is progressed to be perceivable.

### Claim 8 (depends on 1)

The document processing apparatus of claim 1, wherein the electronic document data being output is displayed at least partly as scrolling text.

### Claim 9 (depends on 8)

The document processing apparatus of claim 8, wherein when executed by the processor, said plurality of instructions cause the processor to operate with the display device to output audio data in synchronization with the scrolling of the scrolling text, the audio output including a reading of the scrolling text.

### Claim 10 (depends on 1)

The document processing apparatus of claim 1, wherein the categorization model is created on the basis of the categorization that has been manually performed by the user.

### Claim 11 (depends on 1)

The document processing apparatus of claim 1, wherein when executed by the processor, said plurality of instructions cause the processor to operate with the display device to create the automatic categorization using a word sense relevance value.

### Claim 12 (depends on 1)

The document processing apparatus of claim 1, wherein when executed by the processor, said plurality of instructions cause the processor to operate with the display device to control the outputting of the read out audio by embedding attribute information into a tag file.

### Claim 13 (depends on 1)

The document processing apparatus of claim 1, wherein the attribute information includes pronunciation data.

### Claim 14

A document processing method comprising:
detecting video data designation information attached to electronic document data, the electronic document data including: (i) a first element having a first central activation value used to generate an index; (ii) a second element having a second central activation value used to generate said index; and (iii) read out audio attribute information;
generating a summary of said electronic document data, wherein said generation of said summary includes spreading said first central activation value to said second central activation value;
selecting video data in accordance with said detected video data designation information;
storing a categorization model including a plurality of data categories;
creating an automatic categorization based on any one of said video data and electronic document data in accordance with the categorization model;
updating the categorization model with the automatic categorization;
controlling an output of said summary of said electronic document data such that said summary of said electronic data being output is automatically progressed based on at least one of a size of a display area and a length of time displayed;
controlling an output of said selected video data in correspondence with the output of said summary of said electronic document data such that said selected video data being output is output in synchronization with said progress of the said operation of outputting said summary of said electronic data;
controlling an output of a read out audio based on read out audio attribute information in said electronic document to synthesize said read out audio; and
automatically terminate the output of said video data upon completion of the outputting of said summary of said electronic document data regardless of whether an end of the video data has been reached.

### Claim 15 (depends on 14)

The document processing method of claim 14, further comprising:
controlling the output of a voice reading aloud said electronic document data; and
controlling the outputting of said video data, in synchronization with the progress of the operation of reading aloud said electronic data.

### Claim 16 (depends on 14)

The document processing method of claim 14, wherein said video data designation information is attached in the form of tag data to said electronic document.

### Claim 17 (depends on 14)

The document processing method of claim 14, wherein said video data is motion image data.

### Claim 18 (depends on 14)

The document processing method of claim 14, wherein said video data is still image data.

### Claim 19 (depends on 14)

The document processing method of claim 14, wherein before a portion of said electronic document data being output is progressed to not be perceivable, a corresponding portion of said video data is output.

### Claim 20 (depends on 14)

The document processing method of claim 14, wherein a portion of said video data corresponding to a portion of said electronic document data is not output before said portion of said electronic document data is progressed to be perceivable.

### Claim 21 (depends on 14)

The document processing method of claim 14, wherein creating the automatic categorization uses a word sense relevance value.

### Claim 22 (depends on 14)

The document processing method of claim 14, wherein controlling the outputting of the read out audio controls the outputting of said read out audio by embedding attribute information into a tag file.

### Claim 23 (depends on 14)

The document processing method of claim 14, wherein the attribute information includes pronunciation data.

### Claim 24

A storage medium including a computer-controllable program stored thereon, said program comprising the steps of:
detecting video data designation information attached to electronic document data, the electronic document data including: (i) a first element having a first central activation value used to generate an index; (ii) a second element having a second central activation value used to generate said index; and (iii) read out audio attribute information;
generating a summary of said electronic document data, wherein said generation of said summary includes spreading said first central activation value to said second central activation value;
selecting video data in accordance with said detected video data designation information;
storing a categorization model including a plurality of data categories;
creating an automatic categorization based on any one of said video data and electronic document data in accordance with the categorization model;
updating the categorization model with the automatic categorization;
controlling an output of said summary of said electronic document data such that said summary of said electronic data being output is automatically progressed based on at least one of a size of a display area and a length of time displayed;
controlling an output of said selected video data in correspondence with the output of said summary of said electronic document data such that said selected video data being output is output in synchronization with said progress of the said operation of outputting said summary of said electronic data;
controlling an output of a read out audio based on read out audio attribute information in said electronic document to synthesize said read out audio; and
automatically terminate the output of said video data upon completion of the outputting of said summary of said electronic document data regardless of whether an end of the video data has been reached.

### Claim 25 (depends on 24)

The storage medium including a computer-controllable program stored thereon, according to claim 24, said program further comprising the steps of:
controlling the output of a voice reading aloud said electronic document data; and
controlling the outputting of said video data, in synchronization with the progress of the operation of reading aloud said electronic data.

### Claim 26 (depends on 24)

The storage medium including a computer-controllable program stored thereon, according to claim 24, wherein said video data designation information is attached in the form of tag data to said electronic document.

### Claim 27 (depends on 24)

The storage medium including a computer-controllable program stored thereon, according to claim 24, wherein said video data is motion image data.

### Claim 28 (depends on 24)

The storage medium including a computer-controllable program stored thereon, according to claim 24, wherein said video data is still image data.

### Claim 29 (depends on 24)

The storage medium including a computer-controllable program stored thereon, according to claim 24, wherein before a portion of said electronic document data being output is progressed to not be perceivable, a corresponding portion of said video data is output.

### Claim 30 (depends on 24)

The storage medium including a computer-controllable program stored thereon, according to claim 24, wherein a portion of said video data corresponding to a portion of said electronic document data is not output before said portion of said electronic document data is progressed to be perceivable.

### Claim 31 (depends on 24)

The storage medium including a computer-controllable program stored thereon according to claim 24, said program further comprising the step of creating the automatic categorization uses a word sense relevance value.

### Claim 32 (depends on 24)

The storage medium including a computer-controllable program stored thereon according to claim 24, wherein said step of controlling the outputting of the read out audio controls the outputting of said read out audio by embedding attribute information into a tag file.

### Claim 33 (depends on 24)

The storage medium including a computer-controllable program stored thereon according to claim 24, wherein the attribute information includes pronunciation data.

### Claim 34

A document processing apparatus comprising:
a detection circuit for detecting video data designation information attached to electronic document data, the electronic document data including: (i) a first element having a first central activation value used to generate an index; (ii) a second element having a second central activation value used to generate said index; and (iii) read out audio attribute information;
a generating circuit for generating a summary of said electronic document data, wherein said generation of said summary includes spreading said first central activation value to said second central activation value;
a video data selection circuit for selecting video data in accordance with said detected video data designation information;
a storage device configured to store a categorization model, the categorization model including a plurality of data categories;
a data categorization circuit for creating an automatic categorization based on any one of said video data and electronic document data in accordance with the categorization model and updating the categorization model with the automatic categorization;
a document output control circuit for controlling an output of said summary of said electronic document data such that said summary of said electronic data being output is automatically progressed based on at least one of a size of a display area and a length of time displayed;
a video output control circuit for controlling an output of said selected video data in correspondence with the output of said summary of said electronic document data such that said selected video data being output is output in synchronization with said progress of the said operation of outputting said summary of said electronic data;
an audio output circuit for controlling an output of a read out audio based on read out audio attribute information in said electronic document to synthesize said read out audio; and
wherein said video output control circuit automatically terminates the output of said video data upon completion of the outputting of said summary of said electronic document data regardless of whether an end of the video data has been reached.

### Claim 35 (depends on 34)

The document processing apparatus of claim 34, further comprising a voice output control circuit for controlling the output of a voice reading aloud said electronic document data,
wherein said video output control circuit controls the outputting of said video data, in synchronization with the progress of the operation of reading aloud said electronic data.

### Claim 36 (depends on 34)

The document processing apparatus of claim 34, wherein said video data designation information is attached in the form of tag data to said electronic document.

### Claim 37 (depends on 34)

The document processing apparatus of claim 34, wherein said video data is motion image data.

### Claim 38 (depends on 34)

The document processing apparatus of claim 34, wherein said video data is still image data.

### Claim 39 (depends on 34)

The document processing apparatus of claim 34, wherein before a portion of said electronic document data being output is progressed to not be perceivable, a corresponding portion of said video data is output.

### Claim 40 (depends on 34)

The document processing apparatus of claim 34, wherein a portion of said video data corresponding to a portion of said electronic document data is not output before said portion of said electronic document data is progressed to be perceivable.

### Claim 41 (depends on 24)

The document processing apparatus of claim 24, wherein creating the automatic categorization uses a word sense relevance value.

### Claim 42 (depends on 34)

The document processing apparatus of claim 34, wherein audio output control circuit controls the outputting of the read out audio by embedding attribute information into a tag file.

### Claim 43 (depends on 34)

The document processing apparatus of claim 34, wherein the attribute information includes pronunciation data.