# US 2019/0171760 A1 · System, summarization apparatus, summarization system, and method of controlling summarization apparatus, for acquiring summary information

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0171760 A1
- Document ID: US20190171760A1
- Status: Published application
- Kind: A1
- Application number: 16/199,591
- Filed: 2018-11-26
- Published: 2019-06-06
- Earliest claimed: 2017-12-01
- Assignees: Canon Kabushiki Kaisha
- Applicants: Canon Kabushiki Kaisha
- Inventors: Motoki Ikeda
- Web page: https://patentgrep.com/patent/US20190171760A1

## CPC

- G06F 16/345: {Summarisation for human users}
- G06F 16/313: {Selection or weighting of terms for indexing}
- G06F 18/25: Fusion techniques
- G06V 10/80: Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level (multimodal speaker identification or verification G10L17/10)
- G06V 20/62: Text, e.g. of license plates, overlay texts or captions on TV images
- G10L 15/08: Speech classification or search
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 15/1822: {Parsing for meaning understanding}
- G10L 2015/088: {Word spotting}
- G10L 2015/225: {Feedback of the input speech}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A system that outputs information generated by summarizing contents of voices and images as texts. A CPU of the system performs, according to a program stored in a memory, recording voice data and captured image data, generating first text information by performing speech recognition on the acquired voice data, generating second text information by performing character recognition on the acquired image data, and generating summary text information smaller in the number of characters than the first text information and the second text information, based on the first text information and the second text information, according to a predetermined criterion.

## Claims

### Claim 1

A system that outputs information generated by summarizing contents of voice and images, as texts, including:
a memory that stores a program, and
at least one processor that performs, according to the program, processing comprising:
obtaining recorded voice data and captured image data;
generating first text information by performing speech recognition on the obtained voice data;
generating second text information by performing character recognition on the obtained image data; and
generating summary text information corresponding to a summary of the obtained voice data and the obtained image data, based on the first text information generated by the speech recognition and the second text information generated by the character recognition.

### Claim 2 (depends on 1)

The system according to claim 1, wherein the generating of the summary text information comprises:
selecting at least one content from the first text information generated by the speech recognition and selecting at least one content from the second text information generated by the character recognition; and
merging the at least one content selected from the first text information and the at least one content selected from the second text information, into the summary text information.

### Claim 3 (depends on 2)

The system according to claim 2, wherein the merging does not merge, into the summary text information, any contents included in the first text information and the second text information that are not selected by the selecting.

### Claim 4 (depends on 1)

The system according to claim 1, wherein the processing comprises:
receiving at least one title of an agenda from a user,
wherein the generating of the summary text information generates the summary text information in association with the received title.

### Claim 5 (depends on 1)

The system according to claim 1, wherein the summary text information is smaller in the number of characters than the first text information and the second text information.

### Claim 6 (depends on 1)

The system according to claim 1, wherein the generating of the summary text information generates the summary text information based on the first text information generated by the speech recognition and the second text information generated by the character recognition, according to a predetermined criterion.

### Claim 7 (depends on 6)

The system according to claim 6, wherein when generating the summary text information, the at least one processor selects one or more texts used to generate the summary text information out of the texts included in the first text information and the texts included in the second text information, according to the predetermined criterion.

### Claim 8 (depends on 7)

The system according to claim 7, wherein the predetermined criterion causes a text including a word included in the first text information generated based on the voice data to be selected as a text used to generate the summary text information out of the texts included in the second text information generated based on the image data.

### Claim 9 (depends on 8)

The system according to claim 8, wherein the predetermined criterion is the appearance frequency of a word.

### Claim 10 (depends on 1)

The system according to claim 1, wherein the voice data and the image data are obtained by performing voice recording and image capturing in the same meeting.

### Claim 11

A summarization apparatus comprising:
a first generation unit configured to generate first text information by converting voice data to texts;
a second generation unit configured to generate second text information by converting image data to texts;
an integration unit configured to integrate the first text information generated by the first generation unit and the second text information generated by the second generation unit to thereby generate integrated text information formed by at least one text; and
a summarization unit configured to generate summary information by summarizing the integrated text information integrated by the integration unit.

### Claim 12 (depends on 11)

The summarization apparatus according to claim 11, wherein the summarization unit generates the summary information by determining a text to be extracted out of the integrated text information, and extracting the determined text from the integrated text information.

### Claim 13 (depends on 12)

The summarization apparatus according to claim 12, wherein the summarization unit determines a text to be extracted based on the appearance frequency of each word included in the integrated text information, and extracts the determined text from the integrated text information.

### Claim 14 (depends on 12)

The summarization apparatus according to claim 12, wherein the summarization unit extracts all of texts generated based on the second text information out of the integrated text information.

### Claim 15 (depends on 12)

The summarization apparatus according to claim 12, wherein the summarization unit extracts a text including a predetermined word from the integrated text information.

### Claim 16 (depends on 12)

The summarization apparatus according to claim 12, wherein the summarization unit determines a text to be extracted by giving a weight to a text including a word included in the second text information out of the integrated text information, and extracts the determined text from the integrated text information.

### Claim 17 (depends on 11)

The summarization apparatus according to claim 11, wherein the summarization unit summarizes the integrated text information for each item to be discussed.

### Claim 18 (depends on 17)

The summarization apparatus according to claim 17, wherein a time period is associated with each item of agenda,
wherein a generation time is associated with each text included in the integrated text information, and
wherein the summarization unit generates the summary information such that a text generated during the time period associated with each item of agenda is associated with the item of agenda.

### Claim 19 (depends on 11)

The summarization apparatus according to claim 11, further comprising a creation unit configured to create minutes based on the integrated text information.

### Claim 20 (depends on 11)

The summarization apparatus according to claim 11, wherein the first generation unit generates the first text information by performing speech recognition on the voice data.

### Claim 21 (depends on 11)

The summarization apparatus according to claim 11, wherein the second generation unit generates the second text information by performing image recognition on characters included in the image data.

### Claim 22 (depends on 11)

The summarization apparatus according to claim 11, further comprising a third generation unit configured to generate third text information by converting electronically input data to texts, and
wherein the integration unit generates the integrated text information by integrating the first text information, the second text information, and the third text information generated by the third generation unit.

### Claim 23 (depends on 11)

The summarization apparatus according to claim 11, further comprising a voice acquisition unit configured to acquire the voice data from speech of a user.

### Claim 24 (depends on 11)

The summarization apparatus according to claim 11, further comprising an image acquisition unit configured to acquire the image data by image capturing of characters entered by a user.

### Claim 25 (depends on 11)

The summarization apparatus according to claim 11, wherein the voice data and the image data are acquired from moving image data.

### Claim 26

A summarization system including a summarization apparatus, and a terminal unit that is capable of communicating with the summarization apparatus,
the summarization apparatus comprising:
a first generation unit configured to generate first text information by converting voice data to texts;
a second generation unit configured to generate second text information by converting image data to texts;
an integration unit configured to integrate the first text information generated by the first generation unit and the second text information generated by the second generation unit to thereby generate integrated text information formed by at least one text; and
a summarization unit configured to generate summary information by summarizing the integrated text information integrated by the integration unit, and
the terminal unit comprising:
a voice acquisition unit configured to acquire voice data from speech of a user;
an image acquisition unit configured to acquire image data by image capturing of characters entered by a user; and
a transmission unit configured to transmit voice data acquired by the voice acquisition unit and image data acquired by the image acquisition unit to the summarization apparatus,
wherein the summarization apparatus includes a reception unit configured to receive voice data and image data, transmitted by the transmission unit.

### Claim 27 (depends on 26)

The summarization system according to claim 26, wherein the terminal unit receives an instruction for acquiring image data using the image acquisition unit from a user via a display screen during acquisition of voice data by the voice acquisition unit.

### Claim 28 (depends on 26)

The summarization system according to claim 26, further comprising another terminal unit different from the aforementioned terminal unit,
wherein the other terminal unit electronically inputs data, and transmits the input data to the terminal unit,
wherein the terminal unit receives the data transmitted from the other terminal unit, and
wherein the transmission unit of the terminal unit transmits the received data, the voice data, and the image data to the summarization apparatus.

### Claim 29

A method of controlling a summarization apparatus, comprising:
generating first text information by converting voice data to texts;
generating second text information by converting image data to texts;
integrating the generated first text information and the generated second text information to thereby generate integrated text information formed by at least one text; and
generating summary information by summarizing the integrated text information integrated by the integrating.
