# US 2001/0025241 A1 · Method and system for providing automated captioning for AV signals

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2001/0025241 A1
- Document ID: US20010025241A1
- Status: Published application
- Kind: A1
- Application number: 09/800,212
- Filed: 2001-03-06
- Published: 2001-09-27
- Earliest claimed: 2000-03-06
- Assignees: Rochester Institute of Technology
- Inventors: Jeffrey K. Lange; Robert H. Paine; Jeremiah L. Parry-Hill; Steven H. Wunrow
- Web page: https://patentgrep.com/patent/US20010025241A1

## CPC

- G10L 15/005: {Language recognition}
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

System, method and computer-readable medium containing instructions for providing AV signals with open or closed captioning information. The system includes a speech-to-text processing system coupled to a signal separation processor and a signal combination processor for providing automated captioning for video broadcasts contained in AV signals. The method includes separating an audio signal from an AV signal, converting the audio signal to text data, encoding the original AV signal with the converted text data to produce a captioned AV signal and recording and displaying the captioned AV signal. The system may be mobile and portable and may be used in a classroom environment for producing recorded captioned lectures and used for broadcasting live, captioned lectures. Further, the system may automatically translate spoken words in a first language into words in a second language and include the translated words in the captioning information.

## Claims

### Claim 1

A method for providing captioning in an AV signal, the method comprising:
converting an audio signal in the AV signal to caption data automatically; and
associating the caption data with the AV signal at a time substantially corresponding to a video signal associated with the converted audio signal in the AV signal.

### Claim 2 (depends on 1)

The method as set forth in claim 1 further comprising:
capturing the AV signal, the AV signal comprising the audio signal and the video signal; and
providing the audio signal in the AV signal for the converting.

### Claim 3 (depends on 1)

The method as set forth in claim 1 wherein the converting further comprises:
determining a first amount of data in the caption data; and
providing the caption data for the associating when the first amount is greater than a threshold amount or when a first period of time has expired.

### Claim 4 (depends on 1)

The method as set forth in claim 1 wherein the associating further comprises synchronizing the caption data with one or more cues in the AV signal.

### Claim 5 (depends on 1)

The method as set forth in claim 1 wherein the associating further comprises embedding the caption data within the AV signal.

### Claim 6 (depends on 1)

The method as set forth in claim 1 further comprising displaying at least the video signal and the associated caption data.

### Claim 7 (depends on 1)

The method as set forth in claim 1 further comprising storing on at least one recordable medium at least the video signal and the associated caption data.

### Claim 8 (depends on 1)

The method as set forth in claim 1 wherein the converting further comprises translating at least one word in a first language of the audio signal into a second language, the at least one translated word included in the caption data.

### Claim 9

A speech signal processing system, the system comprising:
a speech-to-text processing system that converts an audio signal in an AV signal to caption data; and
a signal combination processing system that associates the caption data with the AV signal at a time substantially corresponding to a video signal associated with the converted audio signal in the AV signal.

### Claim 10 (depends on 9)

The system as set forth in claim 9 further comprising a source for providing the audio signal in the AV signal to the speech-to-text processing system.

### Claim 11 (depends on 9)

The system as set forth in claim 9 wherein the speech-to-text processing system further comprises a counter that determines a first amount of data in the caption data and a timer that determines when a first period of time has expired, wherein the speech-to-text processing system providing the caption data for the associating when the first amount is greater than a threshold amount or when timer indicates that the first period of time has expired.

### Claim 12 (depends on 9)

The system as set forth in claim 9 wherein the signal combination processing system synchronizes the caption data with one or more cues in the AV signal.

### Claim 13 (depends on 9)

The system as set forth in claim 9 wherein the signal combination processing system embeds the caption data within the AV signal.

### Claim 14 (depends on 9)

The system as set forth in claim 9 further comprising a video monitor for displaying at least the video signal and the associated caption data.

### Claim 15 (depends on 9)

The system as set forth in claim 9 further comprising a translator that translates at least one word in a first language of the audio signal into a second language, the at least one translated word included in the caption data, the translating device coupled to the speech recognition processor.

### Claim 16 (depends on 9)

The system as set forth in claim 9 wherein the signal separation processing system, the speech-to-text processing system and the signal combination processing system are integrated within a device, the device being portable and usable in a classroom environment.

### Claim 17

A computer readable medium having stored thereon instructions for providing captioning which when executed by at least one processor, causes the processor to perform steps comprising:
converting an audio signal in an AV signal to caption data automatically; and
associating the caption data with the AV signal at a time substantially corresponding to a video signal associated with the converted audio signal in the AV signal.

### Claim 18 (depends on 17)

The medium as set forth in claim 17 further comprising:
capturing the AV signal, the AV signal comprising the audio signal and the video signal; and
providing the audio signal in the AV signal for the converting.

### Claim 19 (depends on 17)

The medium as set forth in claim 17 wherein the converting further comprises:
determining a first amount of data in the caption data; and
providing the caption data for the associating when the first amount is greater than a threshold amount or when a first period of time has expired.

### Claim 20 (depends on 17)

The medium as set forth in claim 17 wherein the associating further comprises synchronizing the caption data with one or more cues in the AV signal.

### Claim 21 (depends on 17)

The medium as set forth in claim 17 wherein the associating further comprises embedding the caption data within the AV signal.

### Claim 22 (depends on 17)

The medium as set forth in claim 17 further comprising displaying at least the video signal and the associated caption data.

### Claim 23 (depends on 17)

The medium as set forth in claim 17 further comprising storing on at least one recordable medium at least the video signal and the associated caption data.

### Claim 24 (depends on 17)

The medium as set forth in claim 17 wherein the converting further comprises translating at least one word in a first language of the audio signal into a second language, the at least one translated word included in the caption data.
