# US 8,782,722 B1 · Decoding of closed captions at a media server > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 8,782,722 B1 - Document ID: US8782722B1 - Status: Granted - Kind: B1 - Application number: 13/857,572 - Filed: 2013-04-05 - Granted: 2014-07-15 - Earliest claimed: 2013-04-05 - Assignees: Wowza Media Systems, LLC - Applicants: Wowza Media Systems, LLC - Inventors: Scott Kellicker - Web page: https://patentgrep.com/patent/US8782722B1 ## CPC - H04N 7/0882: {for the transmission of character code signals, e.g. for teletext (circuits for the digital non-picture data signal H04N7/035)} - H04N 21/2187: Live feed - H04N 21/234336: {by media transcoding, e.g. video is transformed into a slideshow of still pictures or audio is converted into text} - H04N 21/2355: {involving reformatting operations of additional data, e.g. HTML pages (optimising the visualization of content for information retrieval from the Internet G06F16/9577; tracking of instant messages H04L51/234; media packet handling at the source H04L65/762)} - H04N 21/4312: {involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations} - H04N 21/434: Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream {(demultiplexing of data packets for data networks, e.g. RTP/UDP H04L65/00)} - H04N 21/4382: {Demodulation or channel decoding, e.g. QPSK demodulation (analog front ends or means for connecting modulators, demodulators or transceivers to a transmission line H04L27/0002)} - H04N 21/44008: {involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - H04N 21/47202: {for requesting content on demand, e.g. video on demand} - H04N 21/4856: {for language selection, e.g. for the menu or subtitles} - H04N 21/4884: {for displaying subtitles} - H04N 21/4888: {for displaying teletext characters} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Systems and methods of processing closed captions are disclosed. For example, a media server may receive a first video stream and first closed caption data associated with the first video stream. The media server may interpret at least one command included in the first closed caption data to generate interpreted closed caption data. The media server may transmit, to a destination device, a second video stream including second closed caption data that is generated based on the interpreted closed caption data. ## Claims ### Claim 1 An apparatus comprising: a processor; a network interface to receive, at a media server, a first video stream and first closed caption data associated with the first video stream; a memory storing state information indicating an incrementally updated processing time and a time of a previously received caption; and a closed caption decoder executable by the processor to: interpret at least one command included in the first closed caption data to generate interpreted closed caption data; detect an end of caption in response to determining that a difference between the processing time and the time of the previously received caption exceeds a threshold amount of time; and generate second closed caption data based on the interpreted closed caption data in response to detecting the end of caption, wherein the network interface is further to send, from the media server to a destination device, a second video stream including the second closed caption data. ### Claim 2 (depends on 1) The apparatus of claim 1, wherein the second closed caption data is sent in accordance with action message format (AMF) on TextData, web video text tracks (WebVTT), smooth streaming, or any combination thereof. ### Claim 3 (depends on 1) The apparatus of claim 1, wherein the closed caption decoder is further executable by the processor to detect an end of a first caption in response to: determining that a particular command in data corresponding to the first caption indicates a change from a first closed captioning mode of a plurality of closed captioning modes to a second closed captioning mode of the plurality of closed captioning modes, determining that the particular command includes a carriage return, is an end of caption command, or a combination thereof, determining that the threshold amount of time has elapsed since a previously received second closed caption, or any combination thereof. ### Claim 4 (depends on 3) The apparatus of claim 3, wherein the plurality of closed captioning modes includes a pop-on mode, a roll-up mode, a direct text mode, a paint-on mode, an extended data service (XDS) mode, or any combination thereof. ### Claim 5 (depends on 1) The apparatus of claim 1, wherein the first closed caption data is represented in a first format and further comprising a converter executable by the processor to: convert the interpreted closed caption data to timed text data represented in a platform-independent format; and generate the second closed caption data by converting the timed text data from the platform-independent format to a second format. ### Claim 6 (depends on 1) The apparatus of claim 1, wherein the closed caption decoder comprises: a first character memory; a second character memory, wherein one of the first character memory and the second character memory is designated as displayable and the other of the first character memory and the second character memory is designated as non-displayable; and a non-caption module executable by the processor to process non-caption data received via the first video stream. ### Claim 7 (depends on 1) The apparatus of claim 1, wherein the state information further indicates a caption mode and a caption start time. ### Claim 8 A computer-readable storage device storing instructions that, when executed by a processor, cause the processor to: receive, at a media server, a first video stream and first closed caption data associated with the first video stream; interpret at least one command included in the first closed caption data to generate interpreted closed caption data; detect an end of caption in response to determining that a threshold amount of time has elapsed since a previously received caption; in response to detecting the end of caption, generate second closed caption data based on the interpreted closed caption data; and transmit, from the media server to a destination device, a second video stream including the second closed caption data. ### Claim 9 A method comprising: receiving, at a media server, a first video stream and first closed caption data associated with the first video stream; interpreting at least one command included in the first closed caption data to generate interpreted closed caption data; detecting an end of caption in response to determining that a threshold amount of time has elapsed since a previously received caption; in response to detecting the end of caption, generating second closed caption data based on the interpreted closed caption data; and transmitting, from the media server to a destination device, a second video stream including the second closed caption data. ### Claim 10 (depends on 9) The method of claim 9, wherein interpreting the first closed caption data comprises simulating a hardware closed caption decoder using software executed by the media server. ### Claim 11 (depends on 9) The method of claim 9, wherein the first closed caption data is represented in a first format and further comprising: converting the interpreted closed caption data to timed text data represented in a platform-independent format; and generating the second closed caption data by converting the time text data from the platform-independent format to a second format. ### Claim 12 (depends on 11) The method of claim 11, wherein the first format comprises consumer electronics association (CEA)-608 format, CEA-708, digital video broadcasting (DVB), or any combination thereof. ### Claim 13 (depends on 11) The method of claim 11, further comprising: loading a first converter that is compatible with the first format to convert the first closed caption data from the first format to the platform independent format; receiving third closed caption data associated with the first video stream, wherein the third closed caption data is represented in a third format; and loading a second converter that is compatible with the third format to convert the third closed caption data from the third format to the platform-independent format. ### Claim 14 (depends on 9) The method of claim 9, further comprising performing an optical character recognition (OCR) process on video data to generate the first closed caption data. ### Claim 15 (depends on 9) The method of claim 9, wherein the threshold amount of time is approximately 250 milliseconds. ### Claim 16 (depends on 9) The method of claim 9, wherein the threshold amount of time is configurable by a user. ### Claim 17 (depends on 9) The method of claim 9, wherein the at least one command includes a command to draw a character, move a cursor, erase a character, clear a portion of a display screen, change a text color, change a font, change a background color, or any combination thereof. ### Claim 18 (depends on 9) The method of claim 9, further comprising maintaining a first character memory and a second character memory, wherein the first character memory is designated as displayable and the second character memory is designated as non-displayable. ### Claim 19 (depends on 18) The method of claim 18, further comprising: when a particular command included in particular closed caption data is an off-screen command: in response to determining that the particular command is not a swap command, modifying the second memory based on the particular command; and in response to determining that the particular command is the swap command, designating the first character memory as non-displayable, designating the second character memory as displayable, and generating the interpreted closed caption data from the second character memory; and when the particular command is a direct draw command: in response to determining that the particular command designates an end of a particular caption, generating particular interpreted closed caption data from the first character memory and clearing the first character memory; and in response to determining that the particular command does not designate the end of the particular caption, modifying the first character memory based on the particular command. ### Claim 20 (depends on 9) The method of claim 9, wherein the first closed caption data includes non-caption data comprising a description of a video program, a parental rating associated with the video program, or a combination thereof.