# US 2015/0100981 A1 · Video Processing Method, Terminal, and Caption Server > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2015/0100981 A1 - Document ID: US20150100981A1 - Status: Published application - Kind: A1 - Application number: 14/568,409 - Filed: 2014-12-12 - Published: 2015-04-09 - Earliest claimed: 2012-06-29 - Assignees: Huawei Device Co., Ltd. - Applicants: Huawei Device Co., Ltd. - Inventors: Wenmei Gao; Shunan Fan; Xiaoqiang Lv; Yahui Wang - Web page: https://patentgrep.com/patent/US20150100981A1 ## CPC - H04N 21/8133: {specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video programme} - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - H04N 21/233: Processing of audio elementary streams {(monitoring, identification or recognition of audio in broadcast systems H04H60/58)} - H04N 21/234336: {by media transcoding, e.g. video is transformed into a slideshow of still pictures or audio is converted into text} - H04N 21/237: Communication with additional data server - H04N 21/43074: {of additional data with content streams on the same device, e.g. of EPG data or interactive icon with a TV programme} - H04N 21/435: Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream - H04N 21/437: Interfacing the upstream path of the transmission network, e.g. for transmitting client requests to a VOD server {(flow control in data networks H04L47/10; streaming protocols, e.g. RTP or RTCP, H04L65/65; scheduling or organising the servicing of application requests in data packet switching networks H04L67/60)} - H04N 21/4394: {involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams (arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58)} - H04N 21/440236: {by media transcoding, e.g. video is transformed into a slideshow of still pictures, audio is converted into text} - H04N 21/4722: for requesting additional data associated with the content - H04N 21/4856: {for language selection, e.g. for the menu or subtitles} - H04N 21/4884: {for displaying subtitles} - H04N 21/8547: involving timestamps for synchronizing content - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract A video processing method, a terminal, and a caption server are provided. The method includes receiving a video program stream corresponding to a video program; requesting, from a caption server, obtaining of a caption corresponding to the video program stream; receiving the caption corresponding to the video program stream and returned by the caption server; and displaying the video program and the caption. In the present invention, a caption is obtained in real time according to a video program. ## Claims ### Claim 1 A video processing method, comprising: receiving a video program stream corresponding to a video program; requesting, from a caption server, to obtain a caption corresponding to the video program stream; receiving the caption returned by the caption server; and displaying the video program and the caption. ### Claim 2 (depends on 1) The video processing method according to claim 1, wherein requesting, from the caption server, to obtain the caption corresponding to the video program stream comprises: sending a caption obtaining request to the caption server, wherein the caption obtaining request is used to request obtaining the caption corresponding to the video program stream; and sending an audio stream in the video program stream to the caption server such that the caption server performs speech-to-text conversion on the audio stream according to the caption obtaining request to generate the caption. ### Claim 3 (depends on 1) The video processing method according to claim 1, wherein the received video program stream further comprises a program identifier of the video program, and wherein requesting, from the caption server, to obtain the caption corresponding to the video program stream comprises sending a caption obtaining request to the caption server, wherein the caption obtaining request is used to request obtaining the caption corresponding to the video program stream, and the caption obtaining request carries the program identifier such that the caption server determines the caption according to the caption obtaining request and the program identifier. ### Claim 4 (depends on 3) The video processing method according to claim 3, wherein after sending the caption obtaining request to the caption server and before receiving the caption returned by the caption server, the method further comprises: receiving a connection failure response sent by the caption server, wherein the connection failure response is used to indicate that the caption server fails to connect to a program source according to the program identifier to obtain the audio stream, and the program source is used to generate the video program stream; and sending the audio stream in the video program stream to the caption server according to the connection failure response such that the caption server performs speech-to-text conversion on the audio stream to generate the caption. ### Claim 5 (depends on 1) The video processing method according to claim 1, wherein after receiving the video program stream corresponding to the video program and before displaying the video program and the caption, the method further comprises buffering the received video program stream corresponding to the video program at least until the caption corresponding to the video program stream and returned by the caption server is received. ### Claim 6 (depends on 1) The video processing method according to claim 1, wherein the received caption corresponding to the video program stream and returned by the caption server further comprises a packet identifier of an audio packet corresponding to the caption, and wherein after receiving the caption corresponding to the video program stream and returned by the caption server and before displaying the video program and the caption, the method further comprises synchronizing the caption with the audio stream according to the packet identifier of the audio packet to display the video program and the caption synchronously. ### Claim 7 A video processing method, comprising: receiving a request sent by a terminal and used to obtain a caption corresponding to a video program stream that corresponds to a video program; obtaining, according to the request, the caption corresponding to the video program stream; returning the caption to the terminal such that the terminal displays the video program and the caption. ### Claim 8 (depends on 7) The video processing method according to claim 7, wherein receiving the request sent by the terminal and used to obtain the caption corresponding to the video program stream that corresponds to the video program comprises: receiving a caption obtaining request sent by the terminal, wherein the caption obtaining request is used to request obtaining of the caption corresponding to the video program stream; and receiving an audio stream in the video program stream sent by the terminal, and wherein obtaining the caption corresponding to the video program stream comprises performing speech-to-text conversion on the audio stream according to the caption obtaining request to generate the caption. ### Claim 9 (depends on 7) The video processing method according to claim 7, wherein receiving the request sent by the terminal and used to obtain the caption corresponding to the video program stream that corresponds to the video program comprises receiving a caption obtaining request sent by the terminal, wherein the caption obtaining request is used to request obtaining the caption corresponding to the video program stream, and the caption obtaining request carries a program identifier of the video program, and wherein obtaining the caption corresponding to the video program stream comprises obtaining the caption corresponding to the video program stream according to the caption obtaining request and the program identifier. ### Claim 10 (depends on 9) The video processing method according to claim 9, wherein obtaining the caption corresponding to the video program stream according to the caption obtaining request and the program identifier comprises: determining, according to the caption obtaining request, whether a program source where the video program corresponding to the program identifier is located has been connected; returning the caption to the terminal when the program source has been connected; and establishing a connection with the program source, obtaining an audio stream in the video program stream, and performing speech-to-text conversion on the audio stream to generate the caption when the program source has not been connected. ### Claim 11 (depends on 10) The video processing method according to claim 10, wherein establishment of the connection with the program source fails, and wherein the method further comprises: returning, to the terminal, a connection failure response used to indicate that the connection with the program source fails; receiving an audio stream of the terminal that is sent by the terminal according to the connection failure response; and performing speech-to-text conversion on the audio stream to generate the caption. ### Claim 12 (depends on 8) The video processing method according to claim 8, wherein performing the speech-to-text conversion on the audio stream to generate the caption comprises: performing speech-to-text conversion on the audio stream to generate the caption corresponding to the video program stream; and setting, in the caption, a packet identifier of an audio packet corresponding to the caption such that the terminal synchronizes the caption with the audio stream according to the packet identifier of the audio packet. ### Claim 13 A terminal, comprising: a program receiving unit configured to receive a video program stream corresponding to a video program; a real-time caption client configured to request, from a caption server, obtaining a caption corresponding to the video program stream, and receive the caption returned by the caption server; and a program presenting unit configured to display the video program and the caption. ### Claim 14 (depends on 13) The terminal according to claim 13, wherein the real-time caption client is configured to: send a caption obtaining request to the caption server, wherein the caption obtaining request is used to request obtaining the caption corresponding to the video program stream; and send an audio stream in the video program stream to the caption server such that the caption server performs speech-to-text conversion on the audio stream according to the caption obtaining request to generate the caption. ### Claim 15 (depends on 13) The terminal according to claim 13, wherein the real-time caption client is configured to send a caption obtaining request to the caption server, wherein the caption obtaining request is used to obtain the caption corresponding to the video program stream, and the caption obtaining request carries a program identifier of the video program such that the caption server determines the caption corresponding to the video program stream according to the caption obtaining request and the program identifier. ### Claim 16 (depends on 15) The terminal according to claim 15, wherein the real-time caption client is further configured to: receive a connection failure response sent by the caption server after sending the caption obtaining request to the caption server, wherein the connection failure response is used to indicate that the caption server fails to connect to a program source according to the program identifier to obtain the audio stream, and wherein the program source is used to generate the video program stream; and send the audio stream in the video program stream to the caption server according to the connection failure response such that the caption server performs speech-to-text conversion on the audio stream to generate the caption. ### Claim 17 (depends on 13) The terminal according to claim 13, wherein the real-time caption client is further configured to buffer the video program stream after the video program stream corresponding to the video program is received at least until the caption corresponding to the video program stream and returned by the caption server is received. ### Claim 18 (depends on 13) The terminal according to claims 13, wherein the real-time caption client is further configured to synchronize the caption with the audio stream according to a packet identifier of an audio packet comprised in the received caption corresponding to the video program stream and returned by the caption server such that the program presenting unit displays the video program and the caption synchronously. ### Claim 19 A caption server, comprising: a request receiving unit configured to receive a request sent by a terminal and used to obtain a caption corresponding to a video program stream that corresponds to a video program; a caption obtaining unit configured to obtain, according to the request, the caption corresponding to the video program stream; and a caption sending unit configured to return the caption to the terminal such that the terminal displays the video program and the caption. ### Claim 20 (depends on 19) The caption server according to claim 19, wherein the request receiving unit is configured to receive a caption obtaining request sent by the terminal, wherein the caption obtaining request is used to request obtaining of the caption corresponding to the video program stream; and receive an audio stream in the video program stream sent by the terminal, and wherein the caption obtaining unit is configured to perform speech-to-text conversion on the audio stream according to the caption obtaining request to generate the caption. ### Claim 21 (depends on 19) The caption server according to claim 19, wherein the request receiving unit is configured to receive a caption obtaining request sent by the terminal, wherein the caption obtaining request is used to request obtaining the caption corresponding to the video program stream, and the caption obtaining request carries a program identifier of the video program, and wherein the caption obtaining unit is configured to obtain the caption corresponding to the video program stream according to the caption obtaining request and the program identifier. ### Claim 22 (depends on 21) The caption server according to claim 21, wherein the caption obtaining unit comprises: a determining subunit configured to determine, according to the caption obtaining request, whether a program source where the video program corresponding to the program identifier is located has been connected, wherein the caption sending unit is configured to return the caption to the terminal when the determination result of the determining subunit is that the program source has been connected; an obtaining subunit configured to establish a connection with the program source and obtain an audio stream in the video program stream when the determination result of the determining subunit is that the program source has not been connected; and a converting subunit configured to perform speech-to-text conversion on the audio stream to generate the caption. ### Claim 23 (depends on 22) The caption server according to claim 22, wherein the caption obtaining unit further comprises a feedback subunit configured to return, to the terminal, a connection failure response used to indicate that the connection with the program source fails when the obtaining subunit fails to establish a connection with the program source, wherein the request receiving unit is further configured to receive the audio stream of the terminal which is sent by the terminal according to the connection failure response such that the converting subunit performs speech-to-text conversion on the audio stream to generate the caption. ### Claim 24 (depends on 22) The caption server according to claim 22, wherein the converting subunit is further configured to set, in the caption, a packet identifier of an audio packet corresponding to the caption when performing speech-to-text conversion on the audio stream to generate the caption such that the terminal synchronizes the caption with the audio stream according to the packet identifier of the audio packet.