# US 10,484,746 B2 · Caption replacement service system and method for interactive service in video on demand > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,484,746 B2 - Document ID: US10484746B2 - Status: Granted - Kind: B2 - Application number: 15/699,085 - Filed: 2017-09-08 - Published: 2017-12-28 - Granted: 2019-11-19 - Earliest claimed: 2013-12-27 - Assignees: Inha University Research and Business Foundation - Applicants: Geun Sik Jo - Inventors: Geun Sik Jo - Web page: https://patentgrep.com/patent/US10484746B2 ## CPC - H04N 21/4884: {for displaying subtitles} - H04N 21/4725: using interactive regions of the image, e.g. hot spots {(details of information retrieval from the Internet by using URLs G06F16/955; processing chained hypermedia data for information retrieval G06F16/94)} - H04N 21/812: {involving advertisement data (advertising per se G06Q30/02)} - H04N 21/8545: for generating interactive applications - H04N 21/8586: {by using a URL (processing chained hypermedia data for information retrieval G06F16/94; information retrieval from the Internet by using URLs G06F16/955; URL in broadcast information H04H20/93; Web-based protocols H04L67/02)} - H04N 5/2723: {Insertion of virtual advertisement; Replacing advertisements physical present in the scene by virtual advertisement (data processing systems or methods specially adapted for marketing G06Q30/02)} - H04N 21/23424: {involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement} - H04N 21/8126: {involving additional data, e.g. news, sports, stocks, weather forecasts} - H04N 21/858: Linking data to content, e.g. by linking an URL to a video object, by creating a hotspot - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Provided is a caption replacement service system and method for servicing an interactive advertisement on a video on demand (VOD). A system for making an annotation may include an annotation creator configured to create annotation information on a moving object included in a video and an annotation player configured to play a preview on the annotation information based on a frame unit or a shot unit of the video. The video and the annotation information may be used to provide an interactive advertisement by replacing a caption system of the video with a meaningful annotation. ## Claims ### Claim 1 A caption replacement service method performed by a computer system, the method comprising: playing a video present on a web, wherein the video is not annotated and comprises a series of frames, each frame being a visual image; identifying automatically a portion of a visual image of a frame of the video that contains an image of text, the image of text containing content that is original to the video; evaluating the image of text on the portion of the visual image of the frame with image text cognition to determine a content of the textual information on the portion of the visual image of the frame; determining whether the content of the textual information on the portion of the visual image of the frame contains non-current information based on a current context making a meaning of the content outdated; searching, in response to the content of the textual information being non-current, for an augmented virtual object area corresponding to the image of text on the portion of the visual image of the frame of the video from metadata associated with the video, the augmented virtual object area comprising an overlay that hides the portion of the visual image of the frame of the video that contains the image of text and being configured to receive a dynamically selected current interactive augmentation; dynamically selecting an interactive augmentation based on a profile of a user; automatically augmenting the interactive augmentation over the non-current information on top of the virtual object area while the video is playing using the searched for information on the augmented virtual object area; and providing an interactive service about an event occurring in the augmented virtual object area to the user. ### Claim 2 (depends on 1) The method of claim 1, wherein the playing of the video comprises playing a video selected by a viewer on the web. ### Claim 3 (depends on 1) The method of claim 1, wherein the searching for information on the augmented virtual object area comprises: searching for information on the augmented virtual object area corresponding to a video selected by a viewer at: a first time when the selected video is played or a predetermined time interval; and determining whether a time at which the selected video is to be played and a start time for displaying the augmented virtual object area match. ### Claim 4 (depends on 1) The method of claim 1, wherein the augmenting of the interactive augmentation comprises displaying the interactive augmentation within the video based on information on a temporal location and a spatial location of the augmented virtual object area in the information on the augmented virtual object area. ### Claim 5 (depends on 1) The method of claim 1, wherein the event occurs in response to at least one of a click and a mouse over on the interactive augmentation. ### Claim 6 (depends on 1) The method of claim 1, wherein the providing of the interactive service comprises providing additional information associated with the interactive augmentation at the interactive service, in response to the occurrence of the event. ### Claim 7 (depends on 1) The method of claim 1, wherein the interactive augmentation comprises an advertisement image extracted from a predetermined advertisement pool. ### Claim 8 (depends on 7) The method of claim 7, wherein the image extracted from the advertisement pool is extracted based on a propensity and a matter of interest of a viewer. ### Claim 9 (depends on 1) The method of claim 1, wherein the information on the augmented virtual object area comprises information on at least one of a start time and an end time for displaying the augmented virtual object area, a size of the augmented virtual object area, a display location of the augmented virtual object area, content of an interactive augmentation to be added to the augmented virtual object area, a URL address of the interactive augmentation, and an event address generated in response to a click on the augmented virtual object area.