# US 2014/0157303 A1 · Annotating an object in a video with virtual information on a mobile terminal > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2014/0157303 A1 - Document ID: US20140157303A1 - Status: Published application - Kind: A1 - Application number: 13/910,666 - Filed: 2013-06-05 - Published: 2014-06-05 - Earliest claimed: 2012-01-20 - Assignees: Geun Sik Jo; Inha-Industry Partnership Institute - Applicants: Inha Industry Partnership Institute - Inventors: Geun Sik Jo - Web page: https://patentgrep.com/patent/US20140157303A1 ## CPC - H04N 21/84: Generation or processing of descriptive data, e.g. content descriptors {(systems specially adapted for using meta-information in broadcast systems H04H60/73)} - H04N 21/4126: {The peripheral being portable, e.g. PDAs or mobile phones} - H04N 21/435: Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream - H04N 21/47205: {for manipulating displayed content, e.g. interacting with MPEG-4 objects, editing locally} - H04N 21/478: Supplemental services, e.g. displaying phone caller identification, shopping application - H04N 21/812: {involving advertisement data (advertising per se G06Q30/02)} - G06F 40/169: Annotation, e.g. comment data or footnotes - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Embodiments of the present invention provide a visual interlock system for providing product information by displaying video content annotated with virtual information (stored as metadata) such as advertisement information or a social network service (SNS) message from a friend. Specifically, among other things, embodiments of the present invention provide a system that allows a video playing on a television to be simultaneously played on a mobile device. The system searches for virtual information related to a visual object in the video. The virtual information is stored as metadata. The virtual object in the video is annotated with the virtual information. The annotated video is displayed on the user terminal. ## Claims ### Claim 1 A video interlock system for displaying annotated video, comprising: an interlock module configured to request a Network Screen (Nscreen) service based on a user request, wherein the NScreen service allows for simultaneously displaying a video being played on a first device to be played on a second device; a search module configured to search for virtual information associated with a visual object in the video, wherein the virtual information is stored as metadata in a metadata maintenance server; an annotation module configured to annotate the visual object in the video with the virtual information; and a display module configured to display the annotated video annotated on the second device. ### Claim 2 (depends on 1) The video interlock system of claim 1, further comprising an annotation module configured to generate virtual information associated with the visual object and store the virtual information as metadata in the metadata maintenance server. ### Claim 3 (depends on 1) The video interlock system of claim 1, wherein the first device is one of a television or mobile device and the second device is a mobile device. ### Claim 4 (depends on 1) The video interlock system of claim 1, wherein the virtual information includes at least one of a product category, product price, product description, product shipping information, multimedia data, social network service (SNS) message, or uniform resource locator (URL) address related to visual object. ### Claim 5 (depends on 1) The video interlock system of claim 1, wherein the annotation module further comprises an annotation tool for inputting information related to a product and retrieving information related to the product in a Linking Open Data (LOD) environment. ### Claim 6 (depends on 1) The video interlock system of claim 1, wherein the metadata maintenance sever includes the annotation module and storage for performing at least one of storing, deleting, updating, reusing, or sharing virtual information, wherein the metadata maintenance server is separate from a contents server. ### Claim 7 (depends on 1) The video interlock system of claim 1, wherein the display module is further configured to display the video annotated with the metadata based on at least one of targeting information or a display method for providing personalized virtual information, wherein the targeting information and display method are defined by a virtual information provider. ### Claim 8 (depends on 1) The video interlock system of claim 1, wherein the search module is further configured to search for metadata related to the visual object based on at least one of targeting information and user profile information, wherein the user profile information is generated based on a user behavior derived from one or more Internet searches. ### Claim 9 (depends on 1) The video interlock system of claim 1, wherein the display module is further configured to display a composed video view, wherein the composed video view comprises a metadata view and a user view. ### Claim 10 (depends on 1) The video interlock system of claim 1, wherein the display module is further configured to display the metadata based on movement route information. ### Claim 11 (depends on 8) The video interlock system of claim 8, wherein the movement route information is stored in the metadata maintenance server. ### Claim 12 A method for displaying annotated video annotated, comprising: requesting a Network Screen (Nscreen) service based on a user request, wherein the NScreen service allows for simultaneously displaying a video being played a first device to be placed on a second device; searching for virtual information related to a visual object in the video, wherein the virtual information is stored as metadata in a metadata maintenance server; annotating the visual object in the video with the virtual information; and displaying the video annotated with the metadata on the second device. ### Claim 13 (depends on 12) The method of claim 12, further comprising generating virtual information associated with the visual object and storing the virtual information as metadata in the metadata maintenance server. ### Claim 14 (depends on 12) The method of claim 12, wherein the first device is one of a television or mobile device and the second device is a mobile device. ### Claim 15 (depends on 12) The method of claim 12, wherein the virtual information includes at least one of a product category, product price, product description, product shipping information, multimedia data, social network service (SNS) message, or uniform resource locator (URL) address related to visual object. ### Claim 16 (depends on 12) The method of claim 12, further comprising inputting information related to a product and retrieving information related to the product using a Linking Open Data (LOD) environment. ### Claim 17 (depends on 12) The method of claim 12, further comprising performing at least one of storing, deleting, updating, reusing, or sharing virtual information on a metadata maintenance server, wherein the metadata maintenance server is separate from a contents server. ### Claim 18 (depends on 11) The method of claim 11, further comprising displaying the video annotated with the metadata based on at least one of targeting information or a display method for providing personalized virtual information, wherein the targeting information and display method are defined by a virtual information provider. ### Claim 19 (depends on 11) The method of claim 11, further comprising searching for metadata related to the visual object based on at least one of targeting information and user profile information, wherein the user profile information is generated based on a user behavior derived from one or more Internet searches. ### Claim 20 (depends on 11) The method of claim 11, further comprising displaying a composed video view, wherein the composed video view comprises a metadata view and a user view. ### Claim 21 (depends on 11) The method of claim 11, further comprising displaying the metadata based on movement route information, wherein the movement route information is stored in the metadata maintenance server. ### Claim 22 A computer program product for displaying annotated video annotated, the computer program product comprising a computer readable storage media, and program instructions stored on the computer readable storage media, to: request a Network Screen (Nscreen) service based on a user request, wherein the NScreen service allows for simultaneously displaying the video being played a first device to be placed on a second device; generate virtual information associated with a visual objects in the video, wherein the virtual information is stored as metadata; search for metadata related to a visual object in the video, wherein the metadata is stored in a metadata maintenance server; annotate the visual object in the video with the virtual information; and display the annotated video on the second device.