# US 2013/0159545 A1 · System to augment a visual data stream with user-specific content > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2013/0159545 A1 - Document ID: US20130159545A1 - Status: Published application - Kind: A1 - Application number: 13/764,503 - Filed: 2013-02-11 - Published: 2013-06-20 - Earliest claimed: 2011-02-08 - Assignees: Longsand Limited - Inventors: Sean Mark Blanchflower; Michael Richard Lynch - Web page: https://patentgrep.com/patent/US20130159545A1 ## CPC - H04L 65/60: Network streaming of media packets - H04N 21/4524: {involving the geographical location of the client (retrieval from the Internet by querying based on geographical locations G06F16/9537; systems specially adapted for using geographical information in broadcast systems H04H60/70; protocols in which the network application is adapted for the location of the user terminal in communication control or processing H04L67/52; services making use of the location of users or terminals in wireless networks H04W4/02; locating users or terminals in wireless networks H04W64/00)} - H04N 21/4532: {involving end-user characteristics, e.g. viewer profile, preferences (monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18)} - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract User profile information for many users may be stored in a user profile database associated with a server computer connected to a network. Based on information received from a mobile computing device, the server computer selects augment information from an augment information database coupled with the user profile database. The information received from the mobile computing device is related to frames of a video stream being captured by the mobile computing device and geographical information associated with these frames. The augment information is also selected based on the user profile associated with the mobile computing device. The augment information is to be transmitted from the server computer to the mobile computing device using wireless communication and is used by the mobile computing device to overlay the frames of the video stream being captured by the mobile computing device. ## Claims ### Claim at position 1 1-20. (canceled) ### Claim 21 A method for adding relevant augment information to a video stream, the method comprising: accessing a first frame and a second frame of a video stream, wherein the first frame comprises objects and the second frame comprises objects; relating patterns in the first frame and the second frame to determine points of interest contained within each of the first frame and the second frame; receiving characteristics information of the points of interest included in the first frame and the second frame, wherein the characteristics information of the points of interest include geometric shapes of the objects contained in the first frame and geometric shapes of the objects contained in the second frame; selecting relevant augment information from an augment information database based upon the received characteristics information; and adding, by a processor, the selected relevant augment information to the accessed video stream. ### Claim 22 (depends on 21) The method according to claim 21, further comprising: decomposing the objects contained in the first frame to form the geometric shapes of the objects contained in the first frame; and decomposing the objects contained in the second frame to form the geometric shapes of the objects contained in the second frame. ### Claim 23 (depends on 21) The method according to claim 21, wherein receiving characteristics information of the points of interest included in the first frame and the second frame further comprises receiving characteristics information that include other distinctive visual features of the first frame and the second frame. ### Claim 24 (depends on 21) The method according to claim 21, wherein selecting relevant augment information further comprises matching the characteristics information with characteristics of objects that are stored in an object database to determine the points of interest captured in the video stream. ### Claim 25 (depends on 21) The method according to claim 21, wherein the video stream comprises a video stream being captured by a mobile computing device, wherein the mobile computing device has identity information that corresponds to a user profile associated with the mobile computing device. ### Claim 26 (depends on 25) The method according to claim 25, further comprising: receiving geographic information associated with the mobile computing device, wherein the geographic information pertains to a location at which the video stream is being captured; combining the characteristics information and the geographic information; and matching the combined characteristics information and the geographic information with characteristics of objects that are stored in an object database to identify the points of interest captured in the video stream. ### Claim 27 (depends on 25) The method according to claim 25, wherein selecting the relevant augment information further comprises selecting the relevant augment information based on the user profile associated with the mobile computing device. ### Claim 28 (depends on 21) The method according to claim 21, wherein adding the selected relevant augment information to the accessed video stream further comprises adding the selected relevant augment information to enable at least one of overlaying and highlighting of the points of interest in the video stream with the relevant augment information. ### Claim 29 An apparatus for adding relevant augment information to a video stream, the apparatus comprising: a memory storing machine readable instructions, that when executed, cause the apparatus to: access a first frame and a second frame of a video stream, wherein the first frame comprises objects and the second frame comprises objects; decompose the objects contained in the first frame and the second frame into geometric shapes; relate patterns in the first frame and the second frame to determine points of interest contained within each of the first frame and the second frame; receive characteristics information of the determined points of interest, wherein the characteristics information of the points of interest include the geometric shapes of the objects contained in the first frame and the second frame; select relevant augment information from an augment information database based upon the received characteristics information; and add the selected relevant augment information to the accessed video stream; and a processor to execute the machine readable instructions. ### Claim 30 (depends on 29) The apparatus according to claim 29, wherein the machine readable instructions, that when executed, are further to cause the apparatus to: receive characteristics information that include other distinctive visual features of the first frame and the second frame. ### Claim 31 (depends on 29) The apparatus according to claim 29, wherein the machine readable instructions, that when executed, are further to cause the apparatus to: match the characteristics information with characteristics of objects that are stored in an object database to determine the points of interest captured in the video stream. ### Claim 32 (depends on 29) The apparatus according to claim 29, wherein the video stream comprises a video stream being captured by a mobile computing device, wherein the mobile computing device has identity information that corresponds to a user profile associated with the mobile computing device. ### Claim 33 (depends on 32) The apparatus according to claim 32, wherein the machine readable instructions, that when executed, are further to cause the apparatus to: receive geographic information associated with the mobile computing device, wherein the geographic information pertains to a location at which the video stream is being captured; combine the characteristics information and the geographic information; and match the combined characteristics information and the geographic information with characteristics of objects that are stored in an object database to determine the points of interest captured in the video stream. ### Claim 34 (depends on 32) The apparatus according to claim 32, wherein, to select the relevant augment information, the machine readable instructions are further to select the relevant augment information based on the user profile associated with the mobile computing device. ### Claim 35 (depends on 29) The apparatus according to claim 29, wherein, to add the selected relevant augment information to the accessed video stream, the machine readable instructions are further to add the selected relevant augment information to enable at least one of overlaying and highlighting of the points of interest in the video stream with the relevant augment information. ### Claim 36 A non-transitory machine readable storage medium on which is stored machine readable instructions that when executed by a processor are to cause the processor to: access a first frame and a second frame of a video stream, wherein the first frame comprises objects and the second frame comprises objects; decompose the objects contained in the first frame and the second frame into geometric shapes; relate patterns in the first frame and the second frame to determine points of interest contained within each of the first frame and the second frame; receive characteristics information of the determined points of interest, wherein the characteristics information of the points of interest include the geometric shapes of the objects contained in the first frame and the second frame; select relevant augment information from an augment information database based upon the received characteristics information; and add the selected relevant augment information to the accessed video stream. ### Claim 37 (depends on 36) The non-transitory computer readable storage medium according to claim 36, wherein the video stream comprises a video stream being captured by a mobile computing device, wherein the mobile computing device has identity information that corresponds to a user profile associated with the mobile computing device. ### Claim 38 (depends on 37) The non-transitory computer readable storage medium according to claim 37, wherein the machine readable instructions are further to cause the processor to: receive geographic information associated with the mobile computing device, wherein the geographic information pertains to a location at which the video stream is being captured; combine the characteristics information and the geographic information; and match the combined characteristics information and the geographic information with characteristics of objects that are stored in an object database to determine the points of interest captured in the video stream. ### Claim 39 (depends on 37) The non-transitory computer readable storage medium according to claim 37, wherein, to select the relevant augment information, the machine readable instructions are to cause the processor to: select the relevant augment information based on the user profile associated with the mobile computing device.