# US 12,182,947 B2 · Systems and methods for displaying and adjusting virtual objects based on interactive and dynamic content > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,182,947 B2 - Document ID: US12182947B2 - Status: Granted - Kind: B2 - Application number: 17/674,561 - Filed: 2022-02-17 - Published: 2023-08-17 - Granted: 2024-12-31 - Earliest claimed: 2022-02-17 - Assignees: Adeia Guides Inc. - Applicants: Adeia Guides Inc. - Inventors: Serhad Doken - Web page: https://patentgrep.com/patent/US12182947B2 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - H04N 21/2187: Live feed - H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - H04N 21/25841: {involving the geographical location of the client (retrieval from the Internet by querying based on geographical locations G06F16/9537; arrangements for identifying locations of receiving stations in broadcast systems H04H60/51; location of the user terminal in data switching networks H04L67/52; services making use of the location of users or terminals in wireless networks H04W4/02; locating users or terminals in wireless networks H04W64/00)} - H04N 21/262: Content or additional data distribution scheduling, e.g. sending additional data at off-peak times, updating software modules, calculating the carousel transmission frequency, delaying a video stream transmission, generating play-lists {(scheduling strategies for dispatcher in multiprogramming arrangements G06F9/4881; arrangements for scheduling broadcast services or broadcast-related services H04H60/06; flow control in packet networks H04L47/10; establishing a time schedule or organising the servicing of application requests H04L67/62)} - H04N 21/4312: {involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations} - H04N 21/4316: {for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window} - H04N 21/4722: for requesting additional data associated with the content - H04N 21/4788: communicating with other users, e.g. chatting {(arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104)} - H04L 67/306: User profiles - H04L 67/53: using third party service providers - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract Systems and methods for overlaying virtual objects in a virtual environment based on user interest and user interactions are disclosed. The methods analyze a live view of a surface viewed through camera or a transparent lens and determine if the policies of the surface allow overlaying of virtual objects. The method fetches a virtual object and calculates a score based on user interest, user gaze, and user engagement. Virtual objects that meet the policies of the surface and a scoring criterion are overlayed on the surface in the virtual environment and enhanced based on a plurality of enhancement factors. Virtual objects are also overlayed in frames of a live broadcast based on their scores. Virtual objects displayed on a conference call interface, which may be meeting tools or icons associated with other conferencing functionality, are enhanced, or removed from the user interface based on their utilization. ## Claims ### Claim 1 A method of overlaying a virtual object on a frame of a live broadcast stream comprising: displaying, on a display of a media device, the frame of the live broadcast stream; identifying one or more objects displayed in the frame; determining that real-time comments associated with the one or more objects are available; and in response to determining that real-time comments are available: calculating a score for the one or more objects, wherein the one or more objects are part of the live broadcast stream and displayed in the frame of the live broadcast stream; accessing a space and dimensions policy associated with the live broadcast stream; determining to overlay real-time comments associated with an object, from the one or more objects, based on the calculated score; and overlaying, a number of real-time comments, from the real-time comments, over a scene of the frame of the live broadcast stream at a set of corresponding locations, wherein the number of real-time comments to be overlayed and their corresponding locations are based on the accessed space and dimensions policy. ### Claim 2 (depends on 1) The method of claim 1, wherein the score is a number of real-time comments available for the object. ### Claim 3 (depends on 2) The method of claim 2, further comprising: ranking the one or more objects based on the calculated scores; and selecting the object, from the one or more objects, based on the ranking. ### Claim 4 (depends on 3) The method of claim 3, further comprising selecting a highest ranked object, from the one or more objects, for overlaying on the frame of the live broadcast stream. ### Claim 5 (depends on 3) The method of claim 3, wherein, the real-time comments are posted by a plurality of viewers of the live broadcast stream. ### Claim 6 (depends on 1) The method of claim 1, wherein, the real-time comments are overlayed within a predetermined distance from the one or more objects in the frame. ### Claim 7 (depends on 1) The method of claim 1, further comprising: obtaining a plurality of comments from one or more databases; aggregating comments that are contextually related into a plurality of groups; and overlaying a comment that is representative of a group, from the plurality of groups. ### Claim 8 (depends on 7) The method of claim 7, further comprising using an artificial intelligence engine to determine if the comments are contextually related. ### Claim 9 (depends on 7) The method of claim 7, further comprising: ranking each group based on a number of comments in each group, wherein the group with a highest number of comments is ranked the highest; and selecting a comment that is representative of the group with the highest ranking to overlay on the frame of the live broadcast stream. ### Claim 10 (depends on 1) The method of claim 1, wherein the live broadcast stream is associated with a single media asset and the one or more objects are objects from the single media asset. ### Claim 11 (depends on 1) The method of claim 1, wherein the space and dimensions policy associated with the live broadcast stream is determined by a provider of content being broadcasted. ### Claim 12 (depends on 1) The method of claim 1, wherein the space and dimensions policy comprises a restricted zone in the frame of the live broadcast stream where the overlay of the real-time comments is not permitted. ### Claim 13 (depends on 1) The method of claim 1, further comprising: determining an amount of space available in the frame for overlaying the real-time comments based on the space and dimensions policy; and selecting the number of real-time comments to be overlayed based on the determined about of space available. ### Claim 14 A method of displaying a virtual object on a frame of a media asset comprising: displaying, on a display of a media device, a frame of the media asset, wherein the frame depicts a plurality of objects; querying a database to determine that comments made by viewers of the media asset for the objects displayed in the frame are available; and in response to determining that comments are available: ranking each object, from the plurality of objects, based on the comments associated with the object; accessing a space and dimensions policy associated with the media asset; transmitting one or more comments to the media device; and overlaying the transmitted one or more comments within a predetermined distance from the associated object, wherein the number and location of the one or more comments in the frame is additionally determined based on the space and dimensions policy. ### Claim 15 (depends on 14) The method of claim 14, wherein the database is queried based on text in a caption file of the media asset that relates to the displayed frame. ### Claim 16 A system for overlaying a virtual object on a frame of a live broadcast stream comprising: communication circuitry configured to access the live broadcast stream; control circuitry configured to: display the frame of the live broadcast stream on a display of a media device; identify one or more objects displayed in the frame; determine that real-time comments associated with the one or more objects are available; and in response to determining that real-time comments are available: calculate a score for the one or more objects, wherein the one or more objects are part of the live broadcast stream and displayed in the frame of the live broadcast stream; accessing a space and dimensions policy associated with the live broadcast stream; determine to overlay real-time comments associated with an object, from the one or more objects, based on the calculated score; and overlay, a number of the real-time comments, from the real-time comments, over a scene of the frame of the live broadcast stream at a set of corresponding locations, wherein the number of the real-time comments to be overlayed and their corresponding locations are based on the accessed space and dimensions policy. ### Claim 17 (depends on 16) The system of claim 16, wherein the score is a number of real-time comments available for the object. ### Claim 18 (depends on 17) The system of claim 17, further comprising, the control circuitry configured to: rank the one or more objects based on the calculated scores; select the object, from the one or more objects, based on the ranking. ### Claim 19 (depends on 18) The system of claim 18, further comprising, the control circuitry configured to select a highest ranked object, from the one or more objects, for overlaying on the frame of the live broadcast stream. ### Claim 20 (depends on 16) The system of claim 16, further comprising, the control circuitry configured to: obtain a plurality of comments from one or more databases; aggregate comments that are contextually related into a plurality of groups; and overlay a comment that is representative of a group, from the plurality of groups. ### Claim 21 (depends on 20) The system of claim 20, further comprising, the control circuitry configured to: rank each group based on a number of comments in each group, wherein the group with a highest number of comments is ranked the highest; and select a comment that is representative of the group with the highest ranking to overlay on the frame of the live broadcast stream.