# US 2019/0336867 A1 · Contextual in-game element recognition, annotation and interaction based on remote user input > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0336867 A1 - Document ID: US20190336867A1 - Status: Published application - Kind: A1 - Application number: 15/973,123 - Filed: 2018-05-07 - Published: 2019-11-07 - Earliest claimed: 2018-05-07 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Arunabh Verma; Raman Kumar Sarin; Alex R. Gregorio - Web page: https://patentgrep.com/patent/US20190336867A1 ## CPC - A63F 13/424: involving acoustic input signals, e.g. by using the results of pitch or rhythm extraction or voice recognition - A63F 13/86: Watching games played by other players - A63F 13/35: Details of game servers - A63F 13/537: using indicators, e.g. showing the condition of a game character on screen - A63F 13/5372: for tagging characters, objects or locations in the game scene, e.g. displaying a circle under the character controlled by the player - A63F 13/63: by the player, e.g. authoring using a level editor - A63F 13/67: adaptively or by learning from player actions, e.g. skill level adjustment or by storing successful combat sequences for re-use - A63F 13/847: Cooperative playing, e.g. requiring coordinated actions from several players to achieve a common goal - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR ## Abstract Systems, methods, and apparatuses are provided for annotating a video frame generated by a video game. A video game model that associates element tags with elements of the video game may be generated. The video game model may be applied by a video game overlay executing concurrently with the video game. The video game overlay may receive a remote user input from one or more remote devices over a network. The remote user input may be multiplexed and/or normalized, and subsequently parsed by applying the video game model to extract an element tag corresponding to the video game. By applying the video game model, an in-game element of the video game corresponding to the element tag may be identified in the video frame. Based on the identified element in the video frame of the video game, the video frame may be annotated and presented to the video game user. ## Claims ### Claim 1 A system for annotating a video frame generated by a video game, the system comprising: at least one processor circuit; at least one memory that stores program code configured to be executed by the at least one processor circuit, the program code comprising: a video game model generator that is configured to generate a video game model that associates element tags with elements of the video game; a video game overlay that is executed concurrently with the video game and includes: a receiver that is configured to receive a remote user input from one or more remote devices over a network; a parser that is configured to apply the video game model to parse the remote user input and extract therefrom an element tag corresponding to the video game; an element recognizer configured to apply the video game model to identify an element of the video game that corresponds to the element tag; and an annotation renderer configured to annotate the video frame based on the identified element and a location of the identified element. ### Claim 2 (depends on 1) The system of claim 1, wherein the video game model generator is configured to generate the video game model based on one or more of supervised training or unsupervised training. ### Claim 3 (depends on 1) The system of claim 1, wherein the receiver comprises a plurality of plugins via which the remote user input is received, wherein each plugin corresponds to a different remote service. ### Claim 4 (depends on 1) The system of claim 1, wherein the video game overlay further comprises a normalizer that is configured to normalize the remote user input prior to parsing thereof by the parser. ### Claim 5 (depends on 1) The system of claim 1, wherein the remote user input comprises a plurality of remote user input streams and wherein the video game overlay further comprises: a multiplexer that combines the remote user input streams to generate a single multiplexed input stream prior to parsing thereof by the parser. ### Claim 6 (depends on 1) The system of claim 1, wherein the remote user input comprises at least one of: voice input; text input; touch screen input; or pointing device input. ### Claim 7 (depends on 6) The system of claim 6, wherein the remote user input comprises voice input and wherein the video game overlay further comprises a speech-to-text converter that converts the voice input into text input prior to parsing thereof by the parser. ### Claim 8 (depends on 1) The system of claim 1, wherein the video game overlay further comprises: an annotation selector configured to select the annotation for rendering to the video frame based on one or more of the identified element and location. ### Claim 9 (depends on 1) The system of claim 1, wherein the annotation comprises at least one of: a highlight of the element; a visual indication of a direction to the element; a shape around the element; or an outline of the element. ### Claim 10 A method for annotating a video frame generated by a video game, the method comprising: executing a video game overlay concurrently with the video game, the executing the video game overlay including: receiving a remote user input from one or more remote devices over a network; applying a video game model that associates element tags with elements of the video game to: parse the remote user input and extract therefrom an element tag corresponding to the video game; and identify an element of the video game that corresponds to the element tag; and annotating the video frame based on the identified element and a location of the identified element. ### Claim 11 (depends on 10) The method of claim 10, further comprising: generating the video game model based on one or more of supervised training or unsupervised training. ### Claim 12 (depends on 10) The method of claim 10, wherein the receiving the remote user input comprises receiving the remote user input via a plurality of plugins, wherein each plugin corresponds to a different remote service. ### Claim 13 (depends on 10) The method of claim 10, wherein the executing the video game overlay further comprises normalizing the remote user input prior to the parsing the remote user input. ### Claim 14 (depends on 10) The method of claim 10, wherein the remote user input comprises a plurality of remote user input streams and wherein the executing the video game overlay further comprises: combining the remote user input streams to generate a single multiplexed input stream prior to the parsing the remote user input. ### Claim 15 (depends on 10) The method of claim 10, wherein the remote user input comprises at least one of: voice input; text input; touch screen input; or pointing device input. ### Claim 16 (depends on 15) The method of claim 15, wherein the remote user input comprises voice input and wherein the executing the video game overlay further comprises converting the voice input into text input prior to the parsing the remote user input. ### Claim 17 (depends on 10) The method of claim 10, wherein the executing the video game overlay further comprises: selecting the annotation for rendering to the video frame based on one or more of the identified element and location. ### Claim 18 (depends on 10) The method of claim 10, wherein the annotating the video frame comprises at least one of: highlighting the element; displaying a visual indication of a direction to the element; displaying a shape around the element; or outlining the element. ### Claim 19 A computer program product comprising a computer-readable memory having computer program logic recorded thereon that when executed by at least one processor causes the at least one processor to perform a method comprising: executing a video game overlay concurrently with the video game, the executing the video game overlay comprising: receiving a remote user input from one or more remote devices over a network; applying a video game model that associates element tags with elements of the video game to: parse the remote user input and extract therefrom an element tag corresponding to the video game; and identifying an element of the video game that corresponds to the element tag; and annotating the video frame based on the identified element and a location of the identified element. ### Claim 20 (depends on 19) The computer program product of claim 19, wherein the remote user input comprises a plurality of remote user input streams and wherein the executing the video game overlay further comprises: combining the remote user input streams to generate a single multiplexed input stream prior to the parsing the remote user input.