# US 2020/0346114 A1 · Contextual in-game element recognition and dynamic advertisement overlay

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0346114 A1
- Document ID: US20200346114A1
- Status: Published application
- Kind: A1
- Application number: 16/399,664
- Filed: 2019-04-30
- Published: 2020-11-05
- Earliest claimed: 2019-04-30
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Arunabh P. Verma; Eric Hamilton; Raman K. Sarin
- Web page: https://patentgrep.com/patent/US20200346114A1

## CPC

- A63F 13/537: using indicators, e.g. showing the condition of a game character on screen
- A63F 13/61: using advertising information
- A63F 13/53: involving additional visual information provided to the game scene, e.g. by overlay to simulate a head-up display [HUD] or displaying a laser sight in a shooting game
- A63F 13/79: involving player-related data, e.g. identities, accounts, preferences or play histories
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR

## Abstract

Systems, methods, and apparatuses are provided for overlaying content on a video frame generated by a video game. A content overlay engine may be executed concurrently with the execution of a video game. An element recognizer may obtain the video frame and identify an element of the video game in the frame, such as an in-game element. A renderability determiner may determine whether an overlay may be rendered on the element. Based at least on a determination that the overlay is renderable, a content renderer may be configured to overlay the content on the element. The overlaid content may be provided in various ways, such as presenting an overlaid video frame to a local computing device (e.g., a gaming console or a computer), and/or transmitting the overlaid video frame to a remotely located computing device.

## Claims

### Claim 1

A system for overlaying content on a video frame generated by a video game, the system comprising:
at least one processor circuit;
at least one memory that stores program code, the program code including instructions to cause the at least one processor circuit to perform the actions of:
executing a content overlay engine concurrently with the video game, the executing including:
obtaining the video frame;
identifying an element of the video game in the video frame;
determining whether an overlay is renderable on the element; and
overlaying the content on the element in the video frame based at least on a determination that the overlay is renderable.

### Claim 2 (depends on 1)

The system of claim 1, wherein at least one of the identifying the element in the video game or the determining whether the overlay is renderable is based on an application of a machine learning-based model.

### Claim 3 (depends on 1)

The system of claim 1, wherein the program code further comprises instructions to cause the at least one processor circuit to perform the action of:
obtaining an advertisement based at least on one or more of the video game, a type of the element, a user location, or a user preference; and
wherein the content comprises the obtained advertisement.

### Claim 4 (depends on 1)

The system of claim 1, wherein the determining whether the overlay is renderable on the element is based at least on one or more prior executions of the video game by a particular user.

### Claim 5 (depends on 1)

The system of claim 1, wherein the determining whether the overlay is renderable on the element is based at least on one or more prior executions of the video game by a plurality of users.

### Claim 6 (depends on 1)

The system of claim 1, wherein the determining whether the overlay is renderable on the element is based on at least one of:
a length of time the element was visible during at least one previous execution of the video game;
a size of the element in the video frame; or
a rate of movement of the element compared to a previous video frame.

### Claim 7 (depends on 1)

The system of claim 1, wherein the overlaying the content on the element comprises blending the overlaid content into the video frame.

### Claim 8 (depends on 3)

The system of claim 3, wherein the program code further comprises instructions to cause the at least one processor circuit to perform the action of:
providing an incentive to a user account associated with the video game based on the advertisement.

### Claim 9

A method for overlaying content on a video frame generated by a video game, the method comprising:
executing a content overlay engine concurrently with the video game, the executing the content overlay engine including:
obtaining the video frame;
identifying an element of the video game in the video frame;
determining whether an overlay is renderable on the element; and
overlaying the content on the element in the video frame based at least on a determination that the overlay is renderable.

### Claim 10 (depends on 9)

The method of claim 9, wherein at least one of the identifying the element in the video game or the determining whether the overlay is renderable is based on an application of a machine learning-based model.

### Claim 11 (depends on 9)

The method of claim 9, further comprising:
obtaining an advertisement based at least on one or more of the video game, a type of the element, a user location, or a user preference; and
wherein the content comprises the obtained advertisement.

### Claim 12 (depends on 9)

The method of claim 9, wherein the determining whether the overlay is renderable is based at least on one or more prior executions of the video game by a particular user.

### Claim 13 (depends on 9)

The method of claim 9, wherein the determining whether the overlay is renderable is based at least on one or more prior executions of the video game by a plurality of users.

### Claim 14 (depends on 9)

The method of claim 9, wherein the determining whether the overlay is renderable is based on at least one of:
a length of time the element was visible during at least one previous execution of the video game;
a size of the element in the video frame; or
a rate of movement of the element compared to a previous video frame.

### Claim 15

A system for overlaying content on a video frame generated by a video game, the system comprising:
at least one processor circuit;
at least one memory that stores program code, the program code including instructions to cause the at least one processor circuit to perform the actions of:
executing a content overlay engine concurrently with the video game, the executing including:
obtaining the video frame;
identifying an element of the video game in the video frame;
generating a first output frame that comprises a first item of content overlaid on the element in the video frame;
generating a second output frame that comprises a second item of content overlaid on the element in the video frame; and
providing the first output frame to a first remote device and the second output frame to a second remote device.

### Claim 16 (depends on 15)

The system of claim 15, wherein the program code further comprises instructions to cause the at least one processor circuit to perform the action of:
determining whether an overlay is renderable on the element; and
wherein the generating the first output frame comprising the first item of content and the generating the second output frame comprising the second item of content is based at least on a determination that the overlay is renderable.

### Claim 17 (depends on 16)

The system of claim 16, wherein at least one of the identifying the element in the video game or the determining whether the overlay is renderable is based on an application of a machine learning-based model.

### Claim 18 (depends on 16)

The system of claim 16, wherein the determining whether the overlay is renderable on the element is based on at least one of:
a length of time the element was visible during at least one previous execution of the video game;
a size of the element in the video frame; or
a rate of movement of the element compared to a previous video frame.

### Claim 19 (depends on 15)

The system of claim 15, wherein the program code further comprises instructions to cause the at least one processor circuit to perform the actions of:
obtaining a first advertisement and a second advertisement based at least on one or more of the video game, a type of the element, a location of the first remote device, a location of the second remote device, or a user preference; and
wherein the first item of content comprises the first advertisement and the second item of content comprises the second advertisement.

### Claim 20 (depends on 15)

The system of claim 15, wherein the generating the first output frame comprising the first item of content and the generating the second output frame comprising the second item of content comprises blending the first item of content in the first output frame and blending the second item of content in the second output frame, respectively.
