# US 2022/0414947 A1 · Controlling interactivity of digital content overlaid onto displayed data via graphics processing circuitry using a frame buffer

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0414947 A1
- Document ID: US20220414947A1
- Status: Published application
- Kind: A1
- Application number: 17/902,787
- Filed: 2022-09-02
- Published: 2022-12-29
- Earliest claimed: 2021-04-30
- Assignees: Mobeus Industries, Inc.
- Applicants: Mobeus Industries, Inc.
- Inventors: Dharmendra ETWARU; Michael R. SUTCLIFF; Aram ANDRIASYAN
- Web page: https://patentgrep.com/patent/US20220414947A1

## CPC

- G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06T 11/00: Two-dimensional [2D] image generation
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G06F 3/14: Digital output to display device {; Cooperation and interconnection of the display device with other functional units}
- G06F 9/451: Execution arrangements for user interfaces
- G06T 1/60: Memory management
- G06T 7/74: {involving reference images or patches}
- G09G 5/14: Display of multiple viewports
- G09G 5/377: for mixing or overlaying two or more graphic patterns (G09G5/02, G09G5/397 take precedence)
- G09G 5/38: with means for controlling the display position
- G09G 5/40: characterised by the way in which both a pattern determined by character code and another pattern are displayed simultaneously, or either pattern is displayed selectively, e.g. with character code memory and APA, i.e. all-points-addressable, memory
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G09G 5/001: {Arbitration of resources in a display system, e.g. control of access to frame buffer by video controller and/or main processor}
- G09G 2340/0464: Positioning
- G09G 2340/10: Mixing of images, i.e. displayed pixel being the result of an operation, e.g. adding, on the corresponding input pixels
- G09G 2340/125: wherein one of the images is motion video
- G09G 2354/00: Aspects of interface with display user
- G09G 2360/08: Power processing, i.e. workload management for processors involved in display operations, such as CPUs or GPUs
- G09G 2360/18: Use of a frame buffer in a display terminal, inclusive of the display panel
- G09G 2370/022: Centralised management of display operation, e.g. in a server instead of locally
- G09G 2370/04: Exchange of auxiliary data, i.e. other than image data, between monitor and graphics controller
- G09G 2370/06: Consumer Electronics Control, i.e. control of another device by a display or vice versa
- G09G 2370/20: Details of the management of multiple sources of image data
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)

## Abstract

An apparatus, method, and computer readable medium that access a frame buffer of a graphics processing unit (GPU), analyze, in the frame buffer, a frame representing displayed data, based on the analyzed frame, identify a reference patch that includes an instruction to retrieve content, generate an overlay including an augmentation layer which includes the content, superimpose the overlay onto the displayed data such that the content is viewable while a portion of the base layer is obscured, detect a user input, determine a location of the user input in the augmentation layer, associate the location in the augmentation layer with a target location in the base layer, and associate, within memory, the target location with an operation such that the user input in the augmentation layer activates an input in the base layer.

## Claims

### Claim 1

An apparatus, comprising:
processing circuitry configured to
generate an overlay comprising an augmentation layer, the augmentation layer including content,
superimpose the overlay onto displayed data to create a composite comprising the augmentation layer and a base layer, such that the content is viewable while a portion of the base layer is obscured from view,
determine a location of a user input in the augmentation layer,
associate the location of the user input in the augmentation layer with a target location in the base layer, and
associate the target location with an operation corresponding to the augmentation layer and the base layer such that the user input in the augmentation layer activates an input in the base layer.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein the processing circuitry is configured to identify, in a frame, an instruction to retrieve the content.

### Claim 3 (depends on 2)

The apparatus of claim 2, wherein the instruction is included in a reference patch.

### Claim 4 (depends on 1)

The apparatus of claim 1, wherein determining the location of the user input in the augmentation layer is performed by
detecting, in a frame, an input marker in the displayed data; and
determining a location of the input marker in the augmentation layer.

### Claim 5 (depends on 4)

The apparatus of claim 4, wherein the input marker comprises an image of a user, who is associated with the user input, captured by a camera and displayed in the augmentation layer.

### Claim 6 (depends on 1)

The apparatus of claim 1, wherein the user input is obtained via a gesture or at least one of a mouse, keyboard, and microphone, wherein the gesture is performed by a user and captured by a camera.

### Claim 7 (depends on 1)

The apparatus of claim 1, wherein the location of the user input in the augmentation layer includes an augmentation layer position on a display device which is different from a base layer position on the display device of the target location in the base layer.

### Claim 8

A method, comprising:
generating an overlay comprising an augmentation layer, the augmentation layer including content;
superimposing the overlay onto displayed data to create a composite comprising the augmentation layer and a base layer, such that the content is viewable while a portion of the base layer is obscured from view;
determining a location of a user input in the augmentation layer;
associating the location of the user input in the augmentation layer with a target location in the base layer; and
associating the target location with an operation corresponding to the augmentation layer and the base layer such that the user input in the augmentation layer activates an input in the base layer.

### Claim 9 (depends on 8)

The method of claim 8, further comprising identifying, in a frame, an instruction to retrieve the content.

### Claim 10 (depends on 9)

The method of claim 9, wherein the instruction is included in a reference patch.

### Claim 11 (depends on 8)

The method of claim 8, wherein determining the location of the user input in the augmentation layer is performed by
detecting, in a frame, an input marker in the displayed data; and
determining a location of the input marker in the augmentation layer.

### Claim 12 (depends on 11)

The method of claim 11, wherein the input marker comprises an image of a user, who is associated with the user input, captured by a camera and displayed in the augmentation layer.

### Claim 13 (depends on 8)

The method of claim 8, wherein the user input is obtained via a gesture or at least one of a mouse, keyboard, and microphone, wherein the gesture is performed by a user and captured by a camera.

### Claim 14 (depends on 8)

The method of claim 8, wherein the location of the user input in the augmentation layer includes an augmentation layer position on a display device which is different from a base layer position on the display device of the target location in the base layer.

### Claim 15

A non-transitory computer-readable storage medium storing computer-readable instructions that, when executed by a computer, cause the computer to perform a method, the method comprising:
generating an overlay comprising an augmentation layer, the augmentation layer including content;
superimposing the overlay onto displayed data to create a composite comprising the augmentation layer and a base layer, such that the content is viewable while a portion of the base layer is obscured from view;
determining a location of a user input in the augmentation layer;
associating the location of the user input in the augmentation layer with a target location in the base layer; and
associating the target location with an operation corresponding to the augmentation layer and the base layer such that the user input in the augmentation layer activates an input in the base layer.

### Claim 16 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, further comprising identifying, in a frame, an instruction to retrieve the content.

### Claim 17 (depends on 16)

The non-transitory computer-readable storage medium of claim 16, wherein the instruction is included in a reference patch.

### Claim 18 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein determining the location of the user input in the augmentation layer is performed by
detecting, in a frame, an input marker in the displayed data; and
determining a location of the input marker in the augmentation layer.

### Claim 19 (depends on 18)

The non-transitory computer-readable storage medium of claim 18, wherein the input marker comprises an image of a user, who is associated with the user input, captured by a camera and displayed in the augmentation layer.

### Claim 20 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein the user input is obtained via a gesture or at least one of a mouse, keyboard, and microphone, wherein the gesture is performed by a user and captured by a camera.
