# US 2022/0020222 A1 · Modulated display ar tracking systems and methods

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0020222 A1
- Document ID: US20220020222A1
- Status: Published application
- Kind: A1
- Application number: 17/492,392
- Filed: 2021-10-01
- Published: 2022-01-20
- Earliest claimed: 2019-11-08
- Assignees: MSG Entertainment Group, LLC
- Applicants: MSG Entertainment Group, LLC
- Inventors: Michael Romaszewicz; Andrew Plourde
- Web page: https://patentgrep.com/patent/US20220020222A1

## CPC

- G06K 7/1095: {the scanner comprising adaptations for scanning a record carrier that is displayed on a display-screen or the like}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G02B 27/01: Head-up displays
- G02B 27/0101: {characterised by optical features (G02B27/0172 takes precedence)}
- G06K 19/06037: {multi-dimensional coding}
- G06V 20/20: in augmented reality scenes
- G02B 2027/014: {comprising information/image processing systems}
- G06K: GRAPHICAL DATA READING (image or video recognition or understanding G06V); PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Modulated display AR tracking systems and methods include a display configured to transmit first image content, the image content comprising a first image content frame and a second image content frame and a device comprising an image sensor, a processor and a memory. The device identifies the first image content frame and the second image content frame by distinguishing at least one feature of the first image content frame. The device recognizes at least one tracking pattern embedded of the second image content frame and retrieves a graphical element based on the at least one tracking pattern of the second image content frame. The device displays augmented image content including the first image content frame and the graphical element.

## Claims

### Claim 1

A mobile device, comprising:
an image sensor configured to capture first image content as the first image content is being displayed by a display device, the first image content including a first image content frame being displayed at a first frame rate that is not perceivable to a human eye; and
a processor configured to:
identify a first plurality of points of a tracking pattern feature of a tracking pattern and a second plurality of points within the first image content frame that match the first plurality of points of the tracking pattern feature,
determine a first plurality of relative distances between the first plurality of points and a second plurality of relative distances between the second plurality of points,
compute a relative position or a relative orientation of the mobile device in relation to the display device based upon a comparison between the first plurality of relative distances and the second plurality of relative distances,
render a graphical element associated with the tracking pattern onto the first image content based on the relative position or the relative orientation to provide second image content; and
a display output configured to display the second image content.

### Claim 2 (depends on 1)

The mobile device of claim 1, wherein the first image content further comprises a second image content frame being displayed at a second frame rate that is perceivable to the human eye.

### Claim 3 (depends on 1)

The mobile device of claim 1, wherein the processor is further configured to identify the first image content frame including the tracking pattern feature to determine that the first image content frame includes the tracking pattern.

### Claim 4 (depends on 3)

The mobile device of claim 3, wherein the processor is configured to identify the first image content frame in response to a first predetermined number of points of the first image content frame matching a second predetermined number of points of the tracking pattern feature.

### Claim 5 (depends on 1)

The mobile device of claim 1, wherein the processor is further configured to:
determine a first configuration and arrangement of the first plurality of points and a second configuration and arrangement of the first plurality of points, and
compute the relative position or the relative orientation of the mobile device in relation to the display device based upon a comparison between the first configuration and arrangement and the second configuration and arrangement.

### Claim 6 (depends on 1)

The mobile device of claim 1, wherein the processor is configured to compute the relative position in relation to the display device based upon differences between the first plurality of relative distances and corresponding relative distances from among the second plurality of relative distances.

### Claim 7 (depends on 1)

The mobile device of claim 1, wherein the processor is configured to change a position and orientation of the graphical element based on the relative position or the relative orientation while rendering the graphical element onto the first image content.

### Claim 8

A method for rendering image content onto a display output of a mobile device, the method comprising:
capturing, by the mobile device, a first image content as the first image content is being displayed by a display device, the first image content including a first image content frame being displayed at a first frame rate that is not perceivable to a human eye;
identifying, by the mobile device, a first plurality of points of a tracking pattern feature of a tracking pattern and a second plurality of points within the first image content frame that match the first plurality of points of the tracking pattern feature;
determining, by the mobile device, a first plurality of relative distances between the first plurality of points and a second plurality of relative distances between the second plurality of points;
determining, by the mobile device, a first configuration and arrangement of the first plurality of points to a second configuration and arrangement of the second plurality of points;
computing, by the mobile device, a relative position or a relative orientation of the mobile device in relation to the display device based upon a comparison between the first plurality of relative distances and the second plurality of relative distances and a comparison of the first configuration and arrangement and the second configuration and arrangement;
rendering, by the mobile device, a graphical element associated with the tracking pattern onto the first image content based on the relative position or the relative orientation to provide second image content; and
displaying, by the display output of the mobile device, the second image content.

### Claim 9 (depends on 8)

The method of claim 8, wherein the first image content further comprises a second image content frame being displayed at a second frame rate that is perceivable to the human eye.

### Claim 10 (depends on 8)

The method of claim 8, further comprising identifying the first image content frame including the tracking pattern feature to determine that the first image content frame includes the tracking pattern.

### Claim 11 (depends on 10)

The method of claim 10, wherein the identifying the first image content frame is in response to a first predetermined number of points of the first image content frame matching a second predetermined number of points of the tracking pattern feature.

### Claim 12 (depends on 8)

The method of claim 8, wherein the computing comprises computing the relative position in relation to the display device based upon differences between the first plurality of relative distances and corresponding relative distances from among the second plurality of relative distances between the second plurality of points.

### Claim 13 (depends on 8)

The method of claim 8, wherein the computing comprises computing the relative orientation in relation to the display device based upon differences between the first configuration and arrangement and the second configuration and arrangement.

### Claim 14

A mobile device, comprising:
a memory configured to store a tracking pattern; and
a processor configured to execute instructions stored in the memory, the instructions, when executed by the processor, configuring the processor to:
identify a first plurality of points of a tracking pattern feature of the tracking pattern stored in the memory and a second plurality of points within a first image content frame that match the first plurality of points of the tracking pattern feature, the first image content frame being included within a first image content that is captured by the mobile device as the first image content is being displayed by a display device at a first frame rate that is not perceivable to a human eye;
determine a first configuration and arrangement of the first plurality of points and a second configuration and arrangement of the first plurality of points,
compute a relative position or a relative orientation of the mobile device in relation to the display device based upon a comparison between the first configuration and arrangement and the second configuration and arrangement,
render a graphical element associated with the tracking pattern onto the first image content based on the relative position or the relative orientation to provide second image content, and
provide the second image content to a display output for display.

### Claim 15 (depends on 14)

The mobile device of claim 14, wherein the first image content further comprises a second image content frame being displayed at a second frame rate that is perceivable to the human eye.

### Claim 16 (depends on 14)

The mobile device of claim 14, wherein the instructions, when executed by the processor, further configure the processor to identify the first image content frame including the tracking pattern feature to determine that the first image content frame includes the tracking pattern.

### Claim 17 (depends on 16)

The mobile device of claim 16, wherein the instructions, when executed by the processor, configure the processor to identify the first image content frame in response to a first predetermined number of points of the first image content frame matching a second predetermined number of points of the tracking pattern feature.

### Claim 18 (depends on 14)

The mobile device of claim 14, wherein the instructions, when executed by the processor, further configure the processor to:
determine a first plurality of relative distances between the first plurality of points to a second plurality of relative distances between the second plurality of points; and
compute the relative position or the relative orientation of the mobile device in relation to the display device based upon a comparison between the first plurality of relative distances and the second plurality of relative distances.

### Claim 19 (depends on 14)

The mobile device of claim 14, wherein the instructions, when executed by the processor, configure the processor to compute the relative orientation in relation to the display device based upon differences between the first configuration and arrangement and the second configuration and arrangement.

### Claim 20 (depends on 14)

The mobile device of claim 14, wherein the instructions, when executed by the processor, configure the processor to change a position and orientation of the graphical element based on the relative position or the relative orientation while rendering the graphical element onto the first image content.
