# US 12,003,806 B2 · Augmented reality system for viewing an event with multiple coordinate systems and automatically generated model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,003,806 B2
- Document ID: US12003806B2
- Status: Granted
- Kind: B2
- Application number: 17/242,265
- Filed: 2021-04-27
- Published: 2022-09-15
- Granted: 2024-06-04
- Earliest claimed: 2021-03-11
- Assignees: Quintar, Inc.
- Applicants: Quintar, Inc.
- Inventors: Sankar Jayaram; Wayne O. Cochran; John Harrison; Timothy P. Heidmann; John Buddy Scott
- Web page: https://patentgrep.com/patent/US12003806B2

## CPC

- H04N 21/4316: {for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window}
- G06T 7/33: using feature-based methods
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 20/20: in augmented reality scenes
- G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44)
- H04N 13/117: the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
- H04N 13/178: Metadata, e.g. disparity information
- H04N 13/183: On-screen display [OSD] information, e.g. subtitles or menus
- H04N 13/194: Transmission of image signals
- H04N 21/2187: Live feed
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- G06T 2207/30221: Sports video; Sports image
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Augmented reality systems provide graphics over views from a mobile device for both in-venue and remote viewing of a sporting or other event. A server system can provide a transformation between the coordinate system of a mobile device (smart phone, tablet computer, head mounted display) and a real world coordinate system. Requested graphics for the event are displayed over a view of an event.

## Claims

### Claim 1

A method, comprising:
receiving image data and corresponding image metadata from each of a plurality of mobile devices at a venue, each of the mobile devices including a camera configured to generate the image data of the venue and a display configured to display a live view of the venue for an event and to display graphics over the view of the venue, and each of the mobile devices independently maintaining a corresponding internal coordinate system and the corresponding image metadata including information specifying a location and an orientation of the corresponding mobile device within the venue for the event in the internal coordinate system of the corresponding mobile device when capturing the image data;
retrieving, from one or more databases, point features of the venue determined in a first coordinate system;
retrieving, from the one or more databases, locations of a set of fiducials features for the venue in a real world coordinate system, the location of each of the set of fiducial features being at corresponding point of the venue in the real world coordinate system as determined in a survey of the venue;
determining, from the set of fiducials features for the venue in a real world coordinate system and the point features of the venue determined in the first coordinate system, the point features of the venue in the real world coordinate system;
building of a model of the venue in the real world coordinate system from the point features of the venue and the locations of the set of fiducials features in the real world coordinate system, the model comprising a reference map including location data of a set reference features in the real world coordinate system;
generating for each of the mobile devices a corresponding transformation between the mobile device's corresponding internal coordinate system and the real world coordinate system from the image data and corresponding image metadata from each of the mobile devices and the model of the venue in the real world coordinate system by:
extracting a set of features from the image data from each of the mobile devices;
comparing the set of features extracted from the image data from each of the mobile devices against the set of reference features in the real world coordinate system;
from comparing the set of features extracted from the image data from each of the mobile devices against the set of reference features in the real world coordinate system and from the location data of the set of reference features in the real world coordinate system, determining a corresponding location and orientation in the venue for each of the mobile devices in the real world coordinate system; and
generating the corresponding transformation between each of the mobile device's corresponding internal coordinate system and the real world coordinate system from the corresponding location and orientation in the real world coordinate system and the corresponding image metadata including the location and an orientation in the mobile device's corresponding internal coordinate system;
transmitting to each of the mobile devices the corresponding coordinate transformation between the mobile device's corresponding internal coordinate system and the real world coordinate system;
receiving from each of the mobile devices corresponding requests for content including graphics to be displayed by the mobile device over the view of the venue as specified by location and orientation in the real world coordinate system;
transmitting to each of the mobile devices the corresponding requested content;
receiving, at each of the mobile devices, the corresponding requested content and the corresponding coordinate transformation;
displaying, by each of the mobile devices on the mobile device's display, the live view of the venue generated by the mobile device's camera; and
displaying, by each of the mobile devices on the mobile device's display, the corresponding requested content displayed over and aligned with the live view by use of the corresponding coordinate transformation.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
receiving survey images of the venue and a set of fiducial points for the venue in the real world coordinate system, one or more of the fiducial points being included in the survey images;
processing the survey images and the set of fiducial point to generate the point features of the venue and the locations of the set of fiducials features in the real world coordinate system; and
storing the point features of the venue and the locations of the set of fiducials features in the real world coordinate system in the one or more databases.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
retrieving the requested content from the one or more databases.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
receiving live content of the event from the venue; and
generating graphics to be displayed by the mobile devices over the view of the venue from the live content.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
receiving live video of the event from the venue; and
transmitting to each of the mobile devices the live video in response to a request for content including the live video.

### Claim 6 (depends on 1)

The method of claim 1, wherein the graphics to be displayed by the mobile device over a view of the venue include three dimensional contours of the venue to be displayed over a two dimensional view of the venue.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
receiving a request from a first mobile device to supply specified content with specified graphics to a second mobile device; and
transmitting the specified content with specified graphics to the second mobile device.

### Claim 8 (depends on 7)

The method of claim 7, wherein the specified content with specified graphics includes gamification content.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
receiving from a first mobile device a gamification request;
transmitting the gamification to a gamification website;
receiving a response to the gamification request from the gamification website; and
transmitting the response to the first mobile device.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
receiving live content of the event from the venue; and
transmitting graphics derived from the live content to the mobile devices.

### Claim 11 (depends on 1)

The method of claim 1, further comprising:
receiving from a first mobile device a request to supply an alternate view of the venue; and
transmitting the alternate view to the first mobile device.

### Claim 12

A system, comprising:
a registration server configured to access data from one or more databases, to receive data, including image data and corresponding image metadata, from and transmit data to a plurality of mobile devices at a venue, each of the mobile devices including a camera configured to generate the image data of the venue and a display configured to display a live view of the venue for an event and to display requested graphics over and aligned with the live view of the venue by use of a corresponding coordinate transformation between a corresponding internal coordinate system of the mobile device and a real word coordinate system of the venue, and each of the mobile devices independently maintaining the corresponding internal coordinate system and the corresponding image metadata including information specifying a location and an orientation of the corresponding mobile device within the venue for the event in the internal coordinate system of the corresponding mobile device when capturing the image data, and the registration server configured to:
retrieve point features of the venue determined in a first coordinate system;
retrieve locations of a set of fiducials features for the venue in a real world coordinate system from the one or more databases, the location of each of the set of fiducial features being at corresponding point of the venue in the real world coordinate system as determined in a survey of the venue;
determine, from the set of fiducials features for the venue in a real world coordinate system and the point features of the venue in the first coordinate system, the point features of the venue in the real world coordinate system;
receive, from each of the mobile devices, the image data and the corresponding image metadata;
from the point features of the venue and the locations of the set of fiducials features in the real world coordinate system, build a model of the venue in the real world coordinate system, the model comprising a reference map including location data of a set reference features in the real world coordinate system;
from the image data and corresponding image metadata from each of the mobile devices and the model of the venue in the real world coordinate system, generate for each of the mobile devices the corresponding transformation between the mobile device's corresponding internal coordinate system and the real world coordinate system; and
transmit to each of the mobile devices the corresponding coordinate transformation between the mobile device's corresponding internal coordinate system and the real world coordinate system; and
a content server configured to receive data from and transmit data to the plurality of mobile devices, the content server configured to:
for each of the mobile devices, receive corresponding requests for graphics to be displayed by the mobile device over a view of the venue as specified by location and orientation in the real world coordinate system; and
transmit to each of the mobile devices the corresponding requested graphics.

### Claim 13 (depends on 12)

The system of claim 12, wherein the requested graphics to be displayed by the mobile device over a view of the venue include three dimensional contours of the venue to be displayed over a two dimensional view of the venue.

### Claim 14 (depends on 12)

The system of claim 12, wherein the content server is further configured to:
receive, from a first mobile device, a request to supply specified content with specified graphics to a second mobile device; and
transmit the specified content with specified graphics to the second mobile device.

### Claim 15 (depends on 14)

The system of claim 14, wherein the specified content with specified graphics includes gamification content.

### Claim 16 (depends on 12)

The system of claim 12, wherein the content server is further configured to:
receive, from a first mobile device, a gamification request;
transmit the gamification request to a gamification website;
receive a response to the gamification request from the gamification website; and
transmit the response to the first mobile device.

### Claim 17 (depends on 12)

The system of claim 12, wherein the content server is further configured to:
receive live content of the event from the venue; and
transmit graphics derived from the live content to the mobile devices.

### Claim 18 (depends on 12)

The system of claim 12, wherein the content server is further configured to:
receive, from a first mobile device, a request to supply an alternate view of the venue; and
transmit the alternate view to the first mobile device.