Back to results
US 12,003,806 B2Granted
Augmented reality system for viewing an event with multiple coordinate systems and automatically generated model
AssigneeQuintar, Inc.
InventorsSankar Jayaram, Wayne O. Cochran, John Harrison, Timothy P. Heidmann, John Buddy Scott
Life of this patent
Estimating the term…
Abstract
Augmented reality systems provide graphics over views from a mobile device for both in-venue and remote viewing of a sporting or other event. A server system can provide a transformation between the coordinate system of a mobile device (smart phone, tablet computer, head mounted display) and a real world coordinate system. Requested graphics for the event are displayed over a view of an event.
Claims18
A method, comprising:
- receiving image data and corresponding image metadata from each of a plurality of mobile devices at a venue, each of the mobile devices including a camera configured to generate the image data of the venue and a display configured to display a live view of the venue for an event and to display graphics over the view of the venue, and each of the mobile devices independently maintaining a corresponding internal coordinate system and the corresponding image metadata including information specifying a location and an orientation of the corresponding mobile device within the venue for the event in the internal coordinate system of the corresponding mobile device when capturing the image data;
- retrieving, from one or more databases, point features of the venue determined in a first coordinate system;
- retrieving, from the one or more databases, locations of a set of fiducials features for the venue in a real world coordinate system, the location of each of the set of fiducial features being at corresponding point of the venue in the real world coordinate system as determined in a survey of the venue;
- determining, from the set of fiducials features for the venue in a real world coordinate system and the point features of the venue determined in the first coordinate system, the point features of the venue in the real world coordinate system;
- building of a model of the venue in the real world coordinate system from the point features of the venue and the locations of the set of fiducials features in the real world coordinate system, the model comprising a reference map including location data of a set reference features in the real world coordinate system;
- generating for each of the mobile devices a corresponding transformation between the mobile device's corresponding internal coordinate system and the real world coordinate system from the image data and corresponding image metadata from each of the mobile devices and the model of the venue in the real world coordinate system by:
- extracting a set of features from the image data from each of the mobile devices;
- comparing the set of features extracted from the image data from each of the mobile devices against the set of reference features in the real world coordinate system;
- from comparing the set of features extracted from the image data from each of the mobile devices against the set of reference features in the real world coordinate system and from the location data of the set of reference features in the real world coordinate system, determining a corresponding location and orientation in the venue for each of the mobile devices in the real world coordinate system;
- and
- generating the corresponding transformation between each of the mobile device's corresponding internal coordinate system and the real world coordinate system from the corresponding location and orientation in the real world coordinate system and the corresponding image metadata including the location and an orientation in the mobile device's corresponding internal coordinate system;
- transmitting to each of the mobile devices the corresponding coordinate transformation between the mobile device's corresponding internal coordinate system and the real world coordinate system;
- receiving from each of the mobile devices corresponding requests for content including graphics to be displayed by the mobile device over the view of the venue as specified by location and orientation in the real world coordinate system;
- transmitting to each of the mobile devices the corresponding requested content;
- receiving, at each of the mobile devices, the corresponding requested content and the corresponding coordinate transformation;
- displaying, by each of the mobile devices on the mobile device's display, the live view of the venue generated by the mobile device's camera;
- and
- displaying, by each of the mobile devices on the mobile device's display, the corresponding requested content displayed over and aligned with the live view by use of the corresponding coordinate transformation.
15 more elements
A system, comprising:
- a registration server configured to access data from one or more databases, to receive data, including image data and corresponding image metadata, from and transmit data to a plurality of mobile devices at a venue, each of the mobile devices including a camera configured to generate the image data of the venue and a display configured to display a live view of the venue for an event and to display requested graphics over and aligned with the live view of the venue by use of a corresponding coordinate transformation between a corresponding internal coordinate system of the mobile device and a real word coordinate system of the venue, and each of the mobile devices independently maintaining the corresponding internal coordinate system and the corresponding image metadata including information specifying a location and an orientation of the corresponding mobile device within the venue for the event in the internal coordinate system of the corresponding mobile device when capturing the image data, and the registration server configured to:
- retrieve point features of the venue determined in a first coordinate system;
- retrieve locations of a set of fiducials features for the venue in a real world coordinate system from the one or more databases, the location of each of the set of fiducial features being at corresponding point of the venue in the real world coordinate system as determined in a survey of the venue;
- determine, from the set of fiducials features for the venue in a real world coordinate system and the point features of the venue in the first coordinate system, the point features of the venue in the real world coordinate system;
- receive, from each of the mobile devices, the image data and the corresponding image metadata;
- from the point features of the venue and the locations of the set of fiducials features in the real world coordinate system, build a model of the venue in the real world coordinate system, the model comprising a reference map including location data of a set reference features in the real world coordinate system;
- from the image data and corresponding image metadata from each of the mobile devices and the model of the venue in the real world coordinate system, generate for each of the mobile devices the corresponding transformation between the mobile device's corresponding internal coordinate system and the real world coordinate system;
- and
- transmit to each of the mobile devices the corresponding coordinate transformation between the mobile device's corresponding internal coordinate system and the real world coordinate system;
- and
- a content server configured to receive data from and transmit data to the plurality of mobile devices, the content server configured to:
- for each of the mobile devices, receive corresponding requests for graphics to be displayed by the mobile device over a view of the venue as specified by location and orientation in the real world coordinate system;
- and
- transmit to each of the mobile devices the corresponding requested graphics.
11 more elements
Dependencies are inferred from the claim text.