US 7,796,155 B1Granted

Method and apparatus for real-time group interactive augmented-reality area monitoring, suitable for enhancing the enjoyment of entertainment events

AssigneeHRL Laboratories, LLC

InventorsHoward Neely, III, Ronald T. Azuma, Jerry Isdale, Mike Daily

Life of this patent

Estimating the term…

Loading patent…

Abstract

A real-time, group interactive, augmented-reality system for monitoring an area, suitable for enhancing the enjoyment of entertainment events, is presented. The system comprises a sensor data acquisition system for receiving sensor data, in real-time, from a plurality of data acquisition sensors disposed about an area to be monitored, and a plurality of display centers communicatively coupled with the data acquisition system for receiving sensor data therefrom. A display center includes an user input, a data acquisition sensor selector, an interaction interface for communicating in real-time with users of other display centers, an interface for connecting a display device, and a local display device augmenting system, which are configured to allow a user to interactively augment in real-time his/her local display of events with user-inputted doodles, user requested information and statistics, and user selected viewing angles. The system allows users to interactively share the locally augmented-reality display with users of other display centers.

Claims14

Dependent claims

A real-time, group interactive, augmented-reality area monitoring system, suitable for enhancing the enjoyment of entertainment events, the system comprising:

a sensor data acquisition system for receiving sensor data, in real-time, from a plurality of data acquisition sensors disposed about an area to be monitored;
a plurality of display centers communicatively coupled with the sensor data acquisition system for receiving sensor data therefrom, with each display center further comprising:
an input for receiving input from a user;
a sensor selector for allowing a user to select at least one data acquisition sensor from which to display sensor data;
an interaction interface for communicating, in real-time, with other display centers to allow a user of a display center to interact with users of other display centers;
a local data augmenting system for locally augmenting the display of the sensor data with user input;
and
an interface for connecting the display center with a display device;
and
an object recognition system for receiving sensor data from the data acquisition system and for recognizing a segment of the sensor data;
wherein the object recognition system comprises a model-extractor for extracting spatially-registered geometric models of objects present in the sensor data;
a central data augmenting system coupled with the sensor data acquisition system for centrally augmenting sensor data from the data acquisition sensors before the sensor data is received by the display centers;
wherein the object recognition system augments the sensor data with information regarding the recognized segment for provision to the central data augmenting system;
a recognition-information database connected with the object recognition system and with the central data augmenting system so that a user may select an object to be recognized so that the system automatically recognizes the object selected and provides information regarding the object from the recognition information database;
a data-marker for marking the sensor data received from the data acquisition sensors, wherein the data acquisition sensors include at least one of a video camera capturing video frames, an audio recording sensor, and a video/audio camera, wherein the video/audio camera records video signals and audio signals concurrently, wherein the video camera is oriented at a line-of-sight (LOS) orientation viewing angle during a video frame, and where the data-marker marks the sensor data received from a video camera with a time-tag corresponding to a video frame and with the line-of-sight (LOS) orientation angle of the video camera at the video frame;
and
an interaction-selector for allowing a user of a display center to selectively choose which other display centers to interact with, and wherein a user of a selected to interact with display center may choose to cancel the interaction with a particular display center at any time.

14 more elements

A real-time, group interactive, augmented-reality area monitoring system, suitable for enhancing the enjoyment of entertainment events, the system comprising:

a sensor data acquisition system for receiving sensor data, in real-time, from a plurality of data acquisition sensors disposed about an area to be monitored;
a display center communicatively coupled with the data acquisition system for receiving sensor data therefrom, with the display center further comprising:
an input for receiving input from a user;
a sensor selector for allowing a user to select at least one data acquisition sensor from which to display sensor data;
a connecting interface for communicating, in real-time, with a communication network, wherein the communication network allows the user of the display center to interact with other users connected to the communication network;
a local augmenting system for locally augmenting the display of the sensor data with user input;
and
an interface for connecting the display center with a display device;
and
an object recognition system for receiving sensor data from the data acquisition system and for recognizing a segment of the sensor data;
wherein the object recognition system comprises a model-extractor for extracting spatially-registered geometric models of objects present in the sensor data;
a central data augmenting system coupled with the sensor data acquisition system for centrally augmenting sensor data from the data acquisition sensors before the sensor data is received by the display center;
wherein the object recognition system augments the sensor data with information regarding the recognized segment for provision to the central data augmenting system;
a recognition-information database connected with the object recognition system and with the central data augmenting system so that a user may select an object to be recognized so that the system automatically recognizes the object selected and provides information regarding the object from the recognition information database;
and
a data-marker for marking the sensor data received from the data acquisition sensors, wherein the data acquisition sensors include at least one of a video camera capturing video frames, an audio recording sensor, and a video/audio camera, wherein the video/audio camera records video signals and audio signals concurrently, wherein the video camera is oriented at a line-of-sight (LOS) orientation viewing angle during a video frame, and where the data-marker marks the sensor data received from a video camera with a time-tag corresponding to a video frame and with the line-of-sight (LOS) orientation angle of the video camera at the video frame.

13 more elements

A real-time, group interactive, augmented-reality area monitoring system, suitable for enhancing the enjoyment of entertainment events, the system comprising:

a sensor data acquisition system for receiving sensor data, in real-time, from a plurality of data acquisition sensors disposed about an area to be monitored;
a display center communicatively coupled with the data acquisition system for receiving sensor data therefrom, with the display center further comprising:
an input for receiving input from a user;
a sensor selector for allowing a user to select at least one data acquisition sensor from which to display sensor data;
a connecting interface for communicating, in real-time, with a communication network, wherein the communication network allows the user of the display center to interact with other users connected to the communication network;
a local augmenting system for locally augmenting the display of the sensor data with user input;
and
an interface for connecting the display center with a display device;
and
an object recognition system for receiving sensor data from the data acquisition system and for recognizing a segment of the sensor data;
wherein the object recognition system comprises a model-extractor for extracting spatially-registered geometric models of objects present in the sensor data;
a central data augmenting system coupled with the sensor data acquisition system for centrally augmenting sensor data from the data acquisition sensors before the sensor data is received by the display centers;
wherein the object recognition system comprises:
a tracker for tracking the sensor data marked with time-tag and (LOS) orientation angle, wherein the tracker detects object features that represent a recognized object, finds feature locations of the object features in the data acquisition sensors, creates three-dimensional feature location data by fusing the feature locations of the object features from the plurality of data acquisition sensors, and tracks the three dimensional feature location data in the data acquisition sensors over time;
wherein the model-extractor extracts the spatially-registered geometric models from the three-dimensional feature location data, and further extracts texture models from the current video frame based on location of the spatially-registered geometric models in the area being monitored;
a parameter-extractor for extracting time-tagged parameters from the spatially-registered geometric models, wherein the parameter-extractor outputs time-tagged parameters that characterize the objects recognized;
a model-matcher for associating a spatially-registered geometric model with an actor performing an activity;
a time-marker for time-tagging the spatially-registered geometric model of the actor;
an activity-storage for storing a time sequence of activities performed by an actor by using time-tagged spatially-registered geometric models of the actor;
and
a spatially-registered geometric model database for storing a time sequence of activities performed by an actor and a spatially-registered geometric model generated by the object recognition system, so that a user may select a spatially-registered geometric model for display on the display center, or so that a user may select to recreate prior activities by displaying on the display center a time sequence of activities previously performed by an actor.

18 more elements

A real-time, group interactive, augmented-reality area monitoring system, suitable for enhancing the enjoyment of entertainment events, the system comprising:

a sensor data acquisition system for receiving sensor data, in real-time, from a plurality of data acquisition sensors disposed about an area to be monitored;
a display center communicatively coupled with the data acquisition system for receiving sensor data therefrom, with the display center further comprising:
an input for receiving input from a user;
a sensor selector for allowing a user to select at least one data acquisition sensor from which to display sensor data;
a connecting interface for communicating, in real-time, with a communication network, wherein the communication network allows the user of the display center to interact with other users connected to the communication network;
a local augmenting system for locally augmenting the display of the sensor data with user input;
and
an interface for connecting the display center with a display device;
and
an object recognition system for receiving sensor data from the data acquisition system and for recognizing a segment of the sensor data;
wherein the object recognition system comprises a model-extractor for extracting spatially-registered geometric models of objects present in the sensor data;
wherein the object recognition system augments the sensor data with information regarding the recognized segment for provision to the local data augmenting system;
wherein the central data augmenting system comprises:
a visual-renderer for rendering time-tagged visual representation overlays of the spatially-registered geometric models of objects present in the sensor data;
a doodle-renderer for rendering time-tagged graphical doodle overlays inputted by the user, wherein a user draws doodles on an electronic writing device connected to the display center, and the user chooses between an absolute position for the doodles where the absolute position is spatially-registered to the monitored area, or a relative position for the doodles where the relative position is spatially-registered to an object within the monitored area, and wherein the doodle-renderer renders a time-tagged graphical doodle overlay by using the inputted doodles, the absolute position of the doodles or the relative position of the doodles, the spatially-registered geometric models, and the viewing angle selected be the user;
and
a compositor for composing a time-tagged visual representation overlay onto the video frames from the data acquisition sensors for display on the display center, wherein the user selects a time-tagged visual representation overlay from the plurality of choices of time-tagged visual representation overlays, and wherein the compositor generates a composited augmented reality audiovisual media stream by combining the selected time-tagged visual representation overlay, the time-tagged video frames, a time-tagged audio signal, and the time-tagged graphical doodle overlay.

14 more elements

A method for real-time, group interactive, augmented-reality area monitoring, suitable for enhancing the enjoyment of entertainment events, the method comprising acts of:

receiving sensor data from a sensor data acquisition system, in real-time, where the sensor data acquisition system includes a plurality of data acquisition sensors disposed about an area to be monitored;
receiving sensor data at a plurality of display centers from the plurality of data acquisition sensors, with the act of receiving sensor data further comprising acts of:
receiving input from a user;
selecting, based on the user input, at least one data acquisition sensor from a sensor selector from which to display sensor data;
communicating, in real-time, with other display centers using an interaction interface to allow a user of a display center to interact with users of other display centers;
locally augmenting the display of the sensor data using a local data augmenting system with user input;
and
transmitting the augmented sensor data for display on a display center with a display device;
and
receiving sensor data from the data acquisition system using an object recognition system and recognizing a segment of the sensor data;
wherein the object recognition system extracts spatially-registered geometric models of objects present in the sensor data with a model-extractor;
centrally augmenting sensor data from the data acquisition sensors using a central data augmenting system coupled with the sensor data acquisition system before the sensor data is received by the display centers;
augmenting the sensor data with information regarding the recognized segment for provision to the central data augmenting system;
communicating with a recognition-information database with the object recognition system and with the central data augmenting system so that a user may select an object to be recognized so that the system automatically recognizes the object selected and provides information regarding the object from the recognition information database;
marking the sensor data received from the data acquisition sensors with a data-marker, wherein the data acquisition sensors include at least one of a video camera capturing video frames, an audio recording sensor, and a video/audio camera, wherein the video/audio camera records video signals and audio signals concurrently, wherein the video camera is oriented at a line-of-sight (LOS) orientation viewing angle during a video frame, and where the data-marker marks the sensor data received from a video camera with a “time-tag” corresponding to a video frame and with the line-of-sight (LOS) orientation angle of the video camera at the video frame;
and
selecting, by user input, which other display centers the display center of a user will interact with by using an interaction-selector, and wherein a user of the selected display center can choose to cancel the interaction with a particular display center at any time.

14 more elements

A method for real-time, group interactive, augmented-reality area monitoring, suitable for enhancing the enjoyment of entertainment events, the method comprising acts of:

receiving sensor data from a sensor data acquisition system, in real-time, where the sensor data acquisition system includes a plurality of data acquisition sensors disposed about an area to be monitored;
and
receiving sensor data at a plurality of display centers from the data acquisition centers, with the act of receiving sensor data further comprising acts of:
receiving input from a user;
selecting, based on the user input, at least one data acquisition sensor from a sensor selector from which to display sensor data;
communicating, in real-time, with other display centers using an interaction interface to allow a user of a display center to interact with users of other display centers;
locally augmenting the display of the sensor data using a local data augmenting system with user input;
and
transmitting the augmented sensor data for display on a display center with a display device;
centrally augmenting sensor data from the data acquisition sensors using a 5central data augmenting
receiving sensor data from the data acquisition system using an object recognition system and recognizing a segment of the sensor data and for augmenting the sensor data with information regarding the recognized segment for provision to the central data augmenting system;
communicating with a current-event database with the display center and with the central data augmenting system, whereby users in the display centers can access information of interest related to current events in the area being monitored;
communicating with a related-information database with the display center and with the central data augmenting system, whereby a user in the display center can access information of interest related to prior events, wherein the prior events are related to current events in the area being monitored;
communicating with a recognition-information database with the object recognition system and with the central data augmenting system so that a user may select an object to be recognized so that the system automatically recognizes the object selected and provides information regarding the object from the recognition information database;
marking the sensor data received from the data acquisition sensors with a data-marker, wherein the data acquisition sensors include at least one of a video camera capturing video frames, an audio recording sensor, and a video/audio camera, wherein the video/audio camera records video signals and audio signals concurrently, wherein the video camera is oriented at a line-of-sight (LOS) orientation viewing angle during a video frame, and where the data-marker marks the sensor data received from a video camera with a time-tag corresponding to a video frame and with the line-of-sight (LOS) orientation angle of the video camera at the video frame;
marking an audio signal with a time-tag using an audio-marker, the time-tag corresponding to a video frame synchronized with the audio signal;
adjusting the viewing angle of an angle-controller, wherein the video cameras are digital cameras configured to allow for viewing angle adjustment, and wherein the angle-controller provides for adjusting the viewing angle of the video cameras;
and
selecting, by user input, which other display centers the display center of a user will interact with by using an interaction-selector, and wherein a user of the selected display center can choose to cancel the interaction with a particular display center at any time.

16 more elements

A method for real-time, group interactive, augmented-reality area monitoring, suitable for enhancing the enjoyment of entertainment events, the method comprising acts of:

receiving sensor data from a sensor data acquisition system, in real-time, where the sensor data acquisition system includes a plurality of data acquisition sensors disposed about an area to be monitored;
and
receiving sensor data at a plurality of display centers from the data acquisition centers, with the act of receiving sensor data further comprising acts of:
receiving input from a user;
selecting, based on the user input, at least one data acquisition sensor from a sensor selector from which to display sensor data;
communicating, in real-time, with other display centers using an interaction interface to allow a user of a display center to interact with users of other display centers;
locally augmenting the display of the sensor data using a local data augmenting system with user input;
and
transmitting the augmented sensor data for display on a display center with a display device;
further comprising an act of centrally augmenting sensor data from the data acquisition sensors before the sensor data is received by the display centers using a central data augmenting system coupled with the sensor data acquisition system;
tracking the sensor data marked with time-tag and (LOS) orientation angle with a tracker, wherein the tracker detects object features that represent a recognized object, finds feature locations of the object features in the data acquisition sensors, creates three-dimensional feature location data by fusing the feature locations of the object features from the plurality of data acquisition sensors, and tracks the three-dimensional feature location data in the data acquisition sensors over time;
extracting spatially-registered geometric models of objects present in the sensor data with a model-extractor, wherein the model-extractor extracts the spatially-registered geometric models from the three-dimensional feature location data, and further extracts texture models from the current video frame based on location of the spatially-registered geometric models in the area being monitored;
extracting time-tagged parameters from the spatially-registered geometric models with a parameter-extractor, wherein the parameter-extractor outputs time-tagged parameters that characterize the objects recognized;
associating a spatially-registered geometric model with an actor performing an activity using a model-matcher;
time-tagging the spatially-registered geometric model of the actor with a time-marker;
storing a time sequence of activities performed by an actor with an activity-storage by using time-tagged spatially-registered geometric models of the actor;
and
storing a time sequence of activities performed by an actor with a spatially-registered geometric model database and a spatially-registered geometric model generated by the object recognition system, so that a user may select a spatially-registered geometric model for display on the display center, or so that a user may select to recreate prior activities by displaying on the display center a time sequence of activities previously performed by an actor.

15 more elements

A method for real-time, group interactive, augmented-reality area monitoring, suitable for enhancing the enjoyment of entertainment events, the method comprising acts of:

receiving sensor data from a sensor data acquisition system, in real-time, where the sensor data acquisition system includes a plurality of data acquisition sensors disposed about an area to be monitored;
and
receiving sensor data at a plurality of display centers from the data acquisition centers, with the act of receiving sensor data further comprising acts of:
receiving input from a user;
selecting, based on the user input, at least one data acquisition sensor from a sensor selector from which to display sensor data;
communicating, in real-time, with other display centers using an interaction interface to allow a user of a display center to interact with users of other display centers;
locally augmenting the display of the sensor data using a local data augmenting system with user input;
and
transmitting the augmented sensor data for display on a display center with a display device;
receiving sensor data from the data acquisition system;
recognizing a segment of the sensor data;
and
augmenting the sensor data with information regarding the recognized segment for provision to the local data augmenting system using an object recognition system,
rendering time-tagged visual representation overlays of the spatially-registered geometric models of objects present in the sensor data with a visual-renderer, wherein the visual-renderer renders a time-tagged visual representation overlay by combining information provided by the current-event database and the related-information database with the spatially-registered geometric models and the viewing angle selected be the user, and wherein the visual-renderer provides the user with a plurality of choices of time-tagged visual representation overlays for each input query requested by the user;
rendering time-tagged graphical doodle overlays inputted by the user with a doodle-renderer, wherein a user draws doodles on an electronic writing device connected to the display center, and the user chooses between an absolute position for the doodles where the absolute position is spatially-registered to the monitored area or a relative position for the doodles where the relative position is spatially-registered to an object within the monitored area, and wherein the doodle-renderer renders a time-tagged graphical doodle overlay by using the inputted doodles, the absolute position of the doodles or the relative position of the doodles, the spatially-registered geometric models, and the viewing angle selected be the user;
and
composing a time-tagged visual representation overlay onto the video frames from the data acquisition sensors for display on the display center using a compositor, wherein the user selects a time-tagged visual representation overlay from the plurality of choices of time-tagged visual representation overlays, and wherein the compositor generates a composited augmented reality audiovisual media stream by combining the selected time-tagged visual representation overlay, the time-tagged video frames, the time-tagged audio signal, and the time-tagged graphical doodle overlay.

14 more elements

Dependencies are inferred from the claim text.

Facts

Application: 11/019,991Filed 20 Dec 2004 · us provisional application
Priority: 19 Dec 2003
Granted: 14 Sep 2010
Term ends: Not estimatedThe source records don’t support an estimate.

Classification9 codes

G06V 20/52
Context or environment of the image: surveillance or monitoring of activities, e.g. for recognising suspicious objects
G06T 19/006
Manipulating three-dimensional [3D] models or images for computer graphics: mixed reality
A63F 2300/636
For controlling the execution of the game in time: involving process of starting or resuming a game
A63F 2300/69
Methods for processing data by generating or executing the game program: involving elements of the real world in the game world, e.g. measurement…