# US 2003/0080978 A1 · Augmented reality system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2003/0080978 A1
- Document ID: US20030080978A1
- Status: Published application
- Kind: A1
- Application number: 10/263,578
- Filed: 2002-10-03
- Published: 2003-05-01
- Earliest claimed: 2001-10-04
- Assignees: Siemens Corporate Research, Inc.
- Inventors: Nassir Navab; Xiang Zhang; Yakup Genc; Vijay Kumar
- Web page: https://patentgrep.com/patent/US20030080978A1

## CPC

- G01S 5/163: {Determination of attitude (using inertial means G01C9/00; control of attitude G05D1/49)}
- G01S 5/16: using electromagnetic waves other than radio waves
- A63F 2300/1012: involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
- G05B 2219/32014: Augmented reality assists operator in maintenance, repair, programming, assembly, use of head mounted display with 2-D 3-D display and voice feedback, voice and gesture command
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

An augmented reality system includes a video source, and a database. The video source resides at a location and produces an image. At least one encoded marker resides within the image. A marker detector is coupled to the video source and is adapted to derive encoded data from the marker residing within the image. A localization processor is adapted to receive data from the marker detector and to generate data regarding location and orientation of the marker. The localization processor retrieves information from the database that is related to the location and orientation of the marker. The localization processor makes the information retrieved from the database available to a user.

## Claims

### Claim 1

An augmented reality system, comprising:
a video source, the video source residing at a location and producing an image;
at least one encoded marker residing within the image;
a marker detector, coupled to the video source, adapted to derive encoded data from the marker residing within the image;
a database; and
a localization processor adapted to receive data from the marker detector and to generate data regarding location and orientation of the marker, the localization processor retrieving information from the database that is related to the location and orientation of the marker, the localization processor making the information retrieved from the database available to a user.

### Claim 2 (depends on 1)

The augmented reality system of claim 1 wherein the encoded marker is formed to store data within a matrix.

### Claim 3 (depends on 2)

The augmented reality system of claim 2 wherein the matrix is formed to include at least one cell reserved for marker orientation data.

### Claim 4 (depends on 2)

The augmented reality system of claim 2 wherein at least some cells within the matrix contain an indicator of a portion of a numerical value.

### Claim 5 (depends on 4)

The augmented reality system of claim 4 wherein the indicator within each cell is a binary indicator.

### Claim 6 (depends on 5)

The augmented reality system of claim 5 wherein the binary indicator comprises:
a first state indicated by a vacant cell; and
a second state indicated by a nonvacant cell.

### Claim 7 (depends on 6)

The augmented reality system of claim 6 wherein a nonvacant cell is occupied by a substantially opaque circle.

### Claim 8 (depends on 2)

The augmented reality system of claim 2 wherein the matrix is surrounded by a substantially opaque frame.

### Claim 9 (depends on 8)

The augmented reality system of claim 8 wherein the matrix and the frame are substantially rectangular.

### Claim 10 (depends on 8)

The augmented reality system of claim 8 wherein the frame is shaped so as to reveal corners when the plane is viewed from a plane not containing the frame.

### Claim 11

A computer assisted localization system, comprising:
a computer;
a database;
a camera;
at least one marker containing encoded data within a matrix;
a tracker, receiving images from the camera, and detecting an image of the marker within the image received by the camera; and
a localization processor, using data from the computer and the tracker to calculate a position of the camera within an environment, and retrieving from the database data associated with the position for display to a user of the system.

### Claim 12 (depends on 11)

The system of claim 11, wherein the tracker further comprises:
a closed string detector, for identifying a series of points within an image that form an enclosed region; and
a calculator, for determining a characteristic interior location for each enclosed region and identifying those regions having similarly located characteristic interior locations that correspond to marker geometry.

### Claim 13 (depends on 12)

The system of claim 12, wherein the marker further comprises:
a substantially opaque frame;
a matrix of rows and columns forming cells within the frame; and
a series of substantially opaque indicators occupying at least some of the cells so as to indicate orientation and identification of the marker.

### Claim 14 (depends on 13)

The system of claim 13 wherein the frame of each marker includes a plurality of corners, the tracker detecting the corners in order to establish the presence of a marker within an image.

### Claim 15 (depends on 13)

The system of claim 13 wherein the frame of each marker is formed so as to have an exterior edge string and an interior edge string, the tracker calculating relative lengths of exterior and interior edge strings to establish the presence of a marker within an image.

### Claim 16

A method of selecting database images and data for use in an augmented reality system, comprising the steps of:
marking items within an environment with respective markers that indicate marker orientation and item identification;
viewing the items with a camera interconnected to a computer which accesses the database;
detecting the markers viewed by the camera; and
associating particular images and data with each marker for display to a user when a particular image is viewed by the camera.

### Claim 17 (depends on 16)

The method of claim 16 further comprising the step of calculating a camera location based on information derived from a marker viewed by the camera, the marker identifying a best available image residing in the database corresponding to the camera location.

### Claim 18 (depends on 16)

The method of claim 16 further comprising the steps of:
calculating a camera location based on information derived from a marker viewed by the camera;
calculating a distance between the camera location and respective virtual camera locations for which images exist in the database; and
selecting an image from the database corresponding to a virtual camera location that is closest to the calculated camera location.

### Claim 19 (depends on 16)

The method of claim 16, further comprising the steps of:
calculating camera orientation; and
selecting an image from the database corresponding to a virtual camera having a substantially similar orientation.

### Claim 20 (depends on 16)

The method of claim 16, further comprising the steps of:
determining an optical axis of the camera;
calculating an optical center of the camera; and
selecting an image from the database produced from a viewpoint and in a direction substantially similar to the optical axis and the optical center of the camera.
