# US 2013/0155307 A1 · Synchronized, interactive augmented reality displays for multifunction devices

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2013/0155307 A1
- Document ID: US20130155307A1
- Status: Published application
- Kind: A1
- Application number: 13/768,072
- Filed: 2013-02-15
- Published: 2013-06-20
- Earliest claimed: 2010-01-05
- Assignees: Apple Inc.
- Inventors: Brett C. Bilbrey; Nicholas V. King; Aleksandar Pance
- Web page: https://patentgrep.com/patent/US20130155307A1

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06F 1/1694: {the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G09G 5/377: for mixing or overlaying two or more graphic patterns (G09G5/02, G09G5/397 take precedence)
- H04N 1/00323: {with a measuring, monitoring or signaling apparatus, e.g. for transmitting measured information to a central location}
- H04N 5/222: Studio circuitry; Studio devices; Studio equipment (cameras or camera modules comprising electronic image sensors, or control thereof H04N23/00)
- H04N 5/272: Means for inserting a foreground image in a background image, i.e. inlay, outlay
- H04N 21/2187: Live feed
- H04N 21/42202: {environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes}
- H04N 21/4312: {involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations}
- H04N 21/4314: {for fitting data in a restricted space on the screen, e.g. EPG data in a rectangular grid}
- H04N 21/472: End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content {(end-user interfaces for retrieving video data from a database G06F16/739; network services for supporting unicast streaming H04L65/612)}
- H04N 21/4788: communicating with other users, e.g. chatting {(arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104)}
- H04N 21/8133: {specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video programme}
- H04N 23/632: {for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters}
- H04N 23/661: Transmitting camera control signals through networks, e.g. control via the Internet
- G06F 2203/04803: Split screen, i.e. subdividing the display area or the window area into separate subareas
- G06T 2219/004: Annotating, labelling
- G09G 2340/12: Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
- G09G 2340/125: wherein one of the images is motion video
- G09G 2340/14: Solving problems related to the presentation of information to be displayed
- G09G 2354/00: Aspects of interface with display user
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A device can receive live video of a real-world, physical environment on a touch sensitive surface. One or more objects can be identified in the live video. An information layer can be generated related to the objects. In some implementations, the information layer can include annotations made by a user through the touch sensitive surface. The information layer and live video can be combined in a display of the device. Data can be received from one or more onboard sensors indicating that the device is in motion. The sensor data can be used to synchronize the live video and the information layer as the perspective of video camera view changes due to the motion. The live video and information layer can be shared with other devices over a communication link.

## Claims

### Claim 1

A machine-readable storage medium comprising a plurality of instructions that, in response to being executed on a computing device, cause the computing device to:
capture live video of a real-world, physical environment, and display the live video on a touch sensitive surface of the computing device;
combine an information layer and the live video, the information layer related to one or more objects in the live video;
generate computer-generated imagery based on the live video;
display the computer-generated imagery representing one or more objects in the live video on the touch sensitive surface;
overlay the information layer on the computer-generated imagery;
receive sensor data from one or more onboard motion sensors indicating that the computing device is in motion; and
synchronize the display of the live video, the computer-generated imagery, and the information layer on the touch sensitive surface using the sensor data.

### Claim 2 (depends on 1)

The machine-readable storage medium of claim 1, further comprising instructions that cause the computing device to:
share at least one of the synchronized live video, the computer-generated imagery, and information layer with a second device over a communication link.

### Claim 3 (depends on 1)

The machine-readable storage medium of claim 1, further comprising instructions that cause the computing device to:
generate a split screen display having first and second display areas, the first display area configured for displaying the live video combined with the information layer and the second display area for displaying the computer-generated imagery combined with the information layer.

### Claim 4 (depends on 1)

The machine-readable storage medium of claim 1, further comprising instructions that cause the computing device to:
display a navigation control for navigating the computer-generated imagery on the touch sensitive surface.

### Claim 5 (depends on 1)

The machine-readable storage medium of claim 1, further comprising instructions that cause the computing device to:
perform object recognition on the live video;
obtain object information for one or more recognized objects; and
generate the information layer for the one or more recognized objects.

### Claim 6 (depends on 5)

The machine-readable storage medium of claim 5, where the instructions that cause the computing device to generate the information layer, further comprise instructions that cause the computing device to:
include in the information layer one or more links to one or more network resources providing information related to the one or more recognized objects.

### Claim 7 (depends on 5)

The machine-readable storage medium of claim 5, further comprising instructions that cause the computing device to:
label the recognized one or more objects in the live video display.

### Claim 8 (depends on 7)

The machine-readable storage medium of claim 7, where at least one object is labeled with a bar code for identifying the object during the object recognition.

### Claim 9 (depends on 7)

The machine-readable storage medium of claim 7, where at least one object is labeled with one or more markers for identifying an orientation of the object in the live video during object recognition.

### Claim 10 (depends on 7)

The machine-readable storage medium of claim 7, where at least one label includes content related to the recognized object.

### Claim 11 (depends on 1)

The machine-readable storage medium of claim 1, further comprising instructions that cause the computing device to:
receive input through the touch sensitive display, the input identifying an object in the live video;
perform object recognition on the live video to identify the identified object;
obtain object information for the recognized object; and
generate an information layer based on the object information.

### Claim 12 (depends on 1)

The machine-readable storage medium of claim 1, where the computing device is a mobile device.

### Claim 13 (depends on 12)

The machine-readable storage medium of claim 12, where at least one of the computing device and the second device is an electronic tablet.

### Claim 14 (depends on 1)

The machine-readable storage medium of claim 1, further comprising instructions that cause the computing device to:
determine a route between the computing device and a destination; and
display the route with the computer-generated imagery.

### Claim 15 (depends on 1)

The machine-readable storage medium of claim 1, where an object in the real-world, physical environment captured by the live video is a document, the machine-readable storage medium further comprising instructions that cause the computing device to:
generate annotations related to the document; and
overlay the annotations proximate the document on the live video.

### Claim 16 (depends on 1)

The machine-readable storage medium of claim 1, where an object in the real-world, physical environment captured by the live video is a machine, circuit board or part thereof, the machine-readable storage medium further comprising instructions that cause the computing device to:
generate annotations related to the object; and
overlay the annotations on the live video or computer-generated imagery.

### Claim 17 (depends on 1)

The machine-readable storage medium of claim 1, where the instructions that cause the computing device to generate the computer-generated imagery based on the live video includes instructions that cause the computing device to generate a three-dimensional model of the one or more objects in the live video.

### Claim 18 (depends on 1)

The machine-readable storage medium of claim 1, where the instructions that cause the computing device to combine the information layer includes instructions to generate one or more annotations for display in response to a touch input at a location on the touch sensitive surface corresponding to the object.

### Claim 19

An augmented reality system, comprising:
a touch sensitive surface configured for receiving touch input;
one or more onboard motion sensors configured for sensing motion;
a video camera for capturing live video of a real-world, physical environment, for display on the touch sensitive surface;
a processor coupled to the touch sensitive surface, the motion sensor and the video camera, the processor configured for,
combining an information layer and the live video for display on the touch sensitive surface,
modeling computer-generated imagery based on the live video,
displaying computer-generated imagery representing one or more objects in the live video on the touch sensitive surface,
overlaying the information layer on the computer-generated imagery for display on the touch sensitive surface,
receiving sensor data from the one or more onboard motion sensors indicating that the apparatus is in motion, and
synchronizing the live video, computer-generated imagery, and the information layer using the sensor data; and
a communication interface configured for sharing the live video over a communication link.

### Claim 20 (depends on 19)

The system of claim 19, where modeling the computer-generated imagery based on the live video includes generating a three-dimensional model of the one or more objects in the live video.

### Claim 21 (depends on 19)

The system of claim 19, where the processor is further configured for:
generating a split screen display having first and second display areas, the first display area configured for displaying the live video combined with the information layer and the second display area for displaying the computer-generated imagery combined with the information layer; and
displaying the split screen on the touch sensitive surface.

### Claim 22 (depends on 19)

The system of claim 19, where the processor is further configured for:
accessing an information resource via the communication interface; and
retrieving information to be presented on the touch sensitive surface as the information layer.

### Claim 23 (depends on 19)

The system of claim 19, where the information layer includes one or more annotations related to an object in the live video, the one or more annotations being displayed in response to the touch input at a location on the touch sensitive surface corresponding to the object.

### Claim 24 (depends on 23)

The system of claim 23, where the one or more annotations may include any one of: specifications, directions, recipes, data sheets, images, video clips, audio files, schemas, user interface elements, thumbnails, text, references, hyperlinks, telephone numbers, notes, part numbers, dictionary definitions, catalog data, serial numbers, order forms, marketing material or advertisements.

### Claim 25

A mobile device comprising:
a touch sensitive display configured for displaying a graphical user interface and receiving touch input;
a sensor configured to generate data related to position of the mobile device;
a video camera to capture live video for display on the touch sensitive display; and
a processor coupled to the touch sensitive display, the sensor and the video camera, the processor configured to,
generate an information layer including data related to an object within the live video captured by the video camera,
integrate the information layer and the live video for display on the touch sensitive display,
model computer-generated imagery based at least in part on the live video,
display the computer-generated imagery representing one or more objects in the live video on the touch sensitive display,
overlay the information layer on the computer-generated imagery for display on the touch sensitive display,
process sensor data from the sensors indicating that the mobile device is in motion, and
synchronize the live video, computer-generated imagery, and the information layer using the sensor data.