# US 12,001,751 B2 · Shared data and collaboration for head-mounted devices

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,001,751 B2
- Document ID: US12001751B2
- Status: Granted
- Kind: B2
- Application number: 18/214,448
- Filed: 2023-06-26
- Published: 2023-11-02
- Granted: 2024-06-04
- Earliest claimed: 2019-04-18
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Marinus Meursing; David A. Schmuck; Brian S. Lau; Jeremy C. Franklin
- Web page: https://patentgrep.com/patent/US12001751B2

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/147: using display panels
- G02B 27/0172: {characterised by optical features}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06V 20/20: in augmented reality scenes
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/0141: {characterised by the informative content of the display}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A system can include head-mounted devices that collaborate to process views from cameras of the respective head-mounted devices and identify objects from different perspectives and/or objects that are within the view of only one of the head-mounted devices. Sharing sensory input between multiple head-mounted devices can complement and enhance individual units by interpreting and reconstructing objects, surfaces, and/or an external environment with perceptive data from multiple angles and positions, which also reduces occlusions and inaccuracies. As more detailed information is available at a specific moment in time, the speed and accuracy of object recognition, hand and body tracking, surface mapping, and/or digital reconstruction can be improved. Such collaboration can provide more effective and efficient mapping of space, surfaces, objects, gestures and users.

## Claims

### Claim 1

A head-mounted device comprising:
a camera for capturing a first view including a first physical object;
a display;
a communication element for receiving data from an additional head-mounted device comprising an additional camera for capturing a second view including a second physical object that is not in the first view, wherein the additional head-mounted device is within the first view; and
a processor configured to:
receive, from the additional head-mounted device, the data, the data being based on the second view; and
determine a characteristic of the first physical object based on the first view captured by the camera and the data from the additional head-mounted device;
transmit, to the additional head-mounted device, a signal indicating the characteristic of the first physical object;
operate the display to output a virtual object as a substitute for the first physical object based on the characteristic of the first physical object;
output a representation of the second physical object based on the data received from the additional head-mounted device; and
determine whether, based on the first view, the additional camera is arranged to capture the second view of the first physical object.

### Claim 2 (depends on 1)

The head-mounted device of claim 1, wherein the characteristic comprises an identity of the first physical object.

### Claim 3 (depends on 1)

The head-mounted device of claim 1, wherein the processor is further configured to:
determine whether the additional camera of the additional head-mounted device is arranged to capture the second view of the first physical object; and
transmit, with the communication element and to the additional head-mounted device, a request for the data based on the second view captured by the additional camera.

### Claim 4 (depends on 1)

The head-mounted device of claim 1, wherein the processor is further configured to:
receive, with the communication element, arrangement information of the additional head-mounted device; and
determine whether, based on the arrangement information, the additional camera is arranged to capture the second view of the first physical object is.

### Claim 5 (depends on 1)

A system comprising:
the head-mounted device of claim 1;
the additional head-mounted device; and
an external device in communication with the head-mounted device and the additional head-mounted device.

### Claim 6 (depends on 5)

The system of claim 5, wherein the data is transmitted from the additional head-mounted device to the head-mounted device via the external device, and the signal is transmitted from the head-mounted device to the additional head-mounted device via the external device.

### Claim 7 (depends on 5)

The system of claim 5, wherein the external device is configured to:
determine an additional characteristic of the first physical object based on the first view and the data; and
transmit, to the head-mounted device and the additional head-mounted device, an additional signal indicating the additional characteristic of the first physical object.

### Claim 8 (depends on 5)

The system of claim 5, wherein the external device is configured to:
receive the data from the additional head-mounted device;
store the data until the camera captures the first view including the first physical object; and
transmit the data to the head-mounted device.

### Claim 9

A head-mounted device comprising:
a camera for capturing a first view including a first physical object;
a display; and
a processor configured to:
receive, with a communication element and from an additional head-mounted device, data based on a second view captured by an additional camera of the additional head-mounted device, the second view including the first physical object and a second physical object, the second physical object not being in the first view;
determine a characteristic of the first physical object based on the first view captured by the camera and the data from the additional head-mounted device;
transmit, with the communication element and to the additional head-mounted device, a signal indicating the characteristic of the first physical object;
operate the display to output the first view with a virtual object that replaces the first physical object in the first view, the virtual object being based on the characteristic of the first physical object;
output a representation of the second physical object based on the data received from the additional head-mounted device;
determine whether the additional camera of the additional head-mounted device is arranged to capture the second view of the first physical object; and
transmit, with the communication element and to the additional head-mounted device, a request for the data.

### Claim 10 (depends on 9)

The head-mounted device of claim 9, wherein:
the additional head-mounted device is within the first view; and
the processor is further configured to determine whether, based on the first view, the additional camera is arranged to capture the second view of the first physical object.

### Claim 11 (depends on 9)

The head-mounted device of claim 9, wherein the processor is further configured to:
receive, with the communication element, arrangement information of the additional head-mounted device; and
determine whether, based on the arrangement information, the additional camera is arranged to capture the second view of the first physical object is.

### Claim 12 (depends on 9)

The head-mounted device of claim 9, wherein the characteristic comprises an identity of the first physical object.

### Claim 13 (depends on 9)

A system comprising:
the head-mounted device of claim 9;
the additional head-mounted device; and
an external device in communication with the head-mounted device and the additional head-mounted device.

### Claim 14 (depends on 13)

The system of claim 13, wherein the data is transmitted from the additional head-mounted device to the head-mounted device via the external device, and the signal is transmitted from the head-mounted device to the additional head-mounted device via the external device.

### Claim 15

A head-mounted device comprising:
a camera;
a display;
a communication element; and
a processor configured to:
capture a first view of a first physical object with the camera;
receive, with the communication element, data from an additional head-mounted device comprising an additional camera, the data being based on a second view of the first physical object captured by the additional camera, the second view further including a second physical object that is not in the first view;
determine a characteristic of the first physical object based on the first view and the data;
transmit, with the communication element and to the additional head-mounted device, a signal indicating the characteristic of the first physical object;
operate the display to output a virtual object as a substitute for the first physical object based on the characteristic of the first physical object; and
output a representation of the second physical object based on the data received from the additional head-mounted device; and
receive, with the communication element, arrangement information of the additional head-mounted device; and
determine whether, based on the arrangement information, the additional camera is arranged to capture the second view of the first physical object.

### Claim 16 (depends on 15)

The head-mounted device of claim 15, wherein the characteristic comprises an identity of the first physical object.

### Claim 17 (depends on 15)

The head-mounted device of claim 15, wherein the processor is further configured to:
determine whether the additional camera of the additional head-mounted device is arranged to capture the second view of the first physical object; and
transmit, with the communication element and to the additional head-mounted device, a request for the data based on the second view captured by the additional camera.