# US 2024/0404218 A1 · Augmented reality system and method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0404218 A1
- Document ID: US20240404218A1
- Status: Published application
- Kind: A1
- Application number: 18/680,899
- Filed: 2024-05-31
- Published: 2024-12-05
- Earliest claimed: 2023-06-02
- Assignees: The Boeing Company
- Applicants: The Boeing Company
- Inventors: Samuel Helm Neblett; David Wesley Ross; Alexus Hunter; Richard T. Moore; Brian Dale Laughlin
- Web page: https://patentgrep.com/patent/US20240404218A1

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

An augmented reality (AR) system for use by a user in a surrounding environment includes a head-mounted display (HMD), a handheld computing device (HCD) and a control module operatively connected with one or both of the HMD and the HCD. The system is configured to enable digital content to be displayed simultaneously on the HMD and on the HCD, and/or to use one of the HMD and the HCD to modify digital content or to execute an action on the other of the HMD and the HCD, with the HCD being positionable by the user outside of the HMD's lens field of view, thereby providing an enhanced or extended user field of view.

## Claims

### Claim 1

An augmented reality (AR) system for use by a user in a surrounding environment, comprising:
a head-mounted display (HMD) having a lens that is generally transparent and/or permits video pass-through functionality and through which the surrounding environment is visible to the user, a predetermined portion of the lens for displaying digital content thereon and/or therethrough for viewing by the user, and a first group of sensors configured for sensing a user action and/or a met condition;
a handheld computing device (HCD) having an HCD display and a second group of sensors configured for sensing the user action and/or the met condition; and
a control module operatively connected with one or both of the HMD and the HCD, wherein the control module is configured to:
sense the user action and/or the met condition; and
if the user action and/or the met condition is sensed by the first group of sensors, then modify digital content displayed on the HCD display based on the sensed user action and/or the sensed met condition; and
if the user action and/or the met condition is sensed by the second group of sensors, then modify digital content displayed on the predetermined portion of the lens based on the sensed user action and/or the sensed met condition.

### Claim 2 (depends on 1)

The AR system of claim 1, wherein when wearing the HMD, the user has a natural field of view, the generally transparent lens has a lens field of view as viewed by the user that is within and smaller than the natural field of view, and the virtual images are projectable onto a virtual image field of view as viewed by the user that is within and smaller than the lens field of view, wherein the HCD may be positioned by the user at least partially outside of the virtual image field of view.

### Claim 3 (depends on 1)

The AR system of claim 1, wherein the user action includes one or more of a roll of the HMD, a pitch of the HMD, a yaw of the HMD, a translation of the HMD, an eye movement, a hand gesture, a tool gesture, a voice command spoken to the HMD, a push of a button on the HCD, a gesture on a touchscreen/touchpad of the HCD, a shaking of the HCD and a voice command spoken to the HCD.

### Claim 4 (depends on 1)

The AR system of claim 1, wherein the control module is further configured to permit a first modification of digital content on one of the predetermined portion of the lens and the HCD display, and simultaneously cause a second modification of digital content on the other of the predetermined portion of the lens and the HCD display, wherein the second modification is substantially the same as the first modification.

### Claim 5 (depends on 1)

The AR system of claim 1, wherein the control module is further configured to permit a first modification of digital content on one of the predetermined portion of the lens and the HCD display without modifying digital content simultaneously on the other of the predetermined portion of the lens and the HCD display.

### Claim 6 (depends on 1)

The AR system of claim 1, wherein the AR system is configured to enable the user to add additional digital content to the HCD display with the additional digital content also being displayed simultaneously on the predetermined portion of the lens.

### Claim 7 (depends on 1)

The AR system of claim 1, wherein the AR system is configured to enable the user to add additional digital content to the HCD display without the additional digital content being displayed simultaneously on the predetermined portion of the lens.

### Claim 8 (depends on 1)

The AR system of claim 1, wherein the AR system is configured to enable the user to add additional digital content to the HCD display and to utilize one or more applications on the HCD for processing the additional digital content.

### Claim 9 (depends on 1)

The AR system of claim 1, further comprising:
one or both of a local ad hoc connection between the HMD and the HCD for facilitating communication therebetween; and
a server operatively connected with the HMD and the HCD for facilitating communication with and/or between the HMD and the HCD.

### Claim 10 (depends on 1)

The AR system of claim 1, wherein the control module is embedded as a respective portion of one or both of the HMD and the HCD.

### Claim 11 (depends on 1)

The AR system of claim 1, wherein the control module is further configured to cause the digital content to be displayed simultaneously on the predetermined portion of the lens and on the HCD display.

### Claim 12

A method of modifying digital content within an AR system that includes a head-mounted display (HMD) having a first group of sensors configured for sensing a user action and/or a met condition, a handheld computing device (HCD) having a second group of sensors configured for sensing the user action and/or the met condition, and a control module operatively connected with one or both of the HMD and the HCD, the method comprising:
sensing the user action and/or the met condition; and
if the user action and/or the met condition is sensed by the first group of sensors, then modifying digital content displayed on the HCD based on the sensed user action and/or the sensed met condition; and
if the user action and/or the met condition is sensed by the second group of sensors, then modifying digital content displayed on the HMD based on the sensed user action and/or the sensed met condition.

### Claim 13 (depends on 12)

The method of claim 12, further comprising:
receiving a first command from the HMD based on the sensed user action and/or the sensed met condition; and
modifying the digital content on the HCD based on the received first command.

### Claim 14 (depends on 12)

The method of claim 12, further comprising:
receiving a second command from the HCD based on the sensed user action and/or the sensed met condition; and
modifying the digital content on the HMD based on the received second command.

### Claim 15 (depends on 12)

The method of claim 12, further comprising:
displaying additional digital content on the HCD with the additional digital content also being displayed simultaneously on the HMD; or
displaying additional digital content on the HCD without the additional digital content also being displayed simultaneously on the HMD.

### Claim 16 (depends on 12)

The method of claim 12, further comprising:
making a first modification of the digital content on one of the HMD and the HCD; and
simultaneously making a second modification of the digital content on the other of the HMD and the HCD, wherein the second modification is substantially the same as the first modification.

### Claim 17 (depends on 12)

The method of claim 12, further comprising:
making a first modification of the digital content on one of the HMD and the HCD without modifying the digital content simultaneously on the other of the HMD and the HCD.

### Claim 18 (depends on 12)

The method of claim 12, further comprising:
displaying additional digital content on the HCD; and
utilizing one or more applications on the HCD for processing the additional digital content.

### Claim 19 (depends on 12)

The method of claim 12, further comprising:
simultaneously displaying the digital content on the HMD and on the HCD based on the sensed user action and/or the sensed met condition.

### Claim 20

A method of using an augmented reality (AR) system by a user in a surrounding environment, wherein the AR system includes a head-mounted display (HMD) having a first group of sensors configured for sensing a user action, a handheld computing device (HCD) having a second group of sensors configured for sensing the user action, and a control module operatively connected with one or both of the HMD and the HCD, the method comprising:
executing the user action by the user;
sensing the user action by one of the HMD, via the first group of sensors, and the HCD, via the second group of sensors; and
modifying, by the control module, digital content displayed on the other of the HMD and the HCD based on the sensed user action.
