# US 12,120,423 B2 · Augmented reality display with frame modulation functionality

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,120,423 B2
- Document ID: US12120423B2
- Status: Granted
- Kind: B2
- Application number: 17/235,820
- Filed: 2021-04-20
- Published: 2021-12-09
- Granted: 2024-10-15
- Earliest claimed: 2018-06-18
- Assignees: Magic Leap, Inc.
- Applicants: Magic Leap, Inc.
- Inventors: Brian Keith Smith; Carlos A. Rivera Cintron; Jose Felix Rodriguez; Matthew Thomas Hull; Gregory Michael Link
- Web page: https://patentgrep.com/patent/US12120423B2

## CPC

- H04N 23/667: Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
- G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking}
- G02B 27/0172: {characterised by optical features}
- G02B 27/0179: {Display position adjusting means not related to the information to be displayed}
- G06F 1/163: {Wearable computers, e.g. on a belt}
- G06F 1/1686: {the I/O peripheral being an integrated camera}
- G06F 1/325: {Power saving in peripheral device}
- G06F 1/3265: {Power saving in display device}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/248: {involving reference images or patches}
- G09G 5/003: {Details of a display terminal, the details relating to the control arrangement of the display terminal and to the interfaces thereto (specific for a CRT G09G1/165; for a flat panel G09G3/2092)}
- H04N 13/279: the virtual viewpoint locations being selected by the viewers or determined by tracking
- H04N 13/344: with head-mounted left-right displays
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- G09G 2320/0261: in the context of movement of objects on the screen or movement of the observer relative to the screen
- G09G 2330/021: Power management, e.g. power saving
- G09G 2340/0435: Change or adaptation of the frame rate of the video stream
- Y02D 10/00: Energy efficient computing, e.g. low power processors, power management or thermal management
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)
- Y02D: CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE

## Abstract

A head mounted display system can process images by assessing relative motion between the head mounted display and one or more features in a user's environment. The assessment of relative motion can include determining whether the head mounted display has moved, is moving and/or is expected to move with respect to one or more features in the environment. Additionally or alternatively, the assessment can include determining whether one or more features in the environment have moved, are moving and/or are expected to move relative to the head mounted display. The image processing can further include determining one or more virtual image content locations in the environment that correspond to a location where renderable virtual image content appears to a user when the location appears in the display and comparing the one or more virtual image content locations in the environment with a viewing zone.

## Claims

### Claim 1

A head mounted display system configured to project light to an eye of a user to display augmented reality image content in a vision field of said user and to image at least a portion of an environment in front of the user wearing the head mounted display system, said head-mounted display system comprising:
a frame configured to be supported on a head of the user;
a head-mounted display disposed on the frame, said display configured to project light into said user's eye to display augmented reality image content to the user's vision field, at least a portion of said display being transparent and disposed at a location in front of the user's eye when the user wears the frame such that said transparent portion transmits light from a portion of the environment in front of the user and said head mounted display to the user's eye to provide a view of said portion of the environment in front of the user and said head-mounted display;
a first camera configured to obtain images at a first frame rate; and
a second camera configured to obtain images at a second frame rate different from the first frame rate while the first camera obtains images at the first frame rate.

### Claim 2 (depends on 1)

The system of claim 1, wherein the first camera is on the left eye side of the head-mounted display and the second camera is on the right eye side of the head-mounted display, and in response to a rotation of the head-mounted display to the left, the first rate is set to be higher than the second frame rate.

### Claim 3 (depends on 1)

The system of claim 1, wherein the first camera is on the left eye side of the head-mounted display and the second camera is on the right eye side of the head-mounted display, and in response to a rotation of the head-mounted display to the right, the second frame rate is set to be higher than the first frame rate.

### Claim 4 (depends on 1)

The system of claim 1, further comprising:
a non-transitory computer readable medium storing computer executable instructions; and
a hardware processor in communication with the first and second cameras and the computer readable medium, the hardware processor configured to execute the computer executable instructions in order to:
determine a property of relative motion between the head-mounted display and one or more features in the environment; and
based on the property of relative motion, set a frame rate of the first camera such that the first frame rate is different than the second frame rate.

### Claim 5 (depends on 4)

The system of claim 4, wherein the hardware processor is further configured to execute the computer executable instructions in order to:
determine a second property of relative motion between the head-mounted display and one or more features in the environment; and
based on the second property of relative motion, set a frame rate of the second camera.

### Claim 6 (depends on 4)

The system of claim 4, wherein the hardware processor is further configured to execute the computer executable instructions in order to change the frame rate of the first camera in response to a determination that a condition has been met.

### Claim 7 (depends on 5)

The system of claim 5, wherein setting the frame rate of the first and/or second camera comprises setting a frame rate different from a frequency of electrical current in the head mounted display or an integer multiple thereof.

### Claim 8 (depends on 4)

The system of claim 4, wherein determining the property of relative motion of the head mounted display and the one or more features in the environment comprises determining the property of relative motion of the head mounted display.

### Claim 9 (depends on 5)

The system of claim 5, wherein each of the property of relative motion and the second property of relative motion comprises corresponding different measurements of a common type of property of motion.

### Claim 10 (depends on 5)

The system of claim 5, wherein the second property of relative motion comprises a second measurement in time.

### Claim 11 (depends on 1)

The system of claim 1, wherein each of the first and second cameras is configured to detect nonidentical portions of the environment.

### Claim 12 (depends on 4)

The system of claim 4, wherein the hardware processor is in communication with the first and second cameras and is configured to execute the computer executable instructions in order to determine a frame rate of the second camera based on an image received from the first camera.

### Claim 13 (depends on 12)

The system of claim 12, wherein the image comprises a direction of a feature of the environment.

### Claim 14 (depends on 12)

The system of claim 12, wherein the image comprises a speed of a feature of the environment.

### Claim 15 (depends on 12)

The system of claim 12, wherein determining a frame rate of the second camera based on the image received from the first camera comprises predicting a trend of a feature in the image received from the first camera being detected by an image received by the second camera.

### Claim 16 (depends on 15)

The system of claim 15, wherein adjusting the frame rate of the second camera based on the image received from the first camera comprises increasing the frame rate of the second camera.

### Claim 17 (depends on 12)

The system of claim 12, wherein the hardware processor is further configured to execute the computer executable instructions in order to adjust the frame rate of the second camera based on the image received from the first camera.

### Claim 18 (depends on 17)

The system of claim 17, wherein adjusting the frame rate of the second camera based on the image received from the first camera comprises decreasing the frame rate of the second camera.

### Claim 19 (depends on 1)

The system of claim 1, wherein the first camera is configured to detect a first portion of the environment and the second camera is configured to detect a second portion of the environment, the first portion of the environment overlapping at least partially with the second portion of the environment.

### Claim 20 (depends on 17)

The system of claim 17, wherein adjusting the frame rate of the second camera based on the image received from the first camera is based at least in part on a frequency of electrical current within the head mounted display system.