# US 11,906,742 B2 · Periocular test for mixed reality calibration

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,906,742 B2
- Document ID: US11906742B2
- Status: Granted
- Kind: B2
- Application number: 17/385,724
- Filed: 2021-07-26
- Published: 2022-01-20
- Granted: 2024-02-20
- Earliest claimed: 2016-10-05
- Assignees: Magic Leap, Inc.
- Applicants: Magic Leap, Inc.
- Inventors: Adrian Kaehler; Gary Bradski; Vijay Badrinarayanan
- Web page: https://patentgrep.com/patent/US11906742B2

## CPC

- G02B 27/0172: {characterised by optical features}
- G02B 27/0179: {Display position adjusting means not related to the information to be displayed}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06T 3/20: Linear translation of whole images or parts thereof, e.g. panning
- G06T 11/60: Creating or editing images; Combining images with text
- G06V 10/462: {Salient features, e.g. scale invariant feature transforms [SIFT]}
- G06V 20/20: in augmented reality scenes
- G06V 40/165: {using facial parts and geometric relationships}
- G06V 40/166: {using acquisition arrangements}
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- G06V 40/19: Sensors therefor
- G06V 40/193: {Preprocessing; Feature extraction}
- G06V 40/197: {Matching; Classification}
- G02B 2027/0127: {comprising devices increasing the depth of field}
- G02B 2027/0134: {of stereoscopic type}
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0178: {Eyeglass type (eyeglass details G02C)}
- G02B 2027/0181: {Adaptation to the pilot/driver}
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A wearable device can include an inward-facing imaging system configured to acquire images of a user's periocular region. The wearable device can determine a relative position between the wearable device and the user's face based on the images acquired by the inward-facing imaging system. The relative position may be used to determine whether the user is wearing the wearable device, whether the wearable device fits the user, or whether an adjustment to a rendering location of virtual object should be made to compensate for a deviation of the wearable device from its normal resting position.

## Claims

### Claim 1

A head-mounted device (HMD) comprising:
a hardware processor programmed to:
receive a set of sequentially captured images comprising a first image and a second image;
determine whether at least one periocular feature of a user of the HMD is in at least one of the set of sequentially captured images; and
in response to a determination that at least one periocular feature of the user is in the first image and not in the second image:
determine, based on the at least one periocular feature of the user, a likelihood that the HMD has been removed from the head of the user; and
in response to determining that the likelihood that the HMD has been removed from the head of the user exceeds a threshold likelihood, provide an indication that the HMD has been removed from the head of the user; or
in response to determining that the likelihood that the HMD has been removed from the head of the user does not exceed the threshold likelihood, provide an indication that the HMD is not properly positioned on the head of the user.

### Claim 2 (depends on 1)

The HMD of claim 1, wherein the hardware processor is programmed to:
access a measurement of a relative position between the HMD and a head of the user; and
calculate an adjustment to a normal rendering position of a virtual object based at least partly on the relative position between the HMD and the head of the user.

### Claim 3 (depends on 2)

The HMD of claim 2, wherein the hardware processor is programmed to:
determine a rendering location of the virtual object based on the adjustment to the normal rendering position; and
render the virtual object at the rendering location.

### Claim 4 (depends on 2)

The HMD of claim 2, wherein the measurement of the relative position between the HMD and the head of the user is calculated by performing at least one of:
tracking the at least one periocular feature using visual keypoints, or matching a region of the face with a dense map encoding at least a portion of the head of the user.

### Claim 5 (depends on 4)

The HMD of claim 4, wherein the visual keypoints are computed using at least one of: scale-invariant feature transform, speeded up robust features, oriented FAST and rotated BRIEF, binary robust invariant scalable keypoints, or fast retina keypoint; or wherein the dense map is calculated using iterative closest point algorithm.

### Claim 6 (depends on 2)

The HMD of claim 2, wherein the hardware processor is further programmed to determine a normal resting position of the user's eye associated with a rendering viewpoint of the HMD, and wherein the normal rendering position of the virtual object corresponds to the normal resting position of the user's eye.

### Claim 7 (depends on 6)

The HMD of claim 6, wherein to calculate the adjustment to the normal rendering position of the virtual object, the hardware processor is programmed to:
determine a shift with respect to the normal resting position of the user's eye based at least partly on the relative position between the HMD and the head of the user; and
shift a coordinate associated with the rendering viewpoint of the HMD based at least partly on the determined shift with respect to the normal resting position.

### Claim 8 (depends on 6)

The HMD of claim 6, wherein the relative position between the HMD and the head of the user comprises one or more of: a horizontal shift, a vertical shift, a depth shift, a tilt to a side, or a forward tilt with respect to the normal resting position.

### Claim 9 (depends on 2)

The HMD of claim 2, wherein the relative position between the HMD and the head of the user comprises a first relative position between the HMD and the head of the user for a first eye of the user and a second relative position between the HMD and the head of the user for a second eye of the user.

### Claim 10 (depends on 1)

The HMD of claim 1, wherein the hardware processor is programmed to:
in response to a determination that at least one periocular feature of the user is in the second image and not in the first image, provide an indication that the HMD is properly positioned on the head of the user.