# US 2021/0343033 A1 · Stabilizing images produced by eye-mounted displays, based on images of the external environment captured by eye-mounted imaging devices

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0343033 A1
- Document ID: US20210343033A1
- Status: Published application
- Kind: A1
- Application number: 16/865,079
- Filed: 2020-05-01
- Published: 2021-11-04
- Earliest claimed: 2020-05-01
- Assignees: Tectus Corporation
- Applicants: Tectus Corporation
- Inventors: Abhishek Deepak Bhat; Ramin Mirjalili; Brian Elliot Lemoff; Joseph Czompo; Michael West Wiemer
- Web page: https://patentgrep.com/patent/US20210343033A1

## CPC

- G06T 7/337: {involving reference images or patches}
- G02B 27/0172: {characterised by optical features}
- G02B 27/0179: {Display position adjusting means not related to the information to be displayed}
- G02C 7/04: Contact lenses for the eyes (disinfection or sterilisation of contact lenses A61L12/00)
- G02C 11/10: {Electronic devices other than hearing aids}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06T 3/20: Linear translation of whole images or parts thereof, e.g. panning
- G06T 3/60: Rotation of whole images or parts thereof
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/32: using correlation-based methods
- H04N 9/3173: {wherein the projection device is specially adapted for enhanced portability}
- H04N 9/3194: {including sensor feedback}
- H04N 23/57: Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- G06T 2207/10016: Video; Image sequence
- H04N 23/54: Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02C: SPECTACLES; SUNGLASSES OR GOGGLES INSOFAR AS THEY HAVE THE SAME FEATURES AS SPECTACLES; CONTACT LENSES (trial frames for testing the eyes A61B3/04; goggles or eyeshields not having the same features as spectacles A61F9/00)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An imaging device contained in a contact lens captures images of the external environment, which for convenience will be referred to as real-world images. These real-world images are used to stabilize images produced by a femtoprojector also in the contact lens. For convenience, the images produced by the femtoprojector will be referred to as augmented reality or AR images. The femtoprojector is inward-facing (i.e., facing towards the interior of the eye) and projects the AR images onto the user's retina, creating the appearance of virtual images in the external environment. The imaging device, referred to as a femtoimager for convenience, is outward-facing and captures a sequence of actual real-world images of the external environment. Because the femtoimager and femtoprojector move together, the real-world images captured by the femtoimager reflect the motion of the virtual AR images from the femtoprojector relative to the external environment.

## Claims

### Claim 1

A method for stabilizing AR images produced by an inward-facing femtoprojector mounted on a user's eye, the femtoprojector projecting the AR images onto the user's retina, the method comprising:
an outward-facing femtoimager capturing a sequence of real-world images of an external environment, wherein the femtoimager is also mounted on the user's eye;
comparing the captured real-world images in the sequence over time; and
adjusting an offset of the projected AR images to stabilize a visual alignment between the AR images and the external environment, wherein the adjustment of the offset is based on the comparison of the captured real-world images over time.

### Claim 2 (depends on 1)

The method of claim 1 wherein adjusting the offset of the projected AR images compensates for visual misalignment caused by a microsaccadic motion of the user's eye.

### Claim 3 (depends on 1)

The method of claim 1 wherein adjusting the offset of the projected AR images compensates for visual misalignment caused by involuntary motion of the user's eye while the user's gaze is fixated at a single point.

### Claim 4 (depends on 1)

The method of claim 1 wherein adjusting the offset of the projected AR images compensates for visual misalignment caused by a smooth pursuit motion of the user's eye.

### Claim 5 (depends on 1)

The method of claim 1 wherein adjusting the offset of the projected AR images includes adjusting a rotational offset of the AR images.

### Claim 6 (depends on 1)

The method of claim 1 wherein comparing the captured real-world images over time comprises:
selecting a template defined by a region of pixels in one of the real-world images; and
tracking the template in real-world images in the sequence.

### Claim 7 (depends on 6)

The method of claim 6 wherein selecting the template is based on detail in and contrast of pixels in the region.

### Claim 8 (depends on 6)

The method of claim 6 wherein selecting the template comprises:
selecting a size of the template based on a speed of motion of the gaze.

### Claim 9 (depends on 6)

The method of claim 6 wherein tracking the template in real-world images in the sequence is based on pixel shifting of the template.

### Claim 10 (depends on 6)

The method of claim 6 wherein tracking the template in real-world images in the sequence is based on cross-correlation of the template with the real-world images.

### Claim 11 (depends on 6)

The method of claim 6 wherein the template is not larger than 100×100 pixels.

### Claim 12 (depends on 1)

The method of claim 1 wherein comparing the captured real-world images over time comprises:
extracting a set of features from one of the real-world images; and
tracking the features in real-world images in the sequence.

### Claim 13 (depends on 12)

The method of claim 12 wherein the set of features include at least one of: corners, and features based on image gradients.

### Claim 14 (depends on 12)

The method of claim 12 wherein the set of features includes at least 10 different features.

### Claim 15 (depends on 12)

The method of claim 12 wherein tracking the features comprises:
tracking a position and an orientation of the features in subsequent real-world images in the sequence.

### Claim 16 (depends on 1)

The method of claim 1 wherein tracking the gaze over time is based on an assumption of rigid transformation of views of the external environment.

### Claim 17 (depends on 1)

The method of claim 1 wherein both the femtoprojector and the femtoimager are mounted in a contact lens and the contact lens is mounted at a fixed position on the user's eye, so that both the femtoprojector and the femtoimager move in unison with the user's eye as the eye rotates.

### Claim 18 (depends on 1)

The method of claim 1 wherein adjusting the offset of the projected AR images comprises:
tracking a motion of the user's eye over time, based on the comparison of the captured real-world images over time; and
adjusting the offset of the projected AR images based on the tracked motion.

### Claim 19 (depends on 1)

The method of claim 1 wherein a field of view of the femtoimager is not more than 15 degrees.

### Claim 20 (depends on 1)

The method of claim 1 wherein the femtoimager captures images of not more than 256×256 pixels.
