# US 2024/0004197 A1 · Dynamic sensor selection for visual inertial odometry systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0004197 A1
- Document ID: US20240004197A1
- Status: Published application
- Kind: A1
- Application number: 18/368,288
- Filed: 2023-09-14
- Published: 2024-01-04
- Earliest claimed: 2020-06-29
- Assignees: Snap Inc.
- Inventors: Olha Borys; Georg Halmetschlager-Funek; Matthias Kalkgruber; Daniel Wolf; Jakob Zillner
- Web page: https://patentgrep.com/patent/US20240004197A1

## CPC

- G02B 27/017: {Head mounted}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 1/163: {Wearable computers, e.g. on a belt}
- G06F 1/3206: Monitoring of events, devices or parameters that trigger a change in power modality
- G06F 3/012: {Head tracking input arrangements}
- G06T 7/50: Depth or shape recovery
- G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)}
- G02B 2027/0178: {Eyeglass type (eyeglass details G02C)}
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Visual-inertial tracking of an eyewear device using sensors. The eyewear device monitors the sensors of a visual inertial odometry system (VIOS) that provide input for determining a position of the device within its environment. The eyewear device determines the status of the VIOS based information from the sensors and adjusts the plurality of sensors (e.g., by turning on/off sensors, changing the sampling rate, of a combination thereof) based on the determined status. The eyewear device then determines the position of the eyewear device within the environment using the adjusted plurality of sensors.

## Claims

### Claim 1

A method for visual-inertial tracking with an eyewear device, the method comprising:
monitoring a plurality of sensors of a visual inertial odometry system (VIOS), wherein each of the plurality of sensors provide input for determining a position of the eyewear device within an environment;
determining a status of the VIOS;
selecting a sampling rate for one of the plurality of sensors based on the determined status;
sampling the one of the plurality of sensors at the selected sampling rate; and
determining the position of the eyewear device within the environment using the one of the plurality of sensors at the selected sampling rate.

### Claim 2 (depends on 1)

The method of claim 1, further comprising adjusting the plurality of sensors based on the determined status.

### Claim 3 (depends on 2)

The method of claim 2, wherein the adjusting further comprises:
wherein the adjusting comprises selecting a subset of the plurality of sensors and powering off the remaining sensors, wherein the adjusting comprises:
selecting a subset of the plurality of sensors; and
powering off the remaining sensors:
wherein the determining the position of the eyewear device comprises determining the position of the eyewear device within the environment using the subset of sensors.

### Claim 4 (depends on 2)

The method of claim 2, wherein the adjusting comprises:
selecting a sampling rate for one of the plurality of sensors based on the determined status; and
sampling the one of the plurality of sensors at the selected sampling rate;
wherein the determining the position of the eyewear device comprises determining the position of the eyewear device within the environment using the one of the plurality of sensors at the selected sampling rate.

### Claim 5 (depends on 1)

The method of claim 1, wherein the plurality of sensors include an inertial measurement unit (IMU) and a first camera.

### Claim 6 (depends on 5)

The method of claim 5, wherein the first camera is a first visible light camera, and wherein the plurality of sensors further includes one or more of a second visible light camera, a first depth camera, a second depth camera, another IMU, a radar system, or a GPS.

### Claim 7 (depends on 5)

The method of claim 5, wherein the determining the status of the VIOS includes:
capturing images with the first camera;
identifying a physical environment of the eyewear device;
comparing the identified physical environment to a prior physical environment;
identifying new information in the identified physical environment; and
determining the status of the VIOS based on the new information in the identified physical environment.

### Claim 8 (depends on 2)

The method of claim 2, wherein the adjusting includes:
adjusting at least one sensor of the plurality of sensors in response to new information, wherein adjusting includes altering rate, resolution, quality, power on, or power off.

### Claim 9 (depends on 2)

The method of claim 2, wherein the determining the status of the VIOS includes:
determining at least one of a motion parameter value or an uncertainty parameter value of the eyewear device; and
mapping the determined at least one of the motion parameter value or the uncertainty parameter value to one of a plurality of VIOS status configuration options.

### Claim 10 (depends on 9)

The method of claim 9, wherein the VIOS status configuration options include at least a low power level and a high power level and wherein the adjusting includes:
placing the VIOS in the low power level when the motion parameter value and the uncertainty parameter value are low; and
placing the VIOS in the high power level when the motion parameter values and the uncertainty parameter value are high.

### Claim 11

An eyewear device with visual-inertial tracking, the eyewear device comprising:
a visual inertial odometry system (VIOS) including a plurality of sensors, wherein the plurality of sensors include an inertial measurement unit (IMU) and a first camera, wherein each of the plurality of sensors provide input for determining a position of the eyewear device within an environment;
a processor configured to determine a status of the VIOS, select a sampling rate for one of the plurality of sensors based on the determined status, sample the one of the plurality of sensors at the selected sampling rate, adjust the plurality of sensors based on the determined status, and determine the position of the eyewear device within the environment using the adjusted plurality of sensors; and
a frame supporting the VIOS and the processor, the frame configured to be worn on the head of a user,
wherein the VIOS is configured to capture images with the first camera, and the processor is configured to identify a physical environment of the eyewear device and determine the status of the VIOS based on the identified physical environment.

### Claim 12 (depends on 11)

The device of claim 11, wherein the plurality of sensors include an inertial measurement unit (IMU) and a first camera.

### Claim 13 (depends on 11)

The device of claim 11, wherein the first camera is a first visible light camera, and wherein the plurality of sensors further includes one or more of a second visible light camera, a first depth camera, a second depth camera, another IMU, a radar system, or a GPS.

### Claim 14 (depends on 12)

The device of claim 12, wherein the VIOS is configured to capture images with the first camera, and the processor is configured to identify a physical environment of the eyewear device and determine the status of the VIOS based on the identified physical environment.

### Claim 15 (depends on 11)

The device of claim 11, wherein the processor is configured to:
capture images with the first camera;
identify a physical environment of the eyewear device;
compare the identified physical environment to a prior physical environment;
identify new information in the identified physical environment; and
determine the status of the VIOS based on the new information in the identified physical environment.

### Claim 16 (depends on 11)

The device of claim 11, wherein the determining the status of the VIOS includes:
determining at least one of a motion parameter value or an uncertainty parameter value of the eyewear device; and
mapping the determined at least one of the motion parameter value or the uncertainty parameter value to one of a plurality of VIOS status configuration options.

### Claim 17 (depends on 16)

The device of claim 16, wherein the VIOS status configuration options include at least a low power level and a high power level and wherein the adjusting includes:
placing the VIOS in the low power level when the motion parameter value and the uncertainty parameter value are low; and
placing the VIOS in the high power level when the motion parameter values and the uncertainty parameter value are high.

### Claim 18

A non-transitory computer-readable medium storing program code for visual-inertial tracking when executed by an eyewear device having a plurality of sensors, a processor, and a memory, the program code, when executed, is operative to cause an electronic processor to perform the steps of:
monitoring a plurality of sensors of a visual inertial odometry system (VIOS), wherein each of the plurality of sensors provide input for determining a position of the eyewear device within an environment;
determining a status of the visual-inertial tracking system;
selecting a sampling rate for one of the plurality of sensors based on the determined status; and
sampling the one of the plurality of sensors at the selected sampling rate, wherein the determining the position of the eyewear device comprises determining the position of the eyewear device within the environment using the one of the plurality of sensors sampled at the selected sampling rate.

### Claim 19 (depends on 18)

The non-transitory computer-readable medium of claim 18, further comprising adjusting the plurality of sensors based on the determined status.

### Claim 20 (depends on 19)

The non-transitory computer-readable medium of claim 19, wherein the step of adjusting further comprises:
selecting a sampling rate for one of the plurality of sensors based on the determined status; and
sampling the one of the plurality of sensors at the selected sampling rate, wherein the determining the position of the eyewear device comprises determining the position of the eyewear device within the environment using the one of the plurality of sensors at the selected sampling rate.
