# US 2023/0410458 A1 · Occlusion Classification and Feedback

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410458 A1
- Document ID: US20230410458A1
- Status: Published application
- Kind: A1
- Application number: 18/211,518
- Filed: 2023-06-19
- Published: 2023-12-21
- Earliest claimed: 2022-06-21
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Brett D. Miller; Daniel K. Boothe; Martin E. Johnson
- Web page: https://patentgrep.com/patent/US20230410458A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G02B 27/0172: {characterised by optical features}
- G06F 1/163: {Wearable computers, e.g. on a belt}
- G06F 1/1686: {the I/O peripheral being an integrated camera}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 10/94: Hardware or software architectures specially adapted for image or video understanding
- H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

In one implementation, a method of detecting an occlusion of an image sensor is performed at a device including one or more image sensors, one or more speakers, one or more processors, and non-transitory memory. The method includes receiving, from the one or more image sensors, an image of a physical environment. The method includes detecting, based on the image of the physical environment, an occlusion at least partially occluding the one or more image sensors. The method includes classifying the occlusion as one of a plurality of occlusion types. The method includes playing, via the one or more speakers, an occlusion notification based on the classification.

## Claims

### Claim 1

A method comprising:
at a device including one or more image sensors, one or more speakers, one or more processors, and non-transitory memory:
receiving, from the one or more image sensors, an image of a physical environment;
detecting, based on the image of the physical environment, an occlusion at least partially occluding the one or more image sensors;
classifying the occlusion as one of a plurality of occlusion types; and
playing, via the one or more speakers, an occlusion notification based on the classification.

### Claim 2 (depends on 1)

The method of claim 1, wherein the plurality of occlusion types includes a plurality of object type occlusion types.

### Claim 3 (depends on 2)

The method of claim 2, wherein the plurality of object type occlusion types includes a hair occlusion type.

### Claim 4 (depends on 2)

The method of claim 2, wherein the plurality of object type occlusion types includes at least one of a dirt occlusion type or a water occlusion type.

### Claim 5 (depends on 1)

The method of claim 1, wherein the plurality of occlusion types includes an opaque occlusion type and a transparent occlusion type.

### Claim 6 (depends on 1)

The method of claim 1, wherein the plurality of occlusion types includes a first occlusion type associated with level of occlusion below a threshold and a second occlusion type associated with level of occlusion above the threshold.

### Claim 7 (depends on 6)

The method of claim 6, wherein playing the occlusion notification based on the classification includes playing the occlusion notification in response to classifying the occlusion as the second occlusion type.

### Claim 8 (depends on 1)

The method of claim 1, wherein the one or more image sensors includes a left image sensor and a right image sensor and wherein the plurality of occlusion types includes a left image sensor occlusion type associated with occlusion of the left image sensor and a right image sensor occlusion type associated with occlusion of the right image sensor.

### Claim 9 (depends on 8)

The method of claim 8, wherein the one or more speakers includes a left speaker and a right speaker and wherein playing the occlusion notification based on the classification includes playing the occlusion notification in one of the left speaker or the right speaker.

### Claim 10 (depends on 1)

The method of claim 1, wherein the occlusion notification includes one or more tones.

### Claim 11 (depends on 1)

The method of claim 1, wherein the occlusion notification is a verbal notification.

### Claim 12 (depends on 11)

The method of claim 11, wherein the verbal notification includes instructions for clearing the occlusion.

### Claim 13 (depends on 1)

The method of claim 1, wherein classifying the occlusion includes applying a neural network to the image of the physical environment.

### Claim 14 (depends on 1)

The method of claim 1, further comprising, after playing the occlusion notification:
receiving, from the one or more image sensors, a second image of the physical environment;
detecting, based on the second image of the physical environment, clearing of the occlusion; and
playing, via the one or more speakers in response to detecting clearing of the occlusion, a clearing notification.

### Claim 15 (depends on 1)

The method of claim 1, wherein the image of the physical environment has a device field-of-view greater than a user field-of-view and wherein the occlusion is detected outside an area of the user field-of-view.

### Claim 16

A device comprising:
one or more image sensors;
one or more speakers;
a non-transitory memory; and
one or more processors to:
receive, from the one or more image sensors, an image of a physical environment;
detect, based on the image of the physical environment, an occlusion at least partially occluding the one or more image sensors;
classify the occlusion as one of a plurality of occlusion types; and
play, via the one or more speakers, an occlusion notification based on the classification.

### Claim 17 (depends on 16)

The device of claim 16, wherein the one or more image sensors includes a left image sensor and a right image sensor and wherein the plurality of occlusion types includes a left image sensor occlusion type associated with occlusion of the left image sensor and a right image sensor occlusion type associated with occlusion of the right image sensor, wherein the one or more speakers includes a left speaker and a right speaker and wherein the one or more processors are playing the occlusion notification in one of the left speaker or the right speaker.

### Claim 18 (depends on 16)

The device of claim 16, wherein the one or more processors are further to, after playing the occlusion notification:
receive, from the one or more image sensors, a second image of the physical environment;
detect, based on the second image of the physical environment, clearing of the occlusion; and
play, via the one or more speakers in response to detecting clearing of the occlusion, a clearing notification.

### Claim 19 (depends on 16)

The device of claim 16, wherein the image of the physical environment has a device field-of-view greater than a user field-of-view and wherein the occlusion is detected outside an area of the user field-of-view.

### Claim 20

A non-transitory memory storing one or more programs, which, when executed by one or more processors of a device including one or more image sensors and one or more speakers cause the device to:
receive, from the one or more image sensors, an image of a physical environment;
detect, based on the image of the physical environment, an occlusion at least partially occluding the one or more image sensors;
classify the occlusion as one of a plurality of occlusion types; and
play, via the one or more speakers, an occlusion notification based on the classification.