# US 2023/0410699 A1 · Structured display shutdown for video pass-through electronic devices

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410699 A1
- Document ID: US20230410699A1
- Status: Published application
- Kind: A1
- Application number: 18/028,205
- Filed: 2021-09-24
- Published: 2023-12-21
- Earliest claimed: 2020-09-25
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Brian C. CLARK
- Web page: https://patentgrep.com/patent/US20230410699A1

## CPC

- G06F 3/147: using display panels
- G09G 3/001: {using specific devices not provided for in groups G09G3/02 - G09G3/36, e.g. using an intermediate record carrier such as a film slide; Projection systems; Display of non-alphanumerical information, solely or in combination with alphanumerical information, e.g. digital display on projected diapositive as background (slide projectors per se G03B23/00 = 42 HP)}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/14: Digital output to display device {; Cooperation and interconnection of the display device with other functional units}
- G06T 11/40: Filling planar surfaces by adding surface attributes, e.g. adding colours or textures
- G09G 2330/021: Power management, e.g. power saving
- G09G 2354/00: Aspects of interface with display user
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

The subject technology provides structured-shutdown-mode versions of a video view of a physical setting of an electronic device. An electronic device may provide, at a first power consumption rate, a video view of a portion of the physical setting, the portion being physically blocked from being directly viewed by a user of the device, by the electronic device itself. Structured-shutdown-mode versions of a video view may be provided, at a second power consumption rate lower than the first power consumption rate, to a display of an electronic device when the device determines that it is in a low power or failure state. In this way, the electronic device can reduce the functionality and power consumption of the display in a structured manner, to continue to provide a version of the video view to the user in the low power or failure mode state.

## Claims

### Claim 1

A device, comprising:
a display; and
one or more processors configured to:
obtain a video feed of at least a portion of a physical setting;
process the video feed to generate a video view of the portion of the physical setting at a first power consumption rate;
provide the video view corresponding to the processed video feed to the display for display;
identify a display modification state for the device; and
responsive to identifying the display modification state, modify the processing of the video feed to provide, at a second power consumption rate lower than the first power consumption rate, a structured-shutdown-mode version of the video view of the portion of the physical setting to the display while the device is in the display modification state.

### Claim 2 (depends on 1)

The device of claim 1, wherein the one or more processors are configured to process the video feed to generate the video view by warping the video feed to conform the video feed from a view of a camera to a view of a user, wherein the display modification state comprises a low power state or a failure state, and wherein the structured-shutdown-mode version of the video view comprises at least an un-warped version of the video view.

### Claim 3 (depends on 2)

The device of claim 2, further comprising at least one eye sensor, wherein the one or more processors are further configured to:
obtain current eye data for the user from the at least one eye sensor; and
warp the video feed to the view of the user based on the current eye data.

### Claim 4 (depends on 3)

The device of claim 3, wherein the one or more processors are configured to modify the processing of the video feed to generate the structured-shutdown-mode version of the video view by:
discontinuing the warping of the video feed based on the current eye data;
obtaining nominal eye data for the user; and
performing a transformation of the video feed to a predefined surface based on the nominal eye data.

### Claim 5 (depends on 4)

The device of claim 4, wherein the current eye data comprises a gaze direction for each eye of the user, and wherein the nominal eye data comprises an interpupillary distance and a neutral gaze.

### Claim 6 (depends on 4)

The device of claim 4, wherein the one or more processors are configured to warp the video feed based on the current eye data by performing a per-pixel warping based on the current eye data and a depth map, and wherein the transformation of the video feed comprises a pixel-independent transformation.

### Claim 7 (depends on 3)

The device of claim 3, wherein the one or more processors are configured to process the video feed to generate the video view by combining virtual content with the warped video feed for display, and to modify the processing of the video feed to generate the structured-shutdown-mode version of the video view by continuing to perform the warping of the video feed based on the current eye data without combining the virtual content with the warped video feed.

### Claim 8 (depends on 1)

The device of claim 1, further comprising a color camera configured to generate the video feed and at least one monochrome camera, and wherein the one or more processors are configured to process the video feed to generate the video view by:
obtaining a monochrome video feed from the at least one monochrome camera;
performing a color estimation for a portion of the monochrome video feed based on the video feed from the color camera; and
filling a portion of the video feed using the monochrome video feed and the color estimation.

### Claim 9 (depends on 8)

The device of claim 8, wherein the one or more processors are configured to modify the processing of the video feed to generate the structured-shutdown-mode version of the video view by discontinuing at least one of the color estimation or the filling.

### Claim 10 (depends on 1)

The device of claim 1, wherein the display is configured to block a direct view of at least a portion of the physical setting from a user when the device is worn by the user, and wherein the one or more processors are further configured to:
detect an imminent failure of the structured-shutdown-mode version of the video view; and
provide an alert to the user to move the display to unblock the direct view due to the imminent failure.

### Claim 11 (depends on 1)

The device of claim 1, wherein the one or more processors is configured to identify the display modification state at least in part by identifying a low power state for the device based at least in part based on a position of a user as detected by the device.

### Claim 12

A method, comprising:
obtaining a video feed of at least a portion of a physical setting, with an electronic device;
processing the video feed to generate a video view of the portion of the physical setting at a first power consumption rate;
providing the video view corresponding to the processed video feed for display;
identifying a display modification state for the electronic device; and
responsive to identifying the display modification state, modifying the processing of the video feed to provide, at a second power consumption rate lower than the first power consumption rate, a structured-shutdown-mode version of the video view of the portion of the physical setting for display while the electronic device is in the display modification state.

### Claim 13 (depends on 12)

The method of claim 12, wherein processing the video feed to generate the video view includes:
obtaining current eye data for a user from at least one eye sensor; and
warping the video feed to conform the video feed from a view of a camera to a view of the user based on the current eye data.

### Claim 14 (depends on 13)

The method of claim 13, wherein modifying the processing of the video feed to generate the structured-shutdown-mode version of the video view includes:
discontinuing the warping of the video feed based on the current eye data;
obtaining nominal eye data for the user; and
performing a transformation of the video feed to a predefined surface based on the nominal eye data.

### Claim 15 (depends on 14)

The method of claim 14, wherein warping the video feed based on the current eye data comprises performing a per-pixel warping based on the current eye data and a depth map, and wherein the transformation of the video feed comprises a pixel-independent transformation.

### Claim 16 (depends on 12)

The method of claim 12, wherein obtaining the video feed comprises obtaining the video feed from a color camera of the electronic device, and wherein processing the video feed to generate the video view includes:
obtaining a monochrome video feed from at least one monochrome camera of the electronic device;
performing a color estimation for a portion of the monochrome video feed based on the video feed from the color camera; and
filling a portion of the video feed using the monochrome video feed and the color estimation.

### Claim 17 (depends on 16)

The method of claim 16, wherein modifying the processing of the video feed to generate the structured-shutdown-mode version of the video view includes discontinuing at least one of the color estimation or the filling.

### Claim 18

A non-transitory computer-readable medium storing instructions that, when executed by one or more processors of a device, cause the one or more processors to:
obtain a video feed of at least a portion of a physical setting, with an electronic device;
process the video feed to generate a video view of the portion of the physical setting at a first power consumption rate;
provide the video view corresponding to the processed video feed for display;
identify a display modification state for the electronic device; and
responsive to identifying the display modification state, modify the processing of the video feed to provide, at a second power consumption rate lower than the first power consumption rate, a structured-shutdown-mode version of the video view of the portion of the physical setting for display while the electronic device is in the display modification state.

### Claim 19 (depends on 18)

The non-transitory computer-readable medium of claim 18, wherein the instructions, when executed by the one or more processors of the device, cause the one or more processors to modify the processing of the video feed by providing the video feed as captured by a camera of the electronic device to the display unmodified.

### Claim 20 (depends on 18)

The non-transitory computer-readable medium of claim 18, wherein the instructions, when executed by the one or more processors of a device, cause the one or more processors to modify the processing of the video feed by discontinuing a warping operation, a filling operation, or a color estimation operation of the processing.
