US 12,749,254 B2Granted

Video pipeline

AssigneeApple Inc.

InventorsArthur Y Zhang, Ray L. Chang, Timothy R. Oriol, Ling Su, Gurjeet S. Saund, Guy Cote, Jim C. Chou, Hao Pan, Tobias Eble, Avi Bar-Zeev, Sheng Zhang, Justin A. Hensley, Geoffrey Stahl

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

A mixed reality system that includes a device and a base station that communicate via a wireless connection The device may include sensors that collect information about the user's environment and about the user. The information collected by the sensors may be transmitted to the base station via the wireless connection. The base station renders frames or slices based at least in part on the sensor information received from the device, encodes the frames or slices, and transmits the compressed frames or slices to the device for decoding and display. The base station may provide more computing power than conventional stand-alone systems, and the wireless connection does not tether the device to the base station as in conventional tethered systems. The system may implement methods and apparatus to maintain a target frame rate through the wireless link and to minimize latency in frame rendering, transmittal, and display.

Claims20

Dependent claims

A system, comprising:

a base station comprising one or more processors;
and
a device, comprising:
one or more processors;
one or more cameras configured to capture frames that include views of a user's environment;
and
a display subsystem for displaying a 3D virtual view to the user;
wherein the device is configured to:
transmit frames captured by the one or more cameras to the base station over a wireless connection;
wherein the base station is configured to:
render the frames or frame portions;
compress the rendered frames or frame portions;
and
transmit the compressed frames or frame portions to the device over the wireless connection;
wherein the device is configured to decompress the compressed frames or frame portions received from the base station and provide the rendered frames or frame portions to the display subsystem for display;
and
wherein the device is configured to:
monitor the wireless connection to the base station;
and
in response to detecting that the wireless connection to the base station has been lost:
render, by the device, one or more frames that include views of the user's environment based on the frames captured by the one or more cameras;
and
provide the rendered one or more frames to the display subsystem for display.

20 more elements

A computing device, comprising:

one or more processors to:
receive frames captured by one or more cameras of a head-mounted display device, wherein the frames are received, by the computing device over a wireless connection;
render the frames or frame portions;
compress the rendered frames or frame portions;
transmit the compressed frames or frame portions over the wireless connection to the head-mounted display device;
monitor the wireless connection to the head-mounted display device;
and
in response to detecting that the wireless connection to the head-mounted display has been restored, resume transmitting the compressed frames or frame portions over the wireless connection to the head-mounted display device,
wherein while the wireless connection is lost, the head-mounted display device:
renders one or more frames that include views of a user's environment based on the frames captured by the one or more cameras;
and
provides the rendered one or more frames that have been rendered to a display subsystem of the head-mounted device for display.

9 more elements

Dependencies are inferred from the claim text.

Facts

Application: 18/797,340Filed 7 Aug 2024 · continuation
Priority: 28 Apr 2017
Published: US 2024/0394952 A128 Nov 2024
Granted: 29 Sep 2026
Term ends: Not estimatedThe source records don’t support an estimate.

Classification9 codes

G06T 15/005
Three-dimensional [3D] image rendering: general purpose rendering architectures
G06F 3/012
Arrangements for interaction with the human body, e.g. for user immersion in virtual reality: head tracking input arrangements
G06T 3/18
Geometric image transformations in the plane of the image: image warping, e.g. rearranging pixels individually
G06T 9/00
Image coding

Family

Application62/492,000Us provisional application · filed 28 Apr 2017
PCT/US2018/029862Parent of this continuation · filed 27 Apr 2018
Application16/662,952Parent of this continuation · filed 24 Oct 2019
Application17/352,080Parent of this continuation · filed 18 Jun 2021
US 11,043,018 B2Parent of this continuation · filed 22 Jun 2021
Application18/344,294Parent of this continuation · filed 29 Jun 2023
US 11,727,619 B2Parent of this continuation · filed 15 Aug 2023
US 2024/0394952 A1Pre-grant publication · published 28 Nov 2024
US 12,086,919 B2Parent of this continuation
US 2020/0058152 A1Published application
US 2021/0312694 A1Published application
US 2023/0351672 A1Published application
US 12,749,254 B2This patent

From the US filing history. Foreign family members aren’t listed.