Low latency chroma keying embedded in a head-mounted display for mixed reality
AssigneeApple Inc.
InventorsVincent Chapdelaine-Couture, Anthony Ghannoum, Yan Cote, Irving Lustigman, Marc-Andre Chenier, Simon Fortin-Deschenes, Bertrand Nepveu
Life of this patent
Estimating the term…
Abstract
A Head-Mounted Display with camera sensors to perform chroma keying in a mixed reality context is presented. Low latency is achieved by embedding the processing in the HMD itself, specifically, format camera images, detect the selected color range and make a composite with the virtual content.
Claims22
A method comprising:
- at a head-mounted device including an image sensor, a display, and one or more processors:
- capturing, via the image sensor, a live image of a real environment;
- performing chromatic aberration correction on the live image, using the one or more processors, to generate a processed image;
- obtaining, using the one or more processors, content to be rendered in association with the processed image of the real environment;
- detecting, using the one or more processors, a first portion of the processed image of the real environment within a color range;
- obtaining, using the one or more processors, a keying mask based on the first portion of the processed image of the real environment;
- filtering, using the one or more processors, the keying mask to generate a filtered keying mask corresponding to a second portion of the processed image of the real environment;
- modifying, using the one or more processors, the second portion of the processed image of the real environment with the content based on the filtered keying mask, in order to generate a display image;
- and
- displaying, on the display, the display image.
A head-mounted device comprising:
- an image sensor to capture a live image of a real environment;
- one or more processors to:
- perform chromatic aberration correction on the live image to generate a processed image;
- obtain content to be rendered in association with the processed image of the real environment;
- detect a first portion of the processed image of the real environment within a color range;
- obtain a keying mask based on the first portion of the processed image of the real environment;
- filter the keying mask to generate a filtered keying mask corresponding to a second portion of the processed image of the real environment;
- modify the second portion of the processed image of the real environment with the content based on the filtered keying mask, in order to generate a display image;
- and
- a display to display the display image.
A non-transitory computer-readable medium having instructions encoded thereon which, when executed by one or more processors of a head-mounted device including an image sensor and a display, cause the head-mounted device to:
- capture, via the image sensor, a live image of a real environment;
- perform chromatic aberration correction on the live image, using the one or more processors, to generate a processed image;
- obtain, using the one or more processors, content to be rendered in association with the processed image of the real environment;
- detect, using the one or more processors, a first portion of the processed image of the real environment within a color range;
- obtain, using the one or more processors, a keying mask based on the first portion of the processed image of the real environment;
- filter, using the one or more processors, the keying mask to generate a filtered keying mask corresponding to a second portion of the processed image of the real environment;
- modify, using the one or more processors, the second portion of the processed image of the real environment with the content based on the filtered keying mask, in order to generate a display image;
- and
- display, on the display, the display image.
Dependencies are inferred from the claim text.