# US 12,015,758 B1 · Holographic video sessions > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,015,758 B1 - Document ID: US12015758B1 - Status: Granted - Kind: B1 - Application number: 17/539,524 - Filed: 2021-12-01 - Granted: 2024-06-18 - Earliest claimed: 2020-12-01 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Gary L. Vondran, Jr.; Dan C. Lelescu; Robert K. Molholm; Nicolas V. Scapel; Kathrin Berkner-Cieslicki; Seung Wook Kim - Web page: https://patentgrep.com/patent/US12015758B1 ## CPC - H04N 13/279: the virtual viewpoint locations being selected by the viewers or determined by tracking - H04N 13/106: Processing image signals (for multi-view video sequence encoding H04N19/597) - H04N 13/25: using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor - H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems - H04N 13/31: using parallax barriers - H04N 13/344: with head-mounted left-right displays - H04N 13/366: using viewer tracking - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Various implementations disclosed herein include devices, systems, and methods that provide images for displaying a 3D appearance of a tracked object on a view dependent device. In some implementations, a set of multiple images of the object from different capture positions is provided and then adjusted such as to render the object in focus when using the adjusted images for display on the view dependent device. In some implementations, the set of multiple images is provided that includes virtual images generated using virtual image sensors that may improve the appearance of out of focus regions (e.g., outside the 3D object appearance) on the view dependent device. In some implementations, the set of multiple images of a person provides a hologram of the person (e.g., left and right eye views provide the appearance of a 3D person) where the images are adjusted to improve gaze direction of eyes of the person. ## Claims ### Claim 1 A method comprising: at a processor: obtaining, from one or more image sensors, a sequence of images of an object; tracking a user feature of interest of the object in the sequence of images; after said tracking and during rendering on a view dependent device, adjusting the sequence of images such that the user feature of interest of the object is aligned on a virtual focal plane in the adjusted images during said rendering, wherein alignment of the user feature of interest results in the user feature of interest being rendered in focus on the view dependent device; and providing the adjusted images for displaying a three-dimensional (3D) appearance of the object on the view dependent device, wherein viewing position determines which of the adjusted images is viewed in a left eye view and which of the adjusted images is viewed in a right eye view. ### Claim 2 (depends on 1) The method of claim 1, wherein said adjusting the sequence of images comprises tracking the virtual focal plane for the feature of interest of the object in the sequence of images. ### Claim 3 (depends on 1) The method of claim 1, wherein said adjusting the sequence of images comprises tracking multiple virtual focal planes for multiple tracked objects in the sequence of images and adjusting the sequence of images based on a corresponding virtual focal plane for a selected one of the multiple tracked objects. ### Claim 4 (depends on 1) The method of claim 1, wherein said obtaining the sequence of images of the object comprises concurrently obtaining the sequence of images from the one or more image sensors, where a location of the virtual focal plane of the object is computed by estimating a distance from the object to the one or more image sensors. ### Claim 5 (depends on 1) The method of claim 1, wherein the one or more image sensors comprise a plurality of virtual cameras, and wherein said obtaining the sequence of images of the object comprises concurrently obtaining the sequence of images from the plurality of virtual cameras. ### Claim 6 (depends on 5) The method of claim 5, wherein said adjusting the sequence of images comprises tracking the virtual focal plane for the object in the sequence of. ### Claim 7 (depends on 5) The method of claim 5, wherein each of the plurality of virtual cameras are spatially posed based on poses of the plurality of virtual cameras, and wherein the plurality of virtual cameras comprises synthetic apertures based on the plurality of virtual cameras or machine learning image generation based on the plurality of virtual cameras. ### Claim 8 (depends on 5) The method of claim 5, wherein a first depth of field for the plurality of virtual cameras is different than a second depth of field for the plurality of virtual cameras and each include a first focal plane corresponding to the tracked feature of interest of the object. ### Claim 9 (depends on 1) The method of claim 1, wherein the view dependent device comprises a 1D lenticular display that allows parallax in a first direction, wherein the 1D lenticular display concurrently displays each of the adjusted images such that each eye of the viewer views a different one of the adjusted images depending upon the viewer's position along the first direction. ### Claim 10 (depends on 1) The method of claim 1, wherein the view dependent device comprises a head-mounted display (HMD) that displays pixels of multiple adjusted images in a right eye display and displays pixels of two or more adjusted images in a left eye display of the HMD, respectively, based on a viewer pose. ### Claim 11 (depends on 1) The method of claim 1, wherein the view dependent device comprises a HMD, an autostereoscopic display, or a light field display, and wherein the adjusted images are encoded and then transmitted to the view dependent display using a prescribed video format. ### Claim 12 (depends on 1) The method of claim 1, wherein the view dependent device is a light field display, further comprising displaying the adjusted images in the light field display causes a single focal plane to be in focus on the light field display. ### Claim 13 (depends on 12) The method of claim 12, wherein the light field display comprises a limited depth of field that includes the single focal plane, further comprising: capturing spatialized audio corresponding to each of the obtained images; and providing adjusted spatialized audio corresponding to the viewing position based on the captured spatialized audio. ### Claim 14 (depends on 1) The method of claim 1, wherein the one or more image sensors comprises a single moving camera. ### Claim 15 (depends on 1) The method of claim 1, further comprising: capturing spatialized audio corresponding to each of the obtained images; and providing adjusted spatialized audio corresponding to the viewing position based on the captured spatialized audio. ### Claim 16 A system comprising: a non-transitory computer-readable storage medium; and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the system to perform operations comprising: obtaining, from one or more image sensors, a sequence of images of an object; tracking a user feature of interest of the object in the sequence of images; after said tracking and during rendering on a view dependent device, adjusting the sequence of images such that the user feature of interest of the object is aligned on a virtual focal plane in the adjusted images during said rendering, wherein alignment of the user feature of interest results in the user feature of interest being rendered in focus on the view dependent device; and providing the adjusted images for displaying a three-dimensional (3D) appearance of the object on the view dependent device, wherein viewing position determines which of the adjusted images is viewed in a left eye view and which of the adjusted images is viewed in a right eye view. ### Claim 17 (depends on 16) The system of claim 16, wherein said adjusting the sequence of images comprises tracking the virtual focal plane for the feature of interest of the object in the sequence of images. ### Claim 18 (depends on 16) The system of claim 16, wherein said adjusting the sequence of images comprises tracking multiple virtual focal planes for multiple tracked objects in the sequence of images and adjusting the sequence of images based on a corresponding virtual focal plane for a selected one of the multiple tracked objects. ### Claim 19 A non-transitory computer-readable storage medium, storing program instructions computer-executable on a computer to perform operations comprising: obtaining, from one or more image sensors, a sequence of images of an object; tracking a user feature of interest of the object in the sequence of images; after said tracking and during rendering on a view dependent device, adjusting the sequence of images such that the user feature of interest of the object is aligned on a virtual focal plane in the adjusted images during said rendering, wherein alignment of the user feature of interest results in the user feature of interest being rendered in focus on the view dependent device; and providing the adjusted images for displaying a three-dimensional (3D) appearance of the object on the view dependent device, wherein viewing position determines which of the adjusted images is viewed in a left eye view and which of the adjusted images is viewed in a right eye view. ### Claim 20 (depends on 1) The method of claim 1, wherein the virtual focal plane is determined by computing a depth from the sequence of images to the user feature of interest of the object.