# US 2021/0344889 A1 · Dynamic Focus 3D Display > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0344889 A1 - Document ID: US20210344889A1 - Status: Published application - Kind: A1 - Application number: 17/378,516 - Filed: 2021-07-16 - Published: 2021-11-04 - Earliest claimed: 2016-01-13 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Alexander Shpunt - Web page: https://patentgrep.com/patent/US20210344889A1 ## CPC - H04N 13/144: for flicker reduction - G02B 3/0056: {arranged along two different directions in a plane, e.g. honeycomb arrangement of lenses (G02B3/0043 takes precedence; miniaturised objectives for electronic devices employing wafer level optics G02B13/0085)} - G02B 26/0833: {the reflecting element being a micromechanical device, e.g. a MEMS mirror, DMD (G02B26/0825 takes precedence; micromechanical devices in general B81B)} - G02B 26/101: {with both horizontal and vertical deflecting means, e.g. raster or XY scanners (colour television using laser beams scanning a display screen H04N9/3129)} - G02B 26/105: {with one or more pivoting mirrors or galvano-mirrors (G02B26/101 takes precedence)} - G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking} - G02B 27/0172: {characterised by optical features} - G02B 27/30: Collimators - G02B 30/35: using reflective optical elements in the optical path between the images and the observer - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - H04N 7/181: {for receiving images from a plurality of remote sources} - H04N 9/3138: {using arrays of modulated light sources (electroluminescent display devices G09G3/30)} - H04N 13/344: with head-mounted left-right displays - H04N 13/363: using image projection screens (volumetric display H04N13/388) - H04N 13/365: using digital micromirror devices [DMD] - G02B 2027/0134: {of stereoscopic type} - G02B 2027/0147: {comprising a device modifying the resolution of the displayed image} - G02B 2027/0185: {Displaying image at variable distance} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A direct retinal projector system that provides dynamic focusing for virtual reality (VR) and/or augmented reality (AR) is described. A direct retinal projector system scans images, pixel by pixel, directly onto the subject's retinas. This allows individual pixels to be optically affected dynamically as the images are scanned to the subject's retinas. Dynamic focusing components and techniques are described that may be used in a direct retinal projector system to dynamically and correctly focus each pixel in VR images as the images are being scanned to a subject's eyes. This allows objects, surfaces, etc. that are intended to appear at different distances in a scene to be projected to the subject's eyes at the correct depths. ## Claims ### Claim 1 A system, comprising: a controller; a projector configured to scan an image of a scene pixel by pixel to a subject's eye, wherein the projector comprises: a plurality of light emitting elements configured to emit beams of light for each pixel under control of the controller; a collimating lens configured to collimate the beams of light emitted by the light emitting elements to generate a combined beam of light for each pixel; and a scanning mirror configured to scan the combined beams of light for the pixels to generate a scanned light field of the image at the subject's eye; wherein the plurality of light emitting elements includes a plurality of focus groups each including one or more of the plurality of light emitting elements, wherein each focus group is configured to focus emitted beams of light at a particular one of a plurality of focus distances; wherein the controller is configured to, for each pixel in the image to be scanned: determine a depth at the pixel in the image; and selectively activate one of the focus groups of light emitting elements that is configured to focus emitted beams of light at a focus distance that corresponds to the determined depth at the pixel in the image. wherein said selectively activating focus groups of light emitting elements that focus emitted beams of light at focus distances that correspond to the determined depths causes content in the scene that is intended to appear at different depths in the scene to be projected to the subject's eye at correct depths. ### Claim 2 (depends on 1) The system as recited in claim 1, wherein the plurality of light emitting elements comprises an array of edge-emitting lasers, wherein the array is tilted with respect to an optical axis of the system such that different ones of the focus groups of the lasers emit light beams at different distances from the collimating lens, wherein the different distances from the collimating lens for the focus groups provided by the tilted array cause the beams of light emitted by the different focus groups to be focused at the different focus distances. ### Claim 3 (depends on 1) The system as recited in claim 1, further comprising an array of microlenses located between the plurality of light emitting elements and the collimating lens, wherein each microlens corresponds to one of the light emitting elements, and wherein the microlenses that correspond to the light emitting elements in each of the focus groups are configured to focus the beams of light emitted by the light emitting elements in the respective focus group at the focus distance of the respective focus group. ### Claim 4 (depends on 1) The system as recited in claim 1, wherein the plurality of light emitting elements comprises an array of vertical cavity surface emitting lasers (VCSELs). ### Claim 5 (depends on 1) The system as recited in claim 1, wherein each focus group includes at least one red light emitting element, at least one blue light emitting element, and at least one green light emitting element. ### Claim 6 (depends on 1) The system as recited in claim 1, wherein, to determine a depth at the pixel in the image, the controller is configured to obtain depth information for the pixel in the scene from a depth map for the image, wherein the depth map indicates a respective depth for each pixel in the image. ### Claim 7 (depends on 1) The system as recited in claim 1, wherein the system comprises two projectors configured to scan a frame of the scene pixel by pixel to the subject's eyes to provide a three-dimensional (3D) view of the scene, wherein the frame comprises a left image scanned by a left projector to the subject's left eye and a right image scanned by a right projector to the subject's right eye, and wherein objects in the scene are shifted in the two images to generate the 3D effect. ### Claim 8 (depends on 7) The system as recited in claim 7, wherein the objects are shifted in the two images as a function of triangulation of distance, where nearer objects are shifted more than more distant objects, and wherein the controller is configured to: generate a depth map for each of the two images according to shift data obtained from the two images, wherein each depth map indicates a respective depth for each pixel in the respective image; and determine the depths at the pixels in the two images according to the depth maps for the two images. ### Claim 9 An apparatus, comprising: a controller; and two projectors configured to scan a frame of a scene pixel by pixel to a subject's eyes under control of the controller to provide a three-dimensional (3D) view of the scene, wherein the frame comprises a left image scanned by a left projector to the subject's left eye and a right image scanned by a right projector to the subject's right eye; wherein each projector comprises an array of light emitting elements, wherein the array of light emitting elements includes a plurality of focus groups each including two or more of the light emitting elements in the array and each configured to focus emitted beams of light at a particular one of a plurality of focus distances; wherein the controller is configured to, for each pixel in the frame to be scanned: determine depth at the pixel in the scene; and selectively activate a focus group in each of the two projectors that is configured to focus emitted beams of light at a focus distance that corresponds to the determined depth; wherein said selectively activating focus groups that focus emitted beams of light at focus distances that correspond to the determined depths causes objects in the scene that are intended to appear at different depths to be projected to the subject's eyes at correct depths. ### Claim 10 (depends on 9) The apparatus as recited in claim 9, wherein each projector further comprises: a collimating lens configured to collimate the beams of light emitted by the array of light emitting elements to generate a combined beam of light for each pixel; and a scanning mirror configured to scan the combined beams of light for the pixels to generate a scanned light field of the image at the subject's respective eye. ### Claim 11 (depends on 9) The apparatus as recited in claim 9, wherein each projector further comprises an array of microlenses located in front of the array of light emitting elements, wherein each microlens corresponds to one of the light emitting elements, and wherein the microlenses that correspond to the light emitting elements in each of the focus groups are configured to focus the beams of light emitted by the light emitting elements in the focus group at the focus distance of the focus group. ### Claim 12 (depends on 11) The apparatus as recited in claim 11, wherein the light emitting elements include vertical cavity surface emitting lasers (VCSELs). ### Claim 13 (depends on 9) The apparatus as recited in claim 9, wherein each focus group includes at least one red light emitting element, at least one blue light emitting element, and at least one green light emitting element. ### Claim 14 (depends on 9) The apparatus as recited in claim 9, wherein each projector further comprises a collimating lens configured to collimate the beams of light emitted by the array of light emitting elements to generate a combined beam of light for each pixel, wherein the array of light emitting elements is tilted with respect to an optical axis of the apparatus such that different ones of the focus groups of the light emitting elements emit light beams at different distances from the collimating lens, wherein the different distances from the collimating lens for the focus groups provided by the tilted array cause the beams of light emitted by the different focus groups to be focused at the different focus distances. ### Claim 15 (depends on 14) The apparatus as recited in claim 14, wherein the light emitting elements include edge emitting lasers. ### Claim 16 (depends on 9) The apparatus as recited in claim 9, wherein objects in the scene are shifted in the two images as a function of triangulation of distance to generate a 3D effect, where nearer objects are shifted more than more distant objects, and wherein the controller is configured to: generate depth maps for the two images according to shift data obtained from the two images, wherein each depth map indicates a respective depth for each pixel in the respective image; and determine the depths at the pixels in the two images according to the depth maps for the two images. ### Claim 17 A method, comprising: scanning, by a projector comprising an array of light emitting elements under control of a controller, an image of a scene pixel by pixel to a subject's eye, wherein said scanning comprises, for each pixel in the image to be scanned: determining, by the controller, a depth at the pixel in the image; selectively activating, by the controller, a focus group including two or more of the light emitting elements in the array that are configured to focus emitted beams of light at a particular one of a plurality of focus distances that corresponds to the determined depth at the pixel in the image; and emitting, by the focus group in response to said activating, two or more beams of light, wherein the two or more beams of light are focused at the focus distance of the focus group; wherein said selectively activating focus groups of light emitting elements that focus emitted beams of light at focus distances that correspond to the determined depths causes content in the scene that is intended to appear at different depths to be projected to the subject's eye at correct depths. ### Claim 18 (depends on 17) The method as recited in claim 17, further comprising: collimating, by a collimating lens of the projector, the beams of light emitted by the focus groups to generate a combined beam of light for each pixel; and scanning, by a scanning mirror of the projector, the combined beams of light for the pixels to generate a scanned light field of the image at the subject's eye. ### Claim 19 (depends on 17) The method as recited in claim 17, wherein the projector further comprises an array of microlenses located in front of the array of light emitting elements, wherein each microlens corresponds to one of the light emitting elements, and wherein, to focus the two or more beams of light at the focus distance of the focus group, the method further comprises the microlenses that correspond to the light emitting elements in the focus group focusing the beams of light emitted by the light emitting elements in the focus group at the focus distance of the focus group. ### Claim 20 (depends on 17) The method as recited in claim 17, wherein, said determining a depth at the pixel in the image comprises obtaining depth information for the pixel in the scene from a depth map for the image, wherein the depth map indicates a respective depth for each pixel in the image.