# US 12,112,449 B2 · Camera-based transparent display > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,449 B2 - Document ID: US12112449B2 - Status: Granted - Kind: B2 - Application number: 18/337,769 - Filed: 2023-06-20 - Published: 2023-10-19 - Granted: 2024-10-08 - Earliest claimed: 2017-09-08 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Brett D. Miller; Minwoong Kim; Ricardo J. Motta - Web page: https://patentgrep.com/patent/US12112449B2 ## CPC - G06T 3/18: Image warping, e.g. rearranging pixels individually - G06T 7/50: Depth or shape recovery - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - H04N 5/2621: {Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability} - H04N 9/73: Colour balance circuits, e.g. white balance circuits or colour temperature control (camera processing pipelines for colour balance H04N23/88) - H04N 23/63: by using electronic viewfinders - H04N 23/88: for colour balance, e.g. white-balance circuits or colour temperature control - H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums - G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light - G06T 7/593: from stereo images - G06T 2207/20164: Salient point detection; Corner detection - G06T 2207/30201: Face - H04N 23/67: Focus control based on electronic image sensor signals - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Presenting an image of a scene may include capturing an image of a scene by a camera of an electronic device, wherein the electronic device comprises the camera and a display, and wherein the camera and the display have a first spatial relationship, determining a second spatial relationship between a viewpoint and the display of the electronic device, warping the image to obtain an image of a first portion of the scene based on the first spatial relationship and the second spatial relationship, and presenting the warped image on the display, wherein, from the viewpoint, the image of the first portion of the scene is substantially contiguous with a second portion of the scene visible outside an edge of the electronic device. ## Claims ### Claim 1 A method for presenting an image of a scene, comprising: capturing an image of a scene by a camera of an electronic device, wherein the electronic device comprises the camera and a display, and wherein the camera and the display have a first spatial relationship; determining a second spatial relationship between a viewpoint and the display of the electronic device; warping the image to obtain an image of a first portion of the scene based on the first spatial relationship and the second spatial relationship so that the first portion of the scene appears at a same scale as a second portion of the scene visible outside an edge of the electronic device; and presenting the warped image on the display. ### Claim 2 (depends on 1) The method of claim 1, wherein warping the image further comprises: back-projecting the image based on intrinsics of the camera and a depth in the scene; and re-projecting the image to the display based on the second spatial relationship. ### Claim 3 (depends on 2) The method of claim 2, further comprising: selecting the depth using at least one selected from a group consisting of an autofocus operation, a stereo estimation, a time of flight operation, and a structured light operation; and backprojecting the image based on the selected depth. ### Claim 4 (depends on 2) The method of claim 2, wherein back-projecting the image comprises: identifying four image corners, based on the intrinsics of the camera and the depth; and intersecting the four image corners with a plane defined by the display to obtain a 2D image transform. ### Claim 5 (depends on 1) The method of claim 1, wherein warping the image further comprises: identifying an object in the scene; and warping the image such that a first portion of the object is visible in the scene from the viewpoint and an image of a second portion of the object is visible on the display from the viewpoint, and wherein the first portion and the second portion are substantially aligned. ### Claim 6 (depends on 1) The method of claim 1, further comprising: detecting a movement of the electronic device; obtaining an updated image of the scene from the electronic device; determining an updated spatial relationship between an updated viewpoint and the display of the electronic device; warping the updated image based, at least in part, on the updated spatial relationship to obtain an updated warped image; and displaying the updated warped image on the display. ### Claim 7 A non-transitory computer readable medium comprising computer readable code for presenting an image of a scene, executable by one or more processors to: capture an image of a scene by a camera of an electronic device, wherein the electronic device comprises the camera and a display, and wherein the camera and the display have a first spatial relationship; determine a second spatial relationship between a viewpoint and the display of the electronic device; warp the image to obtain an image of a first portion of the scene based on the first spatial relationship and the second spatial relationship so that the first portion of the scene appears at a same scale as a second portion of the scene visible outside an edge of the electronic device from the viewpoint; and present the warped image on the display. ### Claim 8 (depends on 7) The non-transitory computer readable medium of claim 7, wherein the computer readable code to warp the image further comprises computer readable code to: back-project the image based on intrinsics of the camera and a depth in the scene; and re-project the image to the display based on the second spatial relationship. ### Claim 9 (depends on 8) The non-transitory computer readable medium of claim 8, further comprising computer readable code to: select the depth using at least one selected from a group consisting of an autofocus operation, a stereo estimation, a time of flight operation, and a structured light operation; and backproject the image based on the selected depth. ### Claim 10 (depends on 8) The non-transitory computer readable medium of claim 8, wherein the computer readable code to back-project the image comprises computer readable code to: identify four image corners, based on the intrinsics of the camera and the depth; and intersect the four image corners with a plane defined by the display to obtain a 2D image transform. ### Claim 11 (depends on 7) The non-transitory computer readable medium of claim 7, wherein the computer readable code to warp the image further comprises computer readable code to perform a color correction on the image. ### Claim 12 (depends on 11) The non-transitory computer readable medium of claim 11, wherein the computer readable code to perform the color correction on the image comprises computer readable code to perform a white balance function on the image. ### Claim 13 (depends on 7) The non-transitory computer readable medium of claim 7, wherein the camera is a back-facing camera, and wherein a front-facing camera captures a second image comprising at least part of a user, and wherein the viewpoint is determined based on the second image. ### Claim 14 (depends on 7) The non-transitory computer readable medium of claim 7, wherein the computer readable code to warp the image further comprises computer readable code to: identify an object in the scene; and warp the image such that a first portion of the object is visible in the scene from the viewpoint and an image of a second portion of the object is visible on the display from the viewpoint, and wherein the first portion and the second portion are substantially aligned. ### Claim 15 (depends on 7) The non-transitory computer readable medium of claim 7, further comprising computer readable code to: detect a movement of the electronic device; obtain an updated image of the scene from the electronic device; determine an updated spatial relationship between an updated viewpoint and the display of the electronic device; warp the updated image based, at least in part, on the updated spatial relationship to obtain an updated warped image; and display the updated warped image on the display. ### Claim 16 A system for presenting an image of a scene, comprising: one or more processors; and one or more memory devices coupled to the one or more processors and comprising computer readable code executable by the one or more processors to: capture an image of a scene by a camera of an electronic device, wherein the electronic device comprises the camera and a display, and wherein the camera and the display have a first spatial relationship; determine a second spatial relationship between a viewpoint and the display of the electronic device; warp the image to obtain an image of a first portion of the scene based on the first spatial relationship and the second spatial relationship so that the first portion of the scene appears at a same scale as a second portion of the scene visible outside an edge of the electronic device from the viewpoint; and present the warped image on the display. ### Claim 17 (depends on 16) The system of claim 16, wherein the computer readable code to warp the image further comprises computer readable code to perform a color correction on the image. ### Claim 18 (depends on 16) The system of claim 16, wherein the camera is a back-facing camera, and wherein a front-facing camera captures a second image comprising at least part of a user, and wherein the viewpoint is determined based on the second image. ### Claim 19 (depends on 16) The system of claim 16, wherein the computer readable code to warp the image further comprises computer readable code to: identify an object in the scene; and warp the image such that a first portion of the object is visible in the scene from the viewpoint and an image of a second portion of the object is visible on the display from the viewpoint, and wherein the first portion and the second portion are substantially aligned. ### Claim 20 (depends on 16) The system of claim 16, further comprising computer readable code to: detect a movement of the electronic device; obtain an updated image of the scene from the electronic device; determine an updated spatial relationship between an updated viewpoint and the display of the electronic device; warp the updated image based, at least in part, on the updated spatial relationship to obtain an updated warped image; and display the updated warped image on the display.