# US 11,113,891 B2 · Systems, methods, and media for displaying real-time visualization of physical environment in artificial reality > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,113,891 B2 - Document ID: US11113891B2 - Status: Granted - Kind: B2 - Application number: 16/773,770 - Filed: 2020-01-27 - Published: 2021-07-29 - Granted: 2021-09-07 - Earliest claimed: 2020-01-27 - Assignees: Facebook Technologies, LLC - Applicants: Facebook Technologies, LLC - Inventors: Gioacchino Noris; Jeng-Weei Lin - Web page: https://patentgrep.com/patent/US11113891B2 ## CPC - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G02B 27/0172: {characterised by optical features} - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - H04N 13/279: the virtual viewpoint locations being selected by the viewers or determined by tracking - H04N 13/344: with head-mounted left-right displays - H04N 13/366: using viewer tracking - G02B 2027/0134: {of stereoscopic type} - G02B 2027/0138: {comprising image capture systems, e.g. camera} - G02B 2027/014: {comprising information/image processing systems} - G06T 2219/2012: Colour editing, changing, or manipulating; Use of colour codes - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Particular embodiments are directed to a passthrough feature. A computing system may display a virtual-reality scene on a device worn by a user. The system may receive a request to display a visual representation of at least a portion of a physical environment surrounding the user. The system may access data associated with the physical environment captured by camera(s) of the device. The system may generate, based the data, depth measurements of one or more objects in the physical environment. The system may generate, based on the depth measurements, one or more models of the one or more objects in the physical environment. The system may render an image based on a viewpoint of the user and the one or more models and, based on the image, generate the visual representation requested by the user. The visual representation may then be displayed with the virtual-reality scene to the user. ## Claims ### Claim 1 A method comprising, by a computing system: displaying a virtual-reality scene on a device worn by a user; receiving a request from the user to display a visual representation of at least a portion of a physical environment surrounding the user; accessing data associated with the physical environment captured by one or more cameras of the device; generating, based on the data, depth measurements of one or more objects in the physical environment; generating, based on the depth measurements, one or more three-dimensional (3D) models of the one or more objects in the physical environment; rendering an output image based on a current viewpoint of an eye of the user and the one or more 3D models of the one or more objects in the physical environment; generating, based on the output image, the visual representation requested by the user; and displaying the visual representation with the virtual-reality scene to the user. ### Claim 2 (depends on 1) The method of claim 1, wherein the visual representation is an outline of the one or more objects in the physical environment. ### Claim 3 (depends on 2) The method of claim 2, wherein the displayed outline of the one or more objects is superimposed over the virtual-reality scene. ### Claim 4 (depends on 2) The method of claim 2, wherein the displayed outline has a uniform color tint. ### Claim 5 (depends on 4) The method of claim 4, wherein the virtual-reality scene is generated by an application and the uniform color tint is selected by the application. ### Claim 6 (depends on 1) The method of claim 1, wherein generating the visual representation comprises applying a gradient filter to the output image. ### Claim 7 (depends on 1) The method of claim 1, wherein the visual representation is generated by an operating system of the computing system. ### Claim 8 (depends on 1) The method of claim 1, wherein the device worn by the user is a head-mounted device that blocks the user from seeing the physical environment directly. ### Claim 9 (depends on 1) The method of claim 1, wherein: the current viewpoint of the eye of the user corresponds to a first eye of the user; and the visual representation and the virtual-reality scene are displayed on a first display of the device configured to be viewed by the first eye of the user. ### Claim 10 (depends on 9) The method of claim 9, further comprising: determining a second viewpoint of a second eye of the user; rendering a second output image based on the second viewpoint of the user and the one or more 3D models of the objects; generating a second visual representation of at least a portion of the physical environment surrounding the user; and displaying the second visual representation with a second virtual-reality scene on a second display of the device configured to be viewed by the second eye of the user. ### Claim 11 One or more computer-readable non-transitory storage media embodying software that is operable when executed to: display a virtual-reality scene on a device worn by a user; receive a request from the user to display a visual representation of at least a portion of a physical environment surrounding the user; access data associated with the physical environment captured by one or more cameras of the device; generate, based on the data, depth measurements of one or more objects in the physical environment; generate, based on the depth measurements, one or more three-dimensional (3D) models of the one or more objects in the physical environment; render an output image based on a current viewpoint of an eye of the user and the one or more 3D models of the one or more objects in the physical environment; generate, based on the output image, the visual representation requested by the user; and display the visual representation with the virtual-reality scene to the user. ### Claim 12 (depends on 11) The media of claim 11, wherein the visual representation is an outline of the one or more objects in the physical environment. ### Claim 13 (depends on 12) The media of claim 12, wherein the displayed outline of the one or more objects is superimposed over the virtual-reality scene. ### Claim 14 (depends on 12) The media of claim 12, wherein the displayed outline has a uniform color tint. ### Claim 15 (depends on 14) The media of claim 14, wherein the virtual-reality scene is generated by an application and the uniform color tint is selected by the application. ### Claim 16 A system comprising: one or more processors; and one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to: display a virtual-reality scene on a device worn by a user; receive a request from the user to display a visual representation of at least a portion of a physical environment surrounding the user; access data associated with the physical environment captured by one or more cameras of the device; generate, based on the data, depth measurements of one or more objects in the physical environment; generate, based on the depth measurements, one or more three-dimensional (3D) models of the one or more objects in the physical environment; render an output image based on a current viewpoint of an eye of the user and the one or more 3D models of the one or more objects in the physical environment; generate, based on the output image, the visual representation requested by the user; and display the visual representation with the virtual-reality scene to the user. ### Claim 17 (depends on 16) The system of claim 16, wherein the visual representation is an outline of the one or more objects in the physical environment. ### Claim 18 (depends on 17) The system of claim 17, wherein the displayed outline of the one or more objects is superimposed over the virtual-reality scene. ### Claim 19 (depends on 17) The system of claim 17, wherein the displayed outline has a uniform color tint. ### Claim 20 (depends on 19) The system of claim 19, wherein the virtual-reality scene is generated by an application and the uniform color tint is selected by the application.