# US 12,182,327 B2 · Method and device for debugging program execution and content playback > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,182,327 B2 - Document ID: US12182327B2 - Status: Granted - Kind: B2 - Application number: 18/523,324 - Filed: 2023-11-29 - Published: 2024-03-21 - Granted: 2024-12-31 - Earliest claimed: 2021-05-03 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Michael J. Gutensohn; Payal Jotwani; Mark E. Drummond; Daniel L. Kovacs - Web page: https://patentgrep.com/patent/US12182327B2 ## CPC - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/012: {Head tracking input arrangements} - G06F 11/362: Debugging of software - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract In one implementation, a method for recording an XR environment. The method includes: presenting, via the display device, a graphical environment with one or more virtual agents, wherein the graphical environment corresponds to a composition of extended reality (XR) content, including the one or more virtual agents, and an image stream of a physical environment captured from a first point-of-view (POV) of the physical environment; detecting, via the one or more input devices, a user input selecting a first virtual agent from among the one or more virtual agents; and in response to detecting the user input, recording a plurality of data streams associated with the graphical environment including a first image stream of the graphical environment from the first POV and one or more data streams of the graphical environment from a current POV of the first virtual agent. ## Claims ### Claim 1 A method comprising: at a computing system including non-transitory memory and one or more processors, wherein the computing system is communicatively coupled to a display device and one or more input devices: detecting, via the one or more inputs devices, a first user input selecting a previously recorded instance of a graphical environment including one or more virtual agents (VAs); in response to detecting the first user input, presenting, via the display device, the previously recorded instance of the graphical environment from a first point-of-view (POV), and presenting, via the display device, a plurality of controls associated with playback of the previously recorded instance of the graphical environment including user interface (UI) elements for changing from the first POV to a second POV and for activating one or more layers associated with the first POV; detecting, via the one or more inputs devices, a second user input selecting a respective VA among the one or more VAs; and in response to detecting the second user input, presenting, via the display device, the previously recorded instance of the graphical environment from the second POV associated with the respective VA. ### Claim 2 (depends on 1) The method of claim 1, wherein the first POV corresponds to one of a neutral observation perspective or a perspective of a user that captured the previously recorded instance of the graphical environment. ### Claim 3 (depends on 1) The method of claim 1, wherein the second POV is associated with a plurality of sensory perceptions of the respective VA based on a perception profile for the respective VA. ### Claim 4 (depends on 3) The method of claim 3, wherein the plurality of sensory perceptions includes at least one of a thermal perception, an optical perception, an auditory perception, or an olfactory perception of the graphical environment from the second POV. ### Claim 5 (depends on 3) The method of claim 3, wherein the graphical environment includes UI elements for toggling the plurality of sensory perceptions. ### Claim 6 (depends on 3) The method of claim 3, wherein the second POV is associated with at least two overlapping sensory perceptions of the respective VA based on the perception profile for the respective VA. ### Claim 7 (depends on 6) The method of claim 6, wherein the at least two overlapping sensory perceptions are associated with visually distinct appearances. ### Claim 8 (depends on 3) The method of claim 3, wherein the second POV is associated with a first sensory perception of the respective VA based on the perception profile for the respective VA. ### Claim 9 (depends on 8) The method of claim 8, wherein the first sensory perception corresponds to a non-optical sense that is visualized as a colorized or a texturized vector field. ### Claim 10 (depends on 8) The method of claim 8, wherein the second POV includes a visualization of at least one of a projected trajectory for the respective VA and actionable items within the graphical environment. ### Claim 11 (depends on 1) The method of claim 1, wherein the graphical environment includes UI elements for toggling one or more layers associated with one or more sensory perception from the first POV, and wherein the one or more layers correspond to one of an occlusion layer, a texture map layer, a point cloud layer, a markup layer, a skeletal rigging layer, or an application-specific layer. ### Claim 12 (depends on 1) The method of claim 1, further comprising: presenting, via the display device, a menu of representations of previously recorded instances, wherein the previously recorded instance of the graphical environment is selected from the menu of representations of previously recorded instances. ### Claim 13 (depends on 1) The method of claim 1, wherein the plurality of controls further includes UI elements for initiating playback, pausing playback, changing playback speed, changing a current playback timestamp, adding markup to the previously recorded instance of the graphical environment, or modifying portions of the previously recorded instance of the graphical environment. ### Claim 14 (depends on 1) The method of claim 1, wherein the first and second user inputs correspond to one of a touch input, a hand-tracking input, a voice command, or an eye tracking input. ### Claim 15 (depends on 1) The method of claim 1, wherein the one or more virtual agents correspond to one of a humanoid, an animal, or a robot. ### Claim 16 A device comprising: one or more processors; a non-transitory memory; an interface for communicating with a display device and one or more input devices; and one or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the device to: detect, via the one or more inputs devices, a first user input selecting a previously recorded instance of a graphical environment including one or more virtual agents (VAs); in response to detecting the first user input, present, via the display device, the previously recorded instance of the graphical environment from a first point-of-view (POV), and present, via the display device, a plurality of controls associated with playback of the previously recorded instance of the graphical environment including user interface (UI) elements for changing from the first POV to a second POV and for activating one or more layers associated with the first POV; detect, via the one or more inputs devices, a second user input selecting a respective VA among the one or more VAs; and in response to detecting the second user input, present, via the display device, the previously recorded instance of the graphical environment from the second POV associated with the respective VA. ### Claim 17 (depends on 16) The device of claim 16, wherein the graphical environment includes UI elements for toggling one or more layers associated with one or more sensory perception from the first POV, and wherein the one or more layers correspond to one of an occlusion layer, a texture map layer, a point cloud layer, a markup layer, a skeletal rigging layer, or an application-specific layer. ### Claim 18 (depends on 16) The device of claim 16, wherein the one or more programs further cause the device to: present, via the display device, a menu of representations of previously recorded instances, wherein the previously recorded instance of the graphical environment is selected from the menu of representations of previously recorded instances. ### Claim 19 (depends on 16) The device of claim 16, wherein the plurality of controls further includes UI elements for initiating playback, pausing playback, changing playback speed, changing a current playback timestamp, adding markup to the previously recorded instance of the graphical environment, or modifying portions of the previously recorded instance of the graphical environment. ### Claim 20 A non-transitory memory storing one or more programs, which, when executed by one or more processors of a device with an interface for communicating with a display device and one or more input devices, cause the device to: detect, via the one or more inputs devices, a first user input selecting a previously recorded instance of a graphical environment including one or more virtual agents (VAs); in response to detecting the first user input, present, via the display device, the previously recorded instance of the graphical environment from a first point-of-view (POV), and present, via the display device, a plurality of controls associated with playback of the previously recorded instance of the graphical environment including user interface (UI) elements for changing from the first POV to a second POV and for activating one or more layers associated with the first POV; detect, via the one or more inputs devices, a second user input selecting a respective VA among the one or more VAs; and in response to detecting the second user input, present, via the display device, the previously recorded instance of the graphical environment from the second POV associated with the respective VA. ### Claim 21 (depends on 20) The non-transitory memory of claim 20, wherein the graphical environment includes UI elements for toggling one or more layers associated with one or more sensory perception from the first POV, and wherein the one or more layers correspond to one of an occlusion layer, a texture map layer, a point cloud layer, a markup layer, a skeletal rigging layer, or an application-specific layer. ### Claim 22 (depends on 20) The non-transitory memory of claim 20, wherein the one or more programs further cause the device to: present, via the display device, a menu of representations of previously recorded instances, wherein the previously recorded instance of the graphical environment is selected from the menu of representations of previously recorded instances. ### Claim 23 (depends on 20) The non-transitory memory of claim 20, wherein the plurality of controls further includes UI elements for initiating playback, pausing playback, changing playback speed, changing a current playback timestamp, adding markup to the previously recorded instance of the graphical environment, or modifying portions of the previously recorded instance of the graphical environment.