# US 12,217,374 B2 · Surface aware lens > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,374 B2 - Document ID: US12217374B2 - Status: Granted - Kind: B2 - Application number: 18/132,940 - Filed: 2023-04-10 - Published: 2023-08-03 - Granted: 2025-02-04 - Earliest claimed: 2017-01-09 - Assignees: Snap Inc. - Applicants: Snap Inc. - Inventors: Andrew James McPhee; Ebony James Charlton; Samuel Edward Hare; Michael John Evans; Jokubas Dargis; Ricardo Sanchez-Saez - Web page: https://patentgrep.com/patent/US12217374B2 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors - G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object - G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) - G06T 15/205: {Image-based rendering} - G06T 15/60: Shadow generation - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program, and a method for rendering three-dimensional virtual objects within real-world environments. Virtual rendering of a three-dimensional virtual object can be altered appropriately as a user moves around the object in the real-world through utilization of a redundant tracking system comprising multiple tracking sub-systems. Virtual object rendering can be with respect to a reference surface in a real-world three-dimensional space depicted in a camera view of a mobile computing device. ## Claims ### Claim 1 A system comprising: a memory; a redundant tracking subsystem comprising a set of tracking subsystems; and at least one hardware processor coupled to the redundant tracking subsystem to perform operations comprising: rendering a virtual object to appear at a position in a three-dimensional space within a camera view; and maintaining, via the redundant tracking subsystem, the virtual object rendered at the position in the three-dimensional space while detecting one or more interruptions to one or more tracking indicia used by the redundant tracking subsystem, the detecting of the one or more interruptions to the one or more tracking indicia comprising determining a tracking indicia used by a first tracking subsystem of the redundant tracking subsystem to track the virtual object is unavailable, the maintaining of the virtual object rendered at the position in the three-dimensional space comprising switching from tracking the virtual object in six degrees of freedom (6DoF) via the first tracking subsystem to tracking the virtual object in three degrees of freedom (3DoF) via a second tracking subsystem based on the tracking indicia being unavailable. ### Claim 2 (depends on 1) The system of claim 1, wherein the rendering of the virtual object includes rendering a separate virtual representation related to the virtual object. ### Claim 3 (depends on 2) The system of claim 2, wherein the separate virtual representation is a simulated shadow of the virtual object. ### Claim 4 (depends on 1) The system of claim 1, further comprising receiving input indicative of a selection of a template from among a set of templates, the template corresponding to the virtual object. ### Claim 5 (depends on 1) The system of claim 1, wherein the virtual object comprises customized personal avatar content. ### Claim 6 (depends on 1) The system of claim 1, wherein the virtual object includes content selected based on external factors related to the three-dimensional space. ### Claim 7 (depends on 1) The system of claim 1, wherein the operations further comprise identifying tracking indicia operable to track the virtual object in the three-dimensional space, wherein the maintaining of the virtual object rendered at the position in the three-dimensional space comprises tracking the virtual object using the tracking indicia. ### Claim 8 (depends on 7) The system of claim 7, wherein: the tracking of the virtual object using the tracking indicia comprises tracking, by the first tracking subsystem from among the set of tracking subsystems, the virtual object at the position in the three-dimensional space using the tracking indicia; and the maintaining of the virtual object rendered at the position in the three-dimensional space further comprises detecting, by the first tracking subsystem, an interruption of the tracking indicia. ### Claim 9 (depends on 1) The system of claim 1, wherein the operations further comprise detecting a reference surface in a three-dimensional space captured within a camera view, wherein the position in three-dimensional space is based on the reference surface. ### Claim 10 (depends on 9) The system of claim 9, wherein the detecting of the reference surface is based on user input. ### Claim 11 (depends on 9) The system of claim 9, wherein the position in the three-dimensional space is a predefined distance from a reference surface. ### Claim 12 (depends on 11) The system of claim 11, wherein maintaining the virtual object at the position in the three-dimensional space comprises maintaining the virtual object rendered at the predefined distance from the reference surface. ### Claim 13 A method comprising: rendering a virtual object to appear at a position in a three-dimensional space within a camera view; and maintaining, via a redundant tracking subsystem comprising a set of tracking subsystems, the virtual object rendered at the position in the three-dimensional space while detecting one or more interruptions to one or more tracking indicia used by the redundant tracking subsystem, the detecting of the one or more interruptions to the one or more tracking indicia comprising determining a tracking indicia used by a first tracking subsystem of the redundant tracking subsystem to track the virtual object is unavailable, the maintaining of the virtual object rendered at the position in the three-dimensional space comprising switching from tracking the virtual object in six degrees of freedom (6DoF) via the first tracking subsystem to tracking the virtual object in three degrees of freedom (3DoF) via a second tracking subsystem based on the tracking indicia being unavailable. ### Claim 14 (depends on 13) The method of claim 13, wherein the rendering of the virtual object includes rendering a separate virtual representation related to the virtual object. ### Claim 15 (depends on 13) The method of claim 13, wherein the virtual object includes content selected based on external factors related to the three-dimensional space. ### Claim 16 (depends on 13) The method of claim 13, further comprising identifying tracking indicia operable to track the virtual object in the three-dimensional space, wherein the maintaining of the virtual object rendered at the position in the three-dimensional space comprises tracking the virtual object using the tracking indicia. ### Claim 17 (depends on 16) The method of claim 16, wherein: the tracking of the virtual object using the tracking indicia comprises tracking, by a first tracking subsystem from among the set of tracking subsystems, the virtual object at the position in the three-dimensional space using the tracking indicia; and the maintaining of the virtual object rendered at the position in the three-dimensional space further comprises detecting, by the first tracking subsystem, an interruption of the tracking indicia. ### Claim 18 (depends on 13) The method of claim 13, further comprising detecting a reference surface in a three-dimensional space captured within a camera view, wherein the position in three-dimensional space is based on the reference surface. ### Claim 19 (depends on 18) The method of claim 18, wherein the position in the three-dimensional space is a predefined distance from a reference surface, wherein maintaining the virtual object at the position in the three-dimensional space comprises maintaining the virtual object rendered at the predefined distance from the reference surface. ### Claim 20 A non-transitory machine-readable storage medium that includes instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising: rendering a virtual object to appear at a position in a three-dimensional space within a camera view; and maintaining, via a redundant tracking subsystem comprising a set of tracking subsystems, the virtual object rendered at the position in the three-dimensional space while detecting one or more interruptions to one or more tracking indicia used by the redundant tracking subsystem, the detecting of the one or more interruptions to the one or more tracking indicia comprising determining a tracking indicia used by a first tracking subsystem of the redundant tracking subsystem to track the virtual object is unavailable, the maintaining of the virtual object rendered at the position in the three-dimensional space comprising switching from tracking the virtual object in six degrees of freedom (6DoF) via the first tracking subsystem to tracking the virtual object in three degrees of freedom (3DoF) via a second tracking subsystem based on the tracking indicia being unavailable.