# US 12,223,610 B2 · Chroma key content management systems and methods > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,223,610 B2 - Document ID: US12223610B2 - Status: Granted - Kind: B2 - Application number: 17/937,308 - Filed: 2022-09-30 - Published: 2023-01-26 - Granted: 2025-02-11 - Earliest claimed: 2013-08-21 - Assignees: Nantmobile, LLC - Applicants: Nantmobile, LLC - Inventors: Evgeny Dzhurinskiy; Ludmila Bezyaeva - Web page: https://patentgrep.com/patent/US12223610B2 ## CPC - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - G06T 15/04: Texture mapping - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - H04N 5/2226: {Determination of depth image, e.g. for foreground/background separation (determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10)} - H04N 5/272: Means for inserting a foreground image in a background image, i.e. inlay, outlay - H04N 9/75: Chroma key - G06T 2210/41: Medical - G06T 2215/12: Shadow map, environment map - G06T 2215/16: Using real world measurements to influence rendering - G06T 2219/2012: Colour editing, changing, or manipulating; Use of colour codes - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A system of properly displaying chroma key content is presented. The system obtains a digital representation of a 3D environment, for example a digital photo, and gathers data from that digital representation. The system renders the digital representation in an environmental model and displays that digital representation upon an output device. Depending upon the context, content anchors of the environmental model are selected which will be altered by suitable chroma key content. The chroma key content takes into consideration the position and orientation of the chroma key content relative to the content anchor and relative to the point of view that the environmental model is displayed from in order to accurately display chroma key content in a realistic manner. ## Claims ### Claim 1 An augmented reality system comprising: at least one non-transitory computer readable medium storing software instructions executable by at least one processor; and a content management engine executable on the at least one processor according to the stored software instructions to: obtain at least one digital image of a 3D environment; execute one or more object recognition algorithms on the at least one digital image to identify one or more surfaces from among a plurality of objects in the at least one digital image; identify at least one content anchor associated with the one or more identified surfaces; obtain first augmented reality content from a content source; establish a first content position and a first content orientation relative to the at least one content anchor; configure an output device to render and display, with respect to a point of view of a camera, the first augmented reality content at the first content position and the first content orientation within a displayed image of the 3D environment, at least a portion of the displayed image of the 3D environment extending beyond the at least one content anchor; obtain second augmented reality content from the content source; establish a second content position and a second content orientation relative to the first augmented reality content; and configure the output device to render and display the second augmented reality content at the second content position and the second content orientation within the displayed image of the 3D environment, wherein the content management engine configures the output device to render the second augmented reality content as a digital representation of a commercial product overlaid on the first augmented reality content. ### Claim 2 (depends on 1) The augmented reality system of claim 1, wherein the at least one content anchor is derived from one or more feature descriptors of the at least one digital image. ### Claim 3 (depends on 1) The augmented reality system of claim 1, wherein the at least one content anchor is based on recognition of an object in the at least one digital image. ### Claim 4 (depends on 1) The augmented reality system of claim 1, wherein the content management engine is further executable on the at least one processor according to the stored software instructions to derive one or more environment attributes from the at least one digital image, and the at least one content anchor is based on the one or more environment attributes. ### Claim 5 (depends on 1) The augmented reality system of claim 1, wherein said identifying the at least one content anchor comprises receiving a user's selection of the at least one content anchor. ### Claim 6 (depends on 1) The augmented reality system of claim 1, wherein said identifying the at least one content anchor comprises determining the at least one content anchor based on one or more characteristics of the plurality of objects in the at least one digital image. ### Claim 7 (depends on 1) The augmented reality system of claim 1, wherein said identifying the at least one content anchor comprises determining the at least one content anchor based on one or more characteristics of the first augmented reality content. ### Claim 8 (depends on 1) The augmented reality system of claim 1, wherein said identifying the at least one content anchor comprises determining the at least one content anchor based on a contextual trigger. ### Claim 9 (depends on 1) The augmented reality system of claim 1, wherein the first content orientation is established based at least in part on a game context. ### Claim 10 (depends on 1) The augmented reality system of claim 1, wherein the content management engine is further executable on the at least one processor according to the stored software instructions to derive one or more environment attributes from the at least one digital image, and the first augmented reality content is obtained at least in part based on the one or more environment attributes. ### Claim 11 (depends on 1) The augmented reality system of claim 1, wherein the content source comprises a database. ### Claim 12 (depends on 1) The augmented reality system of claim 1, wherein the content source comprises a commercial website. ### Claim 13 (depends on 1) The augmented reality system of claim 1, wherein the content management engine is configured to obtain the at least one digital image via the camera. ### Claim 14 (depends on 1) The augmented reality system of claim 1, wherein the second augmented reality content includes a commercial product image. ### Claim 15 (depends on 1) The augmented reality system of claim 1, wherein the first augmented reality content includes 2D or 3D game content. ### Claim 16 (depends on 1) The augmented reality system of claim 1, wherein the content management engine is configured to establish the second content orientation as an orientation of a commercial product in the 3D environment. ### Claim 17 (depends on 1) The augmented reality system of claim 1, wherein the at least one content anchor comprises a vector within an environmental model of the 3D environment. ### Claim 18 (depends on 1) The augmented reality system of claim 1, wherein the at least one content anchor comprises a coordinate within an environmental model of the 3D environment. ### Claim 19 (depends on 1) The augmented reality system of claim 1, wherein the at least one content anchor comprises an anchor position and an anchor orientation relative to the 3D environment. ### Claim 20 (depends on 19) The augmented reality system of claim 19, wherein the anchor orientation is based on the point of view. ### Claim 21 (depends on 19) The augmented reality system of claim 19, wherein the anchor position and the anchor orientation comprise six degrees of freedom. ### Claim 22 A method comprising: obtaining at least one digital image of a 3D environment; executing one or more object recognition algorithms on the at least one digital image to identify one or more surfaces from among a plurality of objects in the at least one digital image; identifying at least one content anchor associated with the one or more identified surfaces; obtaining first augmented reality content from a content source; establishing a first content position and a first content orientation relative to the at least one content anchor; configuring an output device to render and display, with respect to a point of view of a camera, the first augmented reality content at the first content position and the first content orientation within a displayed image of the 3D environment, at least a portion of the displayed image of the 3D environment extending beyond the at least one content anchor; obtaining second augmented reality content from the content source; establishing a second content position and a second content orientation relative to the first augmented reality content; and configuring the output device to render and display the second augmented reality content at the second content position and the second content orientation within the displayed image of the 3D environment, wherein the output device is configured to render the second augmented reality content as a digital representation of a commercial product overlaid on the first augmented reality content. ### Claim 23 A non-transitory computer readable medium containing program instructions for causing a processor to perform operations comprising: obtaining at least one digital image of a 3D environment; executing one or more object recognition algorithms on the at least one digital image to identify one or more surfaces from among a plurality of objects in the at least one digital image; identifying at least one content anchor associated with the one or more identified surfaces; obtaining first augmented reality content from a content source; establishing a first content position and a first content orientation relative to the at least one content anchor; configuring an output device to render and display, with respect to a point of view of a camera, the first augmented reality content at the first content position and the first content orientation within a displayed image of the 3D environment, at least a portion of the displayed image of the 3D environment extending beyond the at least one content anchor; obtaining second augmented reality content from the content source; establishing a second content position and a second content orientation relative to the first augmented reality content; and configuring the output device to render and display the second augmented reality content at the second content position and the second content orientation within the displayed image of the 3D environment, wherein the output device is configured to render the second augmented reality content as a digital representation of a commercial product overlaid on the first augmented reality content.