# US 2019/0172268 A1 · System and Method to Digitally Replace Objects in Images or Video > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0172268 A1 - Document ID: US20190172268A1 - Status: Published application - Kind: A1 - Application number: 16/267,649 - Filed: 2019-02-05 - Published: 2019-06-06 - Earliest claimed: 2011-12-05 - Assignees: AT&T Intellectual Property I, L.P. - Applicants: At&T Intellectual Property I, L.P. - Inventors: Eric Zavesky - Web page: https://patentgrep.com/patent/US20190172268A1 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06T 15/50: Lighting effects - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - G11B 27/036: Insert-editing - H04N 21/21805: {enabling multiple viewpoints, e.g. using a plurality of cameras} - H04N 21/234318: {by decomposing into objects, e.g. MPEG-4 objects} - H04N 21/4394: {involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams (arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58)} - H04N 21/44008: {involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - H04N 21/44012: {involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs} - H04N 21/440236: {by media transcoding, e.g. video is transformed into a slideshow of still pictures, audio is converted into text} - H04N 21/812: {involving advertisement data (advertising per se G06Q30/02)} - H04N 21/8146: {involving graphical data, e.g. 3D object, 2D graphics} - H04N 21/84: Generation or processing of descriptive data, e.g. content descriptors {(systems specially adapted for using meta-information in broadcast systems H04H60/73)} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A method includes receiving, at a computer system, video data of a video frame of video content. The method includes identifying, at the computer system, a first representation of a first object in the video frame to be replaced with a second representation of a second object based on absence of a visible feature in the first representation. The method includes manipulating, at the computer system, a three-dimensional model of the second object to generate the second representation. The second representation includes the visible feature. The method also includes generating, at the computer system, a modified version of the video frame that includes the second representation in place of the first representation. ## Claims ### Claim 1 A method comprising: receiving, at a computer system, video data of a video frame of video content; identifying, at the computer system, a first representation of a first object in the video frame to be replaced with a second representation of a second object, wherein the first representation is to be replaced based on absence of a visible feature in the first representation; manipulating, at the computer system, a three-dimensional model of the second object to generate the second representation, wherein the second representation includes the visible feature; and generating, at the computer system, a modified version of the video frame that includes the second representation in place of the first representation. ### Claim 2 (depends on 1) The method of claim 1, wherein generating the modified version of the video frame includes removing the first representation from the video frame. ### Claim 3 (depends on 2) The method of claim 2, wherein generating the modified version of the video frame further includes inserting the second representation to fill at least a portion of the frame previously occupied by the first representation. ### Claim 4 (depends on 3) The method of claim 3, wherein generating the modified version of the video frame further comprises using pixel hallucination to fill a first portion of the frame previously occupied by the first representation and not occupied by the second representation. ### Claim 5 (depends on 4) The method of claim 4, wherein pixels for the first portion are obtained from the video frame prior to removal of the first representation, a previous video frame of the video content, or a subsequent video frame of the video content. ### Claim 6 (depends on 1) The method of claim 1, wherein generating the modified version comprises: determining that a reflection of the first object is present in another portion of the video frame; and replacing the reflection of the first object with pixels corresponding to a reflection of the second object. ### Claim 7 (depends on 1) The method of claim 1, wherein manipulating the three-dimensional model of the second object includes adjusting a property of the three-dimensional model, the property including a color, size, shape, texture, lighting, shade, bi-directional reflectance distribution function, orientation, topology, or any combination thereof. ### Claim 8 (depends on 1) The method of claim 1, wherein manipulating the three-dimensional model of the second object comprises matching a particular visible feature of the second representation to match a corresponding visible feature of the first representation. ### Claim 9 (depends on 8) The method of claim 8, wherein the particular visible feature comprises a height of the second object, a width of the second object, a depth of the second object, or combinations thereof. ### Claim 10 (depends on 1) The method of claim 1, wherein the visible feature comprises at least a portion of a brand logo. ### Claim 11 An system comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to perform operations, the operations comprising: receiving video data of a video frame of video content; identifying a first representation of a first object in the video frame to be replaced with a second representation of a second object, the first representation to be replaced based on absence of a visible feature in the first representation; manipulating a three-dimensional model of the second object to generate the second representation, wherein the second representation includes the visible feature; and generating a modified version of the video frame that includes the second representation in place of the first representation. ### Claim 12 (depends on 11) The system of claim 11, wherein the processor is configured to: extrapolate the three-dimensional model of the second object based on a media content item that depicts the second object; and store the three-dimensional model in the memory. ### Claim 13 (depends on 12) The system of claim 12, wherein the media content item comprises a graphics file, a video file, or a combination thereof. ### Claim 14 (depends on 11) The system of claim 11, wherein manipulating the three-dimensional model includes rotating the three-dimensional model, changing a scale of the three-dimensional model, applying a texture to the three-dimensional model, or a combination thereof. ### Claim 15 (depends on 11) The system of claim 11, wherein the operations further comprise adjusting a bi-directional reflectance distribution function of the three-dimensional model to correspond to a particular bi-directional reflectance distribution function of the first representation. ### Claim 16 A computer-readable storage device storing instructions that, when executed by a processor, cause the processor to perform operations, the operations comprising: receiving video data of a video frame of video content; identifying a first representation of a first object in the video frame to be replaced with a second representation of a second object, the first representation to be replaced based on absence of a visible feature in the first representation; manipulating a three-dimensional model of the second object to generate the second representation, wherein the second representation includes the visible feature; and generating a modified version of the video frame that includes the second representation in place of the first representation. ### Claim 17 (depends on 16) The computer-readable storage device of claim 16, wherein the operations further comprise: computing the three-dimensional model of the second object based on a media content item that depicts the second object; and storing the three-dimensional model in a memory. ### Claim 18 (depends on 16) The computer-readable storage device of claim 16, wherein the operations further comprise sending the modified version to a user device. ### Claim 19 (depends on 18) The computer-readable storage device of claim 18, wherein the user device comprises a set-top box device, a media device, a mobile device, a personal computer, or combinations thereof. ### Claim 20 (depends on 16) The computer-readable storage device of claim 16, wherein generating the modified version further comprises: removing the first representation from the video frame; and inserting the second representation to fill the portion of the video frame previously occupied by the first representation.