# US 10,771,763 B2 · Volumetric video-based augmentation with user-generated content > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,771,763 B2 - Document ID: US10771763B2 - Status: Granted - Kind: B2 - Application number: 16/201,525 - Filed: 2018-11-27 - Published: 2020-05-28 - Granted: 2020-09-08 - Earliest claimed: 2018-11-27 - Assignees: AT&T Intellectual Property I, L.P. - Applicants: AT&T Intellectual Property I, L.P. - Inventors: Eric Zavesky; Zhu Liu; David Crawford Gibbon; Behzad Shahraray; Tan Xu - Web page: https://patentgrep.com/patent/US10771763B2 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06T 7/55: from multiple images - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 15/04: Texture mapping - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching} - G06V 20/20: in augmented reality scenes - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - G11B 27/036: Insert-editing - H04N 13/117: the virtual viewpoint locations being selected by the viewers or determined by viewer tracking - H04N 13/156: Mixing image signals - H04N 21/8146: {involving graphical data, e.g. 3D object, 2D graphics} - H04N 21/816: {involving special video data, e.g 3D video} - G06T 2207/10016: Video; Image sequence - G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence - G06T 2207/10024: Color image - G06T 2207/20072: Graph-based image processing - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30196: Human being; Person - G06T 2207/30232: Surveillance - G06T 2219/2004: Aligning objects, relative positioning of parts - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A processing system having at least one processor may obtain a two-dimensional source video, select a volumetric video associated with at least one feature of the source video from a library of volumetric videos, identify a first object in the source video, and determine a location of the first object within a space of the volumetric video. The processing system may further obtain a three-dimensional object model of the first object, texture map the first object to the three-dimensional object model of the first object to generate an enhanced three-dimensional object model of the first object, and modify the volumetric video to include the enhanced three-dimensional object model of the first object in the location of the first object within the space of the volumetric video. ## Claims ### Claim 1 A method comprising: obtaining, by a processing system including at least one processor, a source video, wherein the source video is a two-dimensional video; selecting, by the processing system, a volumetric video associated with at least one feature of the source video from a library of volumetric videos; identifying, by the processing system, a first object in the source video; performing, by the processing system, a spatial alignment of the source video to the volumetric video, by detecting key points in both the source video and the volumetric video and calculating a plurality of vectors between the first object and the key points; determining, by the processing system, a location of the first object within a space of the volumetric video, wherein the location of the first object is determined in accordance with the spatial alignment; obtaining, by the processing system, a three-dimensional object model of the first object; texture mapping, by the processing system, the first object to the three-dimensional object model of the first object to generate an enhanced three-dimensional object model of the first object; and modifying, by the processing system, the volumetric video to include the enhanced three-dimensional object model of the first object in the location of the first object within the space of the volumetric video. ### Claim 2 (depends on 1) The method of claim 1, wherein the alignment further comprises a time alignment of the source video and the volumetric video. ### Claim 3 (depends on 1) The method of claim 1, wherein the obtaining the source video further comprises: obtaining, by the processing system, the at least one feature of the source video, wherein the at least one feature comprises at least one of: location information; time information; an event tag; or a keyword. ### Claim 4 (depends on 3) The method of claim 3, wherein the selecting comprises: determining, by the processing system, that the volumetric video matches the at least one feature of the source video. ### Claim 5 (depends on 3) The method of claim 3, wherein the selecting is in accordance with a ranked list of volumetric videos from the library of volumetric videos based upon a level of matching to the at least one feature of the source video. ### Claim 6 (depends on 1) The method of claim 1, further comprising: detecting, by the processing system, a second object in the source video, wherein the at least one feature of the source video comprises the second object. ### Claim 7 (depends on 6) The method of claim 6, wherein the selecting the volumetric video associated with the at least one feature of the source video comprises: detecting, by the processing system, the second object in the volumetric video. ### Claim 8 (depends on 1) The method of claim 1, further comprising: presenting, by the processing system via an endpoint device, the volumetric video that is modified. ### Claim 9 (depends on 1) The method of claim 1, further comprising: generating, by the processing system, an output video comprising a two dimensional traversal of the volumetric video that is modified. ### Claim 10 (depends on 9) The method of claim 9, wherein the output video is generated in accordance with a selection by a user of at least one viewing perspective within the space of the volumetric video. ### Claim 11 (depends on 9) The method of claim 9, further comprising: presenting, by the processing system via an endpoint device, the output video. ### Claim 12 (depends on 11) The method of claim 11, wherein the source video is obtained from the endpoint device. ### Claim 13 (depends on 1) The method of claim 1, wherein the at least one feature is specified by a user associated with the source video. ### Claim 14 (depends on 1) The method of claim 1, wherein the obtaining the three-dimensional object model of the first object is in accordance with a catalog of two-dimensional objects and three-dimensional object models that are matched to the two-dimensional objects. ### Claim 15 (depends on 14) The method of claim 14, wherein the obtaining the three-dimensional object model comprises: matching, by the processing system, the first object to one of the two-dimensional objects in the catalog; and obtaining, by the processing system from the catalog, the three-dimensional object model that is matched to the one two-dimensional object. ### Claim 16 (depends on 15) The method of claim 15, wherein the matching is in accordance with a machine learning-based image detection model. ### Claim 17 (depends on 1) The method of claim 1, wherein the modifying comprises changing voxel data of the volumetric video that corresponds to the first object so that the voxel data corresponds to the enhanced three-dimensional object model. ### Claim 18 (depends on 1) The method of claim 1, wherein the selecting is based on a matching of visual features in the source video to visual features in the volumetric video. ### Claim 19 A non-transitory computer-readable medium storing instructions which, when executed by a processing system including at least one processor, cause the processing system to perform operations, the operations comprising: obtaining a source video, wherein the source video is a two-dimensional video; selecting a volumetric video associated with at least one feature of the source video from a library of volumetric videos; identifying, by the processing system, a first object in the source video; performing, by the processing system, a spatial alignment of the source video to the volumetric video, by detecting key points in both the source video and the volumetric video and calculating a plurality of vectors between the first object and the key points; determining, by the processing system, a location of the first object within a space of the volumetric video, wherein the location of the first object is determined in accordance with the spatial alignment; obtaining a three-dimensional object model of the first object; texture mapping the first object to the three-dimensional object model of the first object to generate an enhanced three-dimensional object model of the first object; and modifying the volumetric video to include the enhanced three-dimensional object model of the first object in the location of the first object within the space of the volumetric video. ### Claim 20 A device comprising: a processing system including at least one processor; and a computer-readable medium storing instructions which, when executed by the processing system, cause the processing system to perform operations, the operations comprising: obtaining a source video, wherein the source video is a two-dimensional video; selecting a volumetric video associated with at least one feature of the source video from a library of volumetric videos; identifying, by the processing system, a first object in the source video; performing, by the processing system, a spatial alignment of the source video to the volumetric video, by detecting key points in both the source video and the volumetric video and calculating a plurality of vectors between the first object and the key points; determining, by the processing system, a location of the first object within a space of the volumetric video, wherein the location of the first object is determined in accordance with the spatial alignment; obtaining a three-dimensional object model of the first object; texture mapping the first object to the three-dimensional object model of the first object to generate an enhanced three-dimensional object model of the first object; and modifying the volumetric video to include the enhanced three-dimensional object model of the first object in the location of the first object within the space of the volumetric video.