3d media elements in 2d video

A method of modifying media content, comprising

• obtaining one or more two dimensional (2D) images representing a 2D scene including the media content;
• generating, based on the one or more of the 2D images, a three-dimensional (3D) representation of media elements present in the 2D scene, two of the three dimensions identifying a location within a respective image of the media element and a third dimension identifying a distance of the media element from a captured perspective of the 2D scene;
• creating a 3D volume representation of the 2D scene by aggregating the 3D representations;
• creating a map of 3D void space in the 2D scene by subtracting the 3D volume representation from a total 3D volume of the 2D scene;
• segmenting the 3D void space into a plurality of 3D void segments;
• scoring each of the plurality of 3D void segments based on one or more of tracking characteristics of the 3D void segment, lighting characteristics of the respective 3D void segment, a duration of the respective 3D void segment, a size of the respective 3D void segment, or a distance of the respective 3D void segment from a center of the 2D scene;
• selecting one of the 3D void segments based on the respective score of each 3D void segment;
• inserting a 3D media element into the selected 3D void segment;
• and
• modifying the one or more 2D images to include the inserted 3D media element within the selected 3D void segment.

A computer readable storage medium comprising instructions that when executed configure hardware processing circuitry to perform operations to modify media content, the operations comprising:

• obtaining one or more two dimensional (2D) images representing a 2D scene including the media content;
• generating, based on the one or more of the 2D images, a three-dimensional (3D) representation of media elements present in the 2D scene, two of the three dimensions identifying a location within a respective image of the media element and a third dimension identifying a distance of the media element from a captured perspective of the 2D scene;
• creating a 3D volume representation of the 2D scene by aggregating the 3D representations;
• creating a map of 3D void space in the 2D scene by subtracting the 3D volume representation from a total 3D volume of the 2D scene;
• segmenting the 3D void space into a plurality of 3D void segments;
• scoring each of the plurality of 3D void segments based on one or more of tracking characteristics of the 3D void segment, lighting characteristics of the respective 3D void segment, a duration of the respective 3D void segment, a size of the respective 3D void segment, or a distance of the respective 3D void segment from a center of the 2D scene;
• selecting one of the 3D void segments based on the respective score of each 3D void segment;
• inserting a 3D media element into the selected 3D void segment;
• and
• modifying the one or more 2D images to include the inserted 3D media element within the selected 3D void segment.