Three dimensional aware video compositing
AssigneeAdobe Inc.
InventorsZhan Xu, Kim P. Pimmel, Jimei Yang
Life of this patent
Estimating the term…
Abstract
Three dimensional aware video compositing techniques are described. In one or more examples, subject data is produced that defines a subject depicted in frames of a subject video and viewpoint data describing movement of a viewpoint with respect to the frames of the subject video. Three-dimensional data is formed that defines a three-dimensional representation of an environment depicted in frames of an environment video. A composited video is generated by aligning the environment with the movement of the viewpoint of the subject based on the subject data and the three-dimensional data, which is then rendered, e.g., presented for display in a user interface.
Claims20
A method comprising:
- producing, by a processing device, subject data defining a subject depicted in frames of a subject video and viewpoint data describing movement of a viewpoint with respect to the frames of the subject video;
- forming, by the processing device using a machine-learning model, three-dimensional data defining a three-dimensional representation of an environment depicted in frames of an environment video;
- generating, by the processing device, a composited video by aligning the environment with the movement of the viewpoint of the subject based on the subject data and the three-dimensional data;
- and
- rendering, by the processing device, the composited video.
A computing device comprising:
- a processing device;
- and
- a computer-readable storage medium storing instructions that, responsive to execution by the processing device, causes the processing device to perform operations including:
- producing subject data defining a subject and a trajectory with respect to frames of a subject video;
- forming a neural radiance field configured as a trained model using machine learning as a three-dimensional representation of an environment depicted in frames of an environment video;
- and
- generating a composited video of the subject from the subject video and the environment from the environment video by rendering the neural radiance field based on the trajectory.
One or more computer-readable storage media storing instructions that, responsive to execution by a processing device, causes the processing device to perform operations comprising:
- generating a composite video using a machine-learning model by synchronizing movement of a viewpoint in relation to a subject captured in a subject video with a three-dimensional representation of an environment generated from an environment video, the synchronizing based on:
- depth data defining a depth of the subject in frames of the subject video;
- or
- extracting viewpoint motion from the subject video and using that motion to render the environment.
Dependencies are inferred from the claim text.