Live style transfer on a mobile device

A mobile device comprising:

• a processor;
• and
• memory, the memory storing instructions for the processor to execute a method, the method comprising:
• receiving, on a mobile device, a plurality of sets of neural network weighting factors from a remote computing device via a network interface, the neural network weighting factors implemented into a first neural network located on the mobile device, wherein the neural network weighting factors are pre-trained on the remote computing device;
• receiving a temporal sequence of video data on the mobile device from a camera associated with the mobile device;
• determining camera poses for frames in the temporal sequence of video data;
• estimating a camera path based on the determined camera poses, wherein the camera path extends outside of a physical trajectory defined by the camera poses associated with two frames in the temporal sequence of video data, wherein the trajectory is defined by camera poses associated with the two frames;
• extrapolating a viewpoint on the camera path outside of the physical trajectory of the two frames in the video data to synthesize an extrapolated frame for a multi-view interactive digital media representation;
• applying the first neural network, including the neural network weighting factors, to frames in the video data and to transform the extrapolated frame to apply a style transformation to the video data and the extrapolated frame;
• outputting the transformed video data and the extrapolated frame to a display of the mobile device;
• and
• recording the transformed video data and the extrapolated frame to a memory module.

A non-transitory computer readable medium storing instructions to cause a processor to execute a method, the method comprising:

• receiving, on a mobile device, a plurality of sets of neural network weighting factors from a remote computing device via a network interface, the neural network weighting factors implemented into a first neural network located on the mobile device, wherein the neural network weighting factors are pre-trained on the remote computing device;
• receiving a temporal sequence of video data on the mobile device from a camera associated with the mobile device;
• determining camera poses for frames in the temporal sequence of video data;
• estimating a camera path based on the determined camera poses, wherein the camera path extends outside of a physical trajectory defined by the camera poses associated with two frames in the temporal sequence of video data, wherein the trajectory is defined by camera poses associated with the two frames;
• extrapolating a viewpoint on the camera path outside of the physical trajectory of the two frames in the video data to synthesize an extrapolated frame for a multi-view interactive digital media representation;
• applying the first neural network, including the neural network weighting factors, to frames in the video data and to transform the extrapolated frame to apply a style transformation to the video data and the extrapolated frame;
• outputting the transformed video data and the extrapolated frame to a display of the mobile device;
• and
• recording the transformed video data and the extrapolated frame to a memory module.