Real-time tracking-compensated image effects
AssigneeSnap Inc.
InventorsSamuel Edward Hare, Fedir Poliakov, Guohui Wang, Xuehan Xiong, Jianchao Yang, Linjie Yang, Shah Tanmay Anilkumar
Life of this patent
Estimating the term…
Abstract
A mobile device can generate real-time complex visual image effects using asynchronous processing pipeline. A first pipeline applies a complex image process, such as a neural network, to keyframes of a live image sequence. A second pipeline generates flow maps that describe feature transformations in the image sequence. The flow maps can be used to process non-keyframes on the fly. The processed keyframes and non-keyframes can be used to display a complex visual effect on the mobile device in real-time or near real-time.
Claims20
A method comprising:
- generating, using an image sensor, an image sequence;
- generating modified keyframe images by processing alternating images from the image sequence using a machine learning scheme, the alternating images from the image sequence being keyframe images that are separated by one or more non-keyframe images;
- generating modified non-keyframe images using flow maps generated between the non-keyframe images and preceding keyframe images, the modified non-keyframe images being generated by applying the flow maps to the modified keyframe images;
- displaying, on a display device, a modified image sequence generated using the modified keyframe images and the modified non-keyframe images.
A system comprising:
- one or more processors of a machine;
- a display device;
- and
- a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising:
- generating an image sequence;
- generating modified keyframe images by processing alternating images from the image sequence using a machine learning scheme, the alternating images from the image sequence being keyframe images that are separated by one or more non-keyframe images;
- generating modified non-keyframe images using flow maps generated between the non-keyframe images and preceding keyframe images, the modified non-keyframe images being generated by applying the flow maps to the modified keyframe images;
- and
- displaying, on the display device, a modified image sequence generated using the modified keyframe images and the modified non-keyframe images.
A non-transitory machine readable medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
- generating an image sequence;
- generating modified keyframe images by processing alternating images from the image sequence using a machine learning scheme, the alternating images from the image sequence being keyframe images that are separated by one or more non-keyframe images;
- generating modified non-keyframe images using flow maps generated between the non-keyframe images and preceding keyframe images, the modified non-keyframe images being generated by applying the flow maps to the modified keyframe images;
- and
- displaying, on a display device, a modified image sequence generated using the modified keyframe images and the modified non-keyframe images.
Dependencies are inferred from the claim text.