Real-time tracking-compensated image effects
AssigneeSnap Inc.
InventorsSamuel Edward Hare, Fedir Poliakov, Guohui Wang, Xuehan Xiong, Jianchao Yang, Linjie Yang, Shah Tanmay Anilkumar
Life of this patent
Estimating the term…
Abstract
A mobile device can generate real-time complex visual image effects using asynchronous processing pipeline. A first pipeline applies a complex image process, such as a neural network, to keyframes of a live image sequence. A second pipeline generates flow maps that describe feature transformations in the image sequence. The flow maps can be used to process non-keyframes on the fly. The processed keyframes and non-keyframes can be used to display a complex visual effect on the mobile device in real-time or near real-time.
Claims20
A method comprising:
- generating, using one or more processors of a user device, an image sequence using an image sensor, the image sequence comprising a plurality of frames including keyframes and non-keyframes;
- generating flow maps by applying an optical flow scheme to the plurality of frames;
- generating modified keyframes by applying a machine learning scheme to the keyframes;
- generating modified non-keyframes by applying the flow maps to the modified keyframes;
- and
- storing, in memory, the modified keyframes and the modified non-keyframes.
A system comprising:
- one or more processors of a machine;
- a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising:
- generating an image sequence using an image sensor, the image sequence comprising a plurality of frames including keyframes and non-keyframes;
- generating flow maps by applying an optical flow scheme to the plurality of frames;
- generating modified keyframes by applying a machine learning scheme to the keyframes;
- generating modified non-keyframes by applying the flow maps to the modified keyframes;
- and
- storing the modified keyframes and the modified non-keyframes.
A non-transitory machine readable medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
- generating an image sequence using an image sensor, the image sequence comprising a plurality of frames including keyframes and non-keyframes;
- generating flow maps by applying an optical flow scheme to the plurality of frames;
- generating modified keyframes by applying a machine learning scheme to the keyframes;
- generating modified non-keyframes by applying the flow maps to the modified keyframes;
- and
- storing the modified keyframes and the modified non-keyframes.
Dependencies are inferred from the claim text.