# US 11,443,772 B2 · Method for triggering events in a video > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,443,772 B2 - Document ID: US11443772B2 - Status: Granted - Kind: B2 - Application number: 17/248,812 - Filed: 2021-02-09 - Published: 2021-06-03 - Granted: 2022-09-13 - Earliest claimed: 2014-02-05 - Assignees: Snap Inc. - Applicants: Snap Inc. - Inventors: Elena Shaburova; Victor Shaburov - Web page: https://patentgrep.com/patent/US11443772B2 ## CPC - G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06N 20/00: Machine learning - G06T 5/77: Retouching; Inpainting; Scratch removal - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G06T 7/251: {involving models} - G06T 7/33: using feature-based methods - G06T 7/90: Determination of colour characteristics - G06T 11/10: Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) - G06T 11/60: Creating or editing images; Combining images with text - G06T 13/40: of characters, e.g. humans, animals or virtual beings - G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation} - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - G06V 10/40: Extraction of image or video features - G06V 10/42: Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation - G06V 10/56: relating to colour - G06V 10/60: relating to illumination properties, e.g. using a reflectance or lighting model - G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44) - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - G06V 40/167: {using comparisons between temporally consecutive images} - G06V 40/168: {Feature extraction; Face representation} - G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships} - G06V 40/174: {Facial expression recognition} - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - H04N 5/262: Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects {; Cameras specially adapted for the electronic generation of special effects} - H04N 9/79: Processing of colour television signals in connection with recording - G06T 2207/10016: Video; Image sequence - G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence - G06T 2207/10024: Color image - G06T 2207/20036: Morphological image processing - G06T 2207/30201: Face - G06V 10/467: {Encoded features or binary features, e.g. local binary patterns [LBP]} - G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A computer implemented method of triggering events in a video, the method comprising: providing a list of objects with their states and corresponding events in video such that each state from the list triggers at least one event of the corresponding events, wherein each object from the list has at least one state triggering at least one event of the corresponding events from the list in video; detecting at least one object from the list that at least partially and at least occasionally is presented in frames of the video; tracking the at least one object and its state; triggering at least one event of the corresponding events from the list in video in case the state of the at least one object matches with one of its states from the list. ## Claims ### Claim 1 A computer-implemented method comprising: detecting, by a processor, a first object within frames of a video; determining that the first object performs a first gesture; based on determining that the first object performs the first gesture, selecting a first virtual animal of a plurality of virtual animals as a visualization; adding the visualization comprising the first virtual animal as a new object into the frames of the video; determining that the first object performs a second gesture after the visualization comprising the first virtual animal has been added into the frames of the video; based on determining that the first object performs the second gesture, selecting a second virtual animal of the plurality of virtual animals as the visualization; and changing the visualization comprising the first virtual animal to the visualization comprising the second virtual animal responsive to determining that the first object performs the second gesture after the visualization comprising the first virtual animal has been added into the frames of the video. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, further comprising: detecting, by the processor, physical displacement of a first portion of the first object relative to a second portion of the first object across the frames of the video; and selecting a virtual animal from the plurality of virtual animals based on a type of gesture performed by the first object corresponding to the physical displacement. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, further comprising: providing a list comprising a set of objects, a set of object states associated with the set of objects, and a set of events, wherein the set of objects are associated with one or more images of a user and each object from the set of objects has at least one object state triggering at least one event from the set of events in the video and each object state is associated with at least one point of the object of the set of objects, wherein the first object is in the list; tracking the first object across two or more frames of the video, the first object having a first object state; identifying a change in the first object state of the first object to a second object state; determining that the second object state of the first object matches a state from the set of object states; and in response to determining the match, triggering at least one event of the set of events in the video. ### Claim 4 (depends on 3) The computer-implemented method of claim 3 further comprising: detecting a face of the user within the frames of the video, the face including a set of landmark points corresponding to facial features; and aligning a mesh with the face of the user, the mesh containing a set of feature reference points, each feature reference point corresponding to a landmark point of the set of landmark points, wherein the second object state comprises a facial gesture, wherein the change from the first object state to the second object state corresponds to movement of a first portion of landmark points relative to one or more feature reference points of the mesh, indicating movement of the first portion of landmark points on the face of the user, while a second portion of landmark points remains aligned with corresponding feature reference points of the mesh. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, further comprising: determining that the first gesture corresponds to one of a plurality of eyes of the first object being closed. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the visualization is added after a time delay from when a first portion of the first object is physically displaced relative to a second portion of the first object, and wherein the visualization is removed from the video in case of detecting a state of the first object, such that the state of the first object corresponds to an average state of the first object based on at least one state prior to a physical displacement that triggered replacement of the at least the portion of the first object. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein the visualization is selected at random from the plurality of visualizations, and wherein the visualization is removed from the video in case of detecting change of a state of the first object from a current state to another state opposite to the current state. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, further comprising: determining that the first gesture corresponds to raising a nose of the first object with a finger; selecting a virtual pig as the virtual animal in response to determining that the first gesture corresponds to raising the nose of the first object with the finger; and turning the first object into the virtual pig in the frames of the video in response to determining that the first gesture corresponds to raising the nose of the first object with the finger. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, further comprising: determining that the first gesture corresponds to showing horns on the first object with fingers; selecting a virtual deer as the virtual animal in response to determining that the first gesture corresponds to showing horns on the first object with fingers; and turning the first object into the virtual deer in the frames of the video in response to determining that the first gesture corresponds to showing horns on the first object with fingers. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein the video comprises a real-time video stream, wherein detecting the first gesture comprises detecting that a first facial portion has been moved relative to a second facial portion, wherein the visualization comprises a first visualization corresponding to physical displacement of the first facial portion relative to the second facial portion, and wherein a second visualization of the plurality of visualization corresponds to physical displacement of a third facial portion relative to the second facial portion. ### Claim 11 A system, comprising: one or more processors of a mobile computing device; and a non-transitory processor-readable storage medium storing processor executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: detecting a first object within frames of a video; determining that the first object performs a first gesture; based on determining that the first object performs the first gesture, selecting a first virtual animal of a plurality of virtual animals as a visualization; adding the visualization comprising the first virtual animal as a new object into the frames of the video; determining that the first object performs a second gesture after the visualizationcomprising the first virtual animal has been added into the frames of the video; based on determining that the first object performs the second gesture, selecting a second virtual animal of the plurality of virtual animals as the visualization; and changing the visualization comprising the first virtual animal to the visualization comprising the second virtual animal responsive to determining that the first object performs the second gesture after the visualization comprising the first virtual animal has been added into the frames of the video. ### Claim 12 (depends on 5) The method of claim 5, wherein the first virtual animal comprises a virtual cat, further comprising turning the first object into the virtual cat in the frames of the video in response to determining that the first gesture corresponds to one of a plurality of eyes of the first object being closed. ### Claim 13 (depends on 11) The system of claim 11, wherein the operations further comprise: providing a list comprising a set of objects, a set of object states associated with the set of objects, and a set of events, wherein the set of objects are associated with one or more images of a user and each object from the set of objects has at least one object state triggering at least one event from the set of events in the video and each object state is associated with at least one point of the object of the set of objects, wherein the first object is in the list; tracking the first object across two or more frames of the video, the first object having a first object state; identifying a change in the first object state of the first object to a second object state; determining that the second object state of the first object matches a state from the set of object states; and in response to determining the match, triggering at least one event of the set of events in the video. ### Claim 14 (depends on 13) The system of claim 13, wherein the operations further comprise: detecting a face of the user within the frames of the video, the face including a set of landmark points corresponding to facial features; and aligning a mesh with the face of the user, the mesh containing a set of feature reference points, each feature reference point corresponding to a landmark point of the set of landmark points, wherein the second object state comprises a facial gesture, wherein the change from the first object state to the second object state corresponds to movement of a first portion of landmark points relative to one or more feature reference points of the mesh, indicating movement of the first portion of landmark points on the face of the user, while a second portion of landmark points remains aligned with corresponding feature reference points of the mesh. ### Claim 15 (depends on 11) The system of claim 11, wherein the operations further comprise: determining that the first gesture corresponds to one of a plurality of eyes of the first object being closed; selecting a virtual cat as the virtual animal in response to determining that the first gesture corresponds to one of a plurality of eyes of the first object being closed; and turning the first object into the virtual cat in the frames of the video in response to determining that the first gesture corresponds to one of a plurality of eyes of the first object being closed. ### Claim 16 (depends on 11) The system of claim 11, wherein the visualization is added after a time delay from when a first portion of the first object is physically displaced relative to a second portion of the first object, and wherein the visualization is removed from the video in case of detecting a state of the first object, such that the state of the first object corresponds to an average state of the first object based on at least one state prior to a physical displacement that triggered replacement of the at least the portion of the first object. ### Claim 17 (depends on 11) The system of claim 11, wherein the operations further comprise: determining that the first gesture corresponds to raising a nose of the first object with a finger; and selecting the first virtual animal in response to determining that the first gesture corresponds to raising the nose of the first object with the finger. ### Claim 18 (depends on 17) The system of claim 17, wherein the first virtual animal comprises a virtual pig, and wherein the operations further comprise turning the first object into the virtual pig in the frames of the video in response to determining that the first gesture corresponds to raising the nose of the first object with the finger. ### Claim 19 (depends on 11) The system of claim 11, wherein the operations further comprise: determining that the first gesture corresponds to showing horns on the first object with fingers; and selecting the first virtual in response to determining that the first gesture corresponds to showing horns on the first object with fingers. ### Claim 20 A non-transitory processor-readable storage medium storing processor executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising: detecting a first object within frames of a video; determining that the first object performs a first gesture; based on determining that the first object performs the first gesture; selecting a first virtual animal of a plurality of virtual animals as a visualization; adding the visualization comprising the first virtual animal as a new object into the frames of the video: determining that the first object performs a second gesture after the visualization comprising the first virtual animal has been added into the frames of the video; based on determining that the first object performs the second gesture, selecting a second virtual animal of the plurality of virtual animals as the visualization; and changing the visualization comprising the first virtual animal to the visualization comprising the second virtual animal responsive to determining that the first object performs the second gesture after the visualization comprising the first virtual animal has been added into the frames of the video.