# US 2019/0377458 A1 · Augmented reality user interface control > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0377458 A1 - Document ID: US20190377458A1 - Status: Published application - Kind: A1 - Application number: 16/547,045 - Filed: 2019-08-21 - Published: 2019-12-12 - Earliest claimed: 2017-12-22 - Assignees: Snap Inc. - Applicants: Snap Inc. - Inventors: Yurii Monastyrshyn; Oleksandr Pyshchenko - Web page: https://patentgrep.com/patent/US20190377458A1 ## CPC - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/012: {Head tracking input arrangements} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G06V 20/20: in augmented reality scenes - G06V 40/161: {Detection; Localisation; Normalisation} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Systems, devices, media, and methods are presented for controlling user interface with an object depicted within the user interface. The systems and methods initiates an augmented reality mode configured to present augmented reality elements within a graphical user interface. A face is detected within a field of view of an image capture component and presented within the graphical user interface. in response to detecting the face, the systems and methods sequentially present a set of augmented reality elements within the graphical user interface. A subset of the augmented reality elements and the face may be depicted contemporaneously. A movement is detected of at least a portion of the face relative to a first augmented reality element and presentation of the first augmented reality element is modified and at least one second augmented reality element is presented. ## Claims ### Claim 1 A method, comprising: detecting, by one or more processors of a device, a face within a field of view of an image capture component associated with the device; presenting, within a graphical user interface, the face and a first augmented reality element; moving the first augmented reality element relative to the face in the graphical user interface; detecting, as the first augmented reality element moves, a change in a feature of the face from a first state to a second state; and in response to detecting the change in the feature of the face from the first state to the second state, modifying a visual attribute of the first augmented reality element based on a distance between the first augmented reality element and the feature of the face in the graphical user interface. ### Claim 2 (depends on 1) The method of claim 1, further comprising initiating an augmented reality mode configured to present augmented reality elements within the graphical user interface. ### Claim 3 (depends on 1) The method of claim 1, wherein the first augmented reality element and the face are depicted within the graphical user interface contemporaneously. ### Claim 4 (depends on 1) The method of claim 1, wherein detecting the change comprises detecting a movement of at least a portion of the face relative to the first augmented reality element. ### Claim 5 (depends on 4) The method of claim 4, wherein the movement of the portion of the face is a movement of a reference feature, the method further comprising: identifying a first state of the reference feature, the first state corresponding to first positions of a set of landmark points corresponding to the reference feature; and determining a change of the reference feature from the first state to a second state corresponding to second positions of at least one landmark point of the set of landmark points. ### Claim 6 (depends on 1) The method of claim 1, further comprising: determining an on-screen position of the face; determining a spatial location of the first augmented reality element within the graphical user interface, the spatial location determined at a time the change in the feature is detected; determining a proximity of the spatial location to the on-screen position of the face as being within a proximity threshold; and based on the proximity, presenting a second augmented reality element within the graphical user interface along with the face. ### Claim 7 (depends on 1) The method of claim 1, wherein the feature is a mouth, and wherein modifying the visual attribute of the first augmented reality element further comprises: determining the first augmented reality element, at the spatial location, obscures at least a portion of the mouth, the portion of the mouth obscured by the first augmented reality element being an interior surface of the mouth; in response to the first augmented reality element obscuring the portion of the mouth, modifying presentation of the first augmented reality element for a period of time prior to terminating presentation of the first augmented reality element; and causing presentation of a second augmented reality element spaced a distance apart from the face, the second augmented reality element moving along a trajectory toward the face. ### Claim 8 (depends on 1) The method of claim 1, wherein detecting the change further comprises: identifying a change of position of the face, within the graphical user interface, from a first position to a second position; in response to identifying the second position, determining a head location within the graphical user interface at a first time; and determining a spatial location of the first augmented reality element at a second time. ### Claim 9 (depends on 1) The method of claim 1, wherein modifying the visual attribute comprises: causing presentation of the first augmented reality element traveling in a first direction relative to the face; determining a contact between the first augmented reality element and a region of the face; and in response to detecting the contact, causing presentation of the first augmented reality element traveling in a second direction relative to the face. ### Claim 10 (depends on 1) The method of claim 1, further comprising: causing presentation of a second augmented reality element representing contact between the first augmented reality element and a region of the face. ### Claim 11 (depends on 11) The method of claim 11, further comprising: causing presentation of a second augmented reality element representing an interaction history of the face relative to the first augmented reality element. ### Claim 12 A system, comprising: one or more processors; and a non-transitory processor-readable storage medium coupled to the one or more processors, the non-transitory processor-readable storage medium storing processor executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: detecting a face within a field of view of an image capture component associated with the device; presenting within a graphical user interface the face and a first augmented reality element; moving the first augmented reality element relative to the face in the graphical user interface; detecting, as the first augmented reality element moves, a change in a feature of the face from a first state to a second state; and in response to detecting the change in the feature of the face from the first state to the second state, modifying a visual attribute of the first augmented reality element based on a distance between the first augmented reality element and the feature of the face in the graphical user interface. ### Claim 13 (depends on 12) The system of claim 12, the operations further comprising initiating an augmented reality mode configured to present augmented reality elements within the graphical user interface. ### Claim 14 (depends on 12) The system of claim 12, wherein the first augmented reality element and the face are depicted within the graphical user interface contemporaneously. ### Claim 15 (depends on 12) The system of claim 12, wherein detecting the change comprises detecting a movement of at least a portion of the face relative to the first augmented reality element. ### Claim 16 (depends on 15) The system of claim 15, wherein the movement of the portion of the face is a movement of a reference feature, the operations further comprising: identifying a first state of the reference feature, the first state corresponding to first positions of a set of landmark points corresponding to the reference feature; and determining a change of the reference feature from the first state to a second state corresponding to second positions of at least one landmark point of the set of landmark points. ### Claim 17 (depends on 12) The system of claim 12, the operations further comprising: determining an on-screen position of the face; determining a spatial location of the first augmented reality element within the graphical user interface, the spatial location determined at a time the change in the feature is detected; determining a proximity of the spatial location to the on-screen position of the face as being within a proximity threshold; and based on the proximity, presenting a second augmented reality element within the graphical user interface along with the face. ### Claim 18 (depends on 12) The system of claim 12, wherein the feature is a mouth, and wherein modifying the visual attribute of the first augmented reality element further comprises: determining the first augmented reality element, at the spatial location, obscures at least a portion of the mouth, the portion of the mouth obscured by the first augmented reality element being an interior surface of the mouth; in response to the first augmented reality element obscuring the portion of the mouth, modifying presentation of the first augmented reality element for a period of time prior to terminating presentation of the first augmented reality element; and causing presentation of a second augmented reality element spaced a distance apart from the face, the second augmented reality element moving along a trajectory toward the face. ### Claim 19 (depends on 12) The system of claim 12, wherein detecting the change further comprises: identifying a change of position of the face, within the graphical user interface, from a first position to a second position; in response to identifying the second position, determining a head location within the graphical user interface at a first time; and determining a spatial location of the first augmented reality element at a second time. ### Claim 20 A non-transitory processor-readable storage medium storing processor executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising: detecting a face within a field of view of an image capture component associated with the device; presenting within a graphical user interface the face and a first augmented reality element; moving the first augmented reality element relative to the face in the graphical user interface; detecting, as the first augmented reality element moves, a change in a feature of the face from a first state to a second state; and in response to detecting the change in the feature of the face from the first state to the second state, modifying a visual attribute of the first augmented reality element based on a distance between the first augmented reality element and the feature of the face in the graphical user interface.