# US 11,442,549 B1 · Placement of 3D effects based on 2D paintings > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,442,549 B1 - Document ID: US11442549B1 - Status: Granted - Kind: B1 - Application number: 16/781,043 - Filed: 2020-02-04 - Granted: 2022-09-13 - Earliest claimed: 2019-02-07 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Florent Andorra - Web page: https://patentgrep.com/patent/US11442549B1 ## CPC - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/0236: {using selection techniques to select from displayed items} - G06T 7/74: {involving reference images or patches} - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06V 20/20: in augmented reality scenes - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - G06T 2200/24: involving graphical user interfaces [GUIs] - G06T 2210/12: Bounding box - G06T 2215/16: Using real world measurements to influence rendering - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Various implementations disclosed herein include devices, systems, and methods that enable more intuitive and efficient selection and positioning of visual effects. In some implementations, a visual effect is selected and a surface of a physical element is identified for application of the visual effect based on tracking one or more positions or gestures of a user's hand. ## Claims ### Claim 1 A non-transitory computer-readable storage medium, storing instructions executable on a device to perform operations comprising: providing visibility of a physical element; obtaining selection of a visual effect; detecting a direction and a movement path of an input object based on projecting a ray from an eye of a user through the input object, wherein the input object is associated with a hand of the user; determining a placement position for the visual effect upon the physical element based on determining an intersection of the ray with a portion of the physical element; determining a translation of the visual effect based on the detected direction and movement path of the input object; and presenting a view of the translation of the visual effect upon the physical element at the placement position. ### Claim 2 (depends on 1) The non-transitory computer-readable storage medium of claim 1, wherein presenting the visual effect at the placement position causes the presented visual effect to abut a surface of the physical element when viewing the physical element from a viewing perspective. ### Claim 3 (depends on 1) The non-transitory computer-readable storage medium of claim 1, wherein the operations further comprise: obtaining images of a physical environment, and wherein providing the visibility of the physical element comprises displaying at least portions of the obtained images on a screen. ### Claim 4 (depends on 1) The non-transitory computer-readable storage medium of claim 1, wherein obtaining the selection comprises identifying a gesture based on tracking hand motion and identifying a type of the visual effect based on the gesture. ### Claim 5 (depends on 1) The non-transitory computer-readable storage medium of claim 1, wherein detecting the direction and the movement path comprises detecting a position of the device relative to the physical element. ### Claim 6 (depends on 1) The non-transitory computer-readable storage medium of claim 1, wherein determining the placement position comprises identifying a type of motion based on the detected direction and the movement path. ### Claim 7 (depends on 1) The non-transitory computer-readable storage medium of claim 1, wherein the operations further comprise displaying a bounding box for the visual effect. ### Claim 8 (depends on 1) The non-transitory computer-readable storage medium of claim 1, wherein the operations further comprise: tracking the direction and the movement path, and translating the visual effect in accordance with the tracked direction and the tracked movement path; and tracking a second direction and a second movement path of the input object, and adjusting a volume of the visual effect in accordance with the tracked second direction and the tracked second movement path. ### Claim 9 (depends on 1) The non-transitory computer-readable storage medium of claim 1, wherein the visual effect is a particle effect. ### Claim 10 (depends on 1) The non-transitory computer-readable storage medium of claim 1, wherein the detected direction and the movement path is based on an extension of: i) a pointing finger of the user, or ii) a wand in the hand of the user. ### Claim 11 (depends on 1) The non-transitory computer-readable storage medium of claim 1, wherein the detected direction and the movement path is based on a reference position from the eye of the user to an end of the hand or an end of a finger of the user. ### Claim 12 (depends on 1) The non-transitory computer-readable storage medium of claim 1, wherein detecting the direction and the movement path comprises projecting a second ray from the hand of the user or another device held by the user. ### Claim 13 A system comprising: a device; a processor; and a computer-readable storage medium comprising instructions that upon execution by the processor cause the system to perform operations, the operations comprising: providing visibility of a physical element; obtaining selection of a visual effect; detecting a direction and a movement path of an input object associated with a hand of a user based on projecting a ray from an eye of the user through the input object; determining a placement position for the visual effect upon the physical element using the detected direction and movement path of the input object based on determining an intersection of the ray with a portion of the physical element; determining a translation of the visual effect based on the detected direction and movement path of the input object; and presenting a view of the translation of the visual effect upon the physical element at the placement position. ### Claim 14 (depends on 13) The system of claim 13, wherein presenting the visual effect at the placement position causes the presented visual effect to abut a surface of the physical element when viewing the physical element from a viewing perspective. ### Claim 15 (depends on 13) The system of claim 13, wherein obtaining the selection comprises identifying a gesture based on tracking hand motion and identifying a type of the visual effect based on the gesture. ### Claim 16 (depends on 13) The system of claim 13, wherein detecting the direction and the movement path comprises detecting a position of the device relative to the physical element. ### Claim 17 (depends on 13) The system of claim 13, wherein determining the placement position comprises identifying a type of motion based on the detected direction and the movement path. ### Claim 18 (depends on 13) The system of claim 13, wherein the operations further comprise: tracking the direction and the movement path, and translating the visual effect in accordance with the tracked direction and the tracked movement path; and tracking a second direction and a second movement path of the input object, and adjusting a volume of the visual effect in accordance with the tracked second direction and the tracked second movement path. ### Claim 19 A method comprising: at a device with a processor: providing visibility of a physical element; obtaining selection of a visual effect; detecting a direction and a movement path of an input object associated with a hand of a user based on projecting a ray from an eye of the user through the input object; determining a placement position for the visual effect upon the physical element based on determining an intersection of the ray with a portion of the physical element; determining a translation of the visual effect based on the detected direction and movement path of the input object; and presenting a view of the translation of the visual effect upon the physical element at the placement position. ### Claim 20 (depends on 19) The method of claim 19, wherein presenting the visual effect at the placement position causes the presented visual effect to abut a surface of the physical element when viewing the physical element from a viewing perspective. ### Claim 21 (depends on 19) The method of claim 19, wherein obtaining the selection comprises identifying a gesture based on tracking hand motion and identifying a type of the visual effect based on the gesture. ### Claim 22 (depends on 19) The method of claim 19, wherein the detected direction and the movement path is based on device position or a position of the eye of the user.