# US 2023/0410441 A1 · Generating user interfaces displaying augmented reality graphics > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2023/0410441 A1 - Document ID: US20230410441A1 - Status: Published application - Kind: A1 - Application number: 17/845,626 - Filed: 2022-06-21 - Published: 2023-12-21 - Earliest claimed: 2022-06-21 - Assignees: Snap Inc. - Applicants: Snap Inc. - Inventors: Sharon Moll - Web page: https://patentgrep.com/patent/US20230410441A1 ## CPC - G06F 1/163: {Wearable computers, e.g. on a belt} - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06F 1/1686: {the I/O peripheral being an integrated camera} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/012: {Head tracking input arrangements} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/015: {Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object - G06F 3/0482: Interaction with lists of selectable items, e.g. menus - G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour - G06F 3/04883: for inputting data by handwriting, e.g. gesture or text - G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 2200/24: involving graphical user interfaces [GUIs] - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract An Augmented Reality (AR) graphics system is provided. The AR graphics system may detect an object in a real-world scene that corresponds to an AR graphics display surface. The AR graphics system may generate AR graphics that are displayed as overlays of the AR graphics display surface. The AR graphics system may track the motion of a graphics input tool with respect to the AR graphics display surface to generate AR graphics based on the motion of the graphics input tool. The AR graphics may be comprised of a number of markings generated based on the motion of the graphics input tool. ## Claims ### Claim 1 A computing device comprising: a camera; one or more processors; and memory storing instructions that, when executed by the one or more processors, cause the computing device to perform operations comprising: capturing, by the camera, one or more images within a field of view of the camera; activating an augmented reality (AR) content item that is executed within a client application; in response to activating the augmented reality content item, providing camera data that includes the one or more images to one or more AR graphics services using one or more application programming interface (API) calls; obtaining AR graphics display data from the one or more AR graphics services, the AR graphics display data indicating an AR graphics display surface included in the field of view and a graphics input tool included in the field of view, the AR graphics display surface including a surface of one or more physical objects included in an environment in which the computing device is located; determining, based on at least one of the AR graphics display data and the camera data, a path of motion of the graphics input tool; and causing display of a user interface overlaid on the AR graphics display surface, the user interface including an AR graphic having at least one of a shape or one or more contours that follow the path of motion of the graphics input tool. ### Claim 2 (depends on 1) The computing device of claim 1, wherein the computing device is a head-worn device that includes at least one display device and the user interface is displayed by the at least one display device. ### Claim 3 (depends on 1) The computing device of claim 1, wherein the memory stores additional instructions that, when executed by the one or more processors, cause the computing device to perform additional operations comprising: in response to determining that the graphics input tool is within a threshold distance of the AR graphics display surface, activating one or more features of the augmented reality content item to at least one of create or modify the AR graphic; determining that the graphics input tool is outside of the threshold distance of the AR graphics display surface; and in response to determining that the graphics input tool is outside of the threshold distance of the AR graphics display surface, deactivating the one or more features of the augmented reality content item. ### Claim 4 (depends on 1) The computing device of claim 1, wherein the memory stores additional instructions that, when executed by the one or more processors, causes the computing device to perform additional operations comprising: causing an AR graphics menu to be displayed within the user interface, the AR graphics menu including a number of user interface elements, individual user interface elements of the number of user interface elements being selectable to activate one or more features of an augmented reality content item to at least one of create or modify at least a portion of the AR graphic. ### Claim 5 (depends on 4) The computing device of claim 4, wherein the memory stores additional instructions that, when executed by the one or more processors, causes the computing device to perform additional operations comprising: determining that the graphics input tool is within an additional threshold distance of a user interface element of the number of user interface elements of the AR graphics menu; and causing a feature of the user interface element to be activated. ### Claim 6 (depends on 1) The computing device of claim 1, wherein the memory stores additional instructions that, when executed by the one or more processors, causes the computing device to perform additional operations comprising: obtaining, based on the camera data, coordinates in real world space for the AR graphics display surface; and determining, based on the camera data, a distance from the camera to the AR graphics display surface. ### Claim 7 (depends on 6) The computing device of claim 6, wherein the memory stores additional instructions that, when executed by the one or more processors, causes the computing device to perform additional operations comprising: determining that the distance from the camera to the AR graphics display surface has changed from a first distance to a second distance; and modifying a level of magnification of the AR graphic within the user interface based on the distance from the camera to the AR graphics display surface changing from the first distance to the second distance. ### Claim at position 8 -20. (canceled) ### Claim 21 A computer-implemented method comprising: capturing, by one or more cameras of one or more computing devices including one or more processors and memory, one or more images within a field of view of the one or more cameras; activating, by at least one computing device of the one or more computing devices, an augmented reality (AR) content item that is executed within a client application; in response to activating the augmented reality content item, providing, by at least one computing device of the one or more computing devices, camera data that includes the one or more images to one or more AR graphics services using one or more application programming interface (API) calls; obtaining, by at least one computing device of the one or more computing devices, AR graphics display data from the one or more AR graphics services, the AR graphics display data indicating an AR graphics display surface included in the field of view and a graphics input tool included in the field of view, the AR graphics display surface including a surface of one or more physical objects included in an environment in which the computing device is located; determining, by at least one computing device of the one or more computing devices and based on at least one of the AR graphics display data and the camera data, a path of motion of the graphics input tool; and causing, by at least one computing device of the one or more computing devices, display of a user interface overlaid on the AR graphics display surface, the user interface including an AR graphic having at least one of a shape or one or more contours that follow the path of motion of the graphics input tool. ### Claim 22 (depends on 21) The computer-implemented method of claim 21, wherein the one or more computing devices include a head-worn device having at least one display device and the user interface is displayed by the at least one display device. ### Claim 23 (depends on 21) The computer-implemented method of claim 21, comprising: in response to determining that the graphics input tool is within a threshold distance of the AR graphics display surface, activating, by at least one computing device of the one or more computing devices, one or more features of the augmented reality content item to at least one of create or modify the AR graphic; determining, by at least one computing device of the one or more computing devices, that the graphics input tool is outside of the threshold distance of the AR graphics display surface; and in response to determining that the graphics input tool is outside of the threshold distance of the AR graphics display surface, deactivating, by at least one computing device of the one or more computing devices, the one or more features of the augmented reality content item. ### Claim 24 (depends on 21) The computer-implemented method of claim 21, comprising: causing, by at least one computing device of the one or more computing devices, an AR graphics menu to be displayed within the user interface, the AR graphics menu including a number of user interface elements, individual user interface elements of the number of user interface elements being selectable to activate one or more features of an augmented reality content item to at least one of create or modify at least a portion of the AR graphic. ### Claim 25 (depends on 24) The computer-implemented method of claim 24, comprising: determining, by at least one computing device of the one or more computing devices, that the graphics input tool is within an additional threshold distance of a user interface element of the number of user interface elements of the AR graphics menu; and causing, by at least one computing device of the one or more computing devices, a feature of the user interface element to be activated. ### Claim 26 (depends on 21) The computer-implemented method of claim 21, comprising: obtaining, by at least one computing device of the one or more computing devices and based on the camera data, coordinates in real world space for the AR graphics display surface; and determining, by at least one computing device of the one or more computing devices and based on the camera data, a distance from the one or more cameras to the AR graphics display surface. ### Claim 27 (depends on 26) The computer-implemented method of claim 26, comprising: determining, by at least one computing device of the one or more computing devices, that the distance from the one or more cameras to the AR graphics display surface has changed from a first distance to a second distance; and modifying, by at least one computing device of the one or more computing devices, a level of magnification of the AR graphic within the user interface based on the distance from the one or more cameras to the AR graphics display surface changing from the first distance to the second distance. ### Claim 28 One or more computer-readable storage media storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: capturing, by one or more cameras of one or more computing devices, one or more images within a field of view of the one or more cameras; activating an augmented reality (AR) content item that is executed within a client application; in response to activating the augmented reality content item, providing camera data that includes the one or more images to one or more AR graphics services using one or more application programming interface (API) calls; obtaining AR graphics display data from the one or more AR graphics services, the AR graphics display data indicating an AR graphics display surface included in the field of view and a graphics input tool included in the field of view, the AR graphics display surface including a surface of one or more physical objects included in an environment in which the computing device is located; determining, based on at least one of the AR graphics display data and the camera data, a path of motion of the graphics input tool; and causing display of a user interface overlaid on the AR graphics display surface, the user interface including an AR graphic having at least one of a shape or one or more contours that follow the path of motion of the graphics input tool. ### Claim 29 (depends on 28) The one or more computer-readable storage media of claim 28, wherein the one or more computing devices include a head-worn device having at least one display device and the user interface is displayed by the at least one display device. ### Claim 30 (depends on 28) The one or more computer-readable storage media of claim 28, storing additional computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform additional operations comprising: in response to determining that the graphics input tool is within a threshold distance of the AR graphics display surface, activating one or more features of the augmented reality content item to at least one of create or modify the AR graphic; determining that the graphics input tool is outside of the threshold distance of the AR graphics display surface; and in response to determining that the graphics input tool is outside of the threshold distance of the AR graphics display surface, deactivating the one or more features of the augmented reality content item. ### Claim 31 (depends on 28) The one or more computer-readable storage media of claim 28, storing additional computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform additional operations comprising: causing an AR graphics menu to be displayed within the user interface, the AR graphics menu including a number of user interface elements, individual user interface elements of the number of user interface elements being selectable to activate one or more features of an augmented reality content item to at least one of create or modify at least a portion of the AR graphic. ### Claim 32 (depends on 31) The one or more computer-readable storage media of claim 31, storing additional computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform additional operations comprising: determining that the graphics input tool is within an additional threshold distance of a user interface element of the number of user interface elements of the AR graphics menu; and causing a feature of the user interface element to be activated. ### Claim 33 (depends on 28) The one or more computer-readable storage media of claim 28, storing additional computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform additional operations comprising: obtaining, based on the camera data, coordinates in real world space for the AR graphics display surface; and determining, based on the camera data, a distance from the one or more cameras to the AR graphics display surface; determining that the distance from the one or more cameras to the AR graphics display surface has changed from a first distance to a second distance; and modifying a level of magnification of the AR graphic within the user interface based on the distance from the one or more cameras to the AR graphics display surface changing from the first distance to the second distance.