# US 11,908,093 B2 · 3D captions with semantic graphical elements > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,908,093 B2 - Document ID: US11908093B2 - Status: Granted - Kind: B2 - Application number: 18/121,893 - Filed: 2023-03-15 - Published: 2023-07-06 - Granted: 2024-02-20 - Earliest claimed: 2019-12-19 - Assignees: Snap Inc. - Applicants: Snap Inc. - Inventors: Kyle Goodrich; Samuel Edward Hare; Maxim Maximov Lazarov; Tony Mathew; Andrew James McPhee; Daniel Moreno; Wentao Shang - Web page: https://patentgrep.com/patent/US11908093B2 ## CPC - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06F 3/012: {Head tracking input arrangements} - G06F 3/04883: for inputting data by handwriting, e.g. gesture or text - G06T 15/80: Shading - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - H04L 51/10: Multimedia information - G06T 2219/2004: Aligning objects, relative positioning of parts - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program and method for performing operations comprising: receiving, by a messaging application, a video feed from a camera of a user device that depicts a face; receiving a request to add a 3D caption to the video feed; identifying a graphical element that is associated with context of the 3D caption; and displaying the 3D caption and the identified graphical element in the video feed at a position in 3D space of the video feed proximate to the face depicted in the video feed. ## Claims ### Claim 1 A system comprising: at least one hardware processor; and a memory storing instructions which, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations comprising: receiving a request to add a three-dimensional (3D) caption to an image that depicts an object; identifying a graphical element that is associated with context of the 3D caption; displaying the 3D caption in the image; and selectively displaying the identified graphical element in the image based on whether the 3D caption is displayed proximate to the object or on a surface depicted in the image. ### Claim 2 (depends on 1) The system of claim 1, wherein the operations further comprise: detecting input indicating that a user tapped the 3D caption that is displayed in the image; and in response to detecting the input, presenting text of the 3D caption in two-dimensions (2D) to enable the user to modify text of the 3D caption. ### Claim 3 (depends on 1) The system of claim 1, wherein the operations further comprise: determining that the image is received from a front-facing camera; and in response to determining that the image is received from the front-facing camera, automatically populating the 3D caption with the identified graphical element. ### Claim 4 (depends on 1) The system of claim 1, wherein the operations further comprise: determining that the image is received from a rear-facing camera; and in response to determining that the image is received from a rear-facing camera, populating the 3D caption with the identified graphical element in response to a user request. ### Claim 5 (depends on 1) The system of claim 1, wherein the operations further comprise orienting the 3D caption and the graphical element above the object. ### Claim 6 (depends on 1) The system of claim 1, wherein the operations further comprise curving the 3D caption and graphical element around a bottom of the object. ### Claim 7 (depends on 1) The system of claim 1, wherein the operations further comprise: receiving input that selects the identified graphical element that is displayed; and in response to receiving the input, presenting text of the 3D caption in two-dimensions in place of the 3D caption while maintaining display of the identified graphical element in the image. ### Claim 8 (depends on 7) The system of claim 7, wherein the operations further comprise: presenting a list of alternate graphical elements while maintaining display of the identified graphical element in the image; receiving a selection of a given alternate graphical element from the list of alternate graphical elements; and replacing the identified graphical element that is displayed in the image with the given alternate graphical element. ### Claim 9 (depends on 1) The system of claim 1, wherein the operations further comprise searching a database of graphical elements based on text of the 3D caption to identify the graphical element that is associated with the text of the 3D caption. ### Claim 10 (depends on 1) The system of claim 1, wherein the operations further comprise: determining context associated with the image; and automatically populating text of the 3D caption based on the context associated with the image. ### Claim 11 (depends on 10) The system of claim 10, wherein the context associated with the image is determined based on a current date and time or day of the week. ### Claim 12 (depends on 1) The system of claim 1, wherein the operations further comprise dimming a screen in which text of the 3D caption is presented to bring focus to the text of the 3D caption. ### Claim 13 (depends on 12) The system of claim 12, wherein the operations further comprise: determining that a user tapped on the screen at a location between two characters of the text; and positioning a cursor to modify the text starting at the location between the two characters of the text in response to determining that the user tapped on the screen at the location between the two characters of the text. ### Claim 14 (depends on 1) The system of claim 1, wherein the 3D caption includes words or phrases, and wherein the operations further comprise selecting a default graphical element to be displayed as the identified graphical element when none of the words or phrases in the 3D caption matches words or phrases stored in a database that associates words or phrases with graphical elements. ### Claim 15 (depends on 1) The system of claim 1, wherein the operations further comprise: in response to receiving the request to add the 3D caption, presenting a two-dimensional (2D) text entry interface; receiving a 2D text string from the 2D text entry interface; identifying the graphical element based on the 2D text string; converting the 2D text string to the 3D caption; and displaying the 3D caption and the identified graphical element in response to receiving indication of completion of entry of the 2D text string. ### Claim 16 (depends on 1) The system of claim 1, the operations comprising: in response to determining that the object is no longer detected in the image, moving the 3D caption to the surface depicted in the image without displaying the graphical element. ### Claim 17 A method comprising: receiving a request to add a three-dimensional (3D) caption to an image that depicts an object; identifying a graphical element that is associated with context of the 3D caption; displaying the 3D caption in the image; and selectively displaying the identified graphical element in the image based on whether the 3D caption is displayed proximate to the object or on a surface depicted in the image. ### Claim 18 (depends on 17) The method of claim 17, further comprising: detecting input indicating that a user tapped the 3D caption that is displayed in the image; and in response to detecting the input indicating that the user tapped the 3D caption, presenting text of the 3D caption in two-dimensions to enable the user to modify text of the 3D caption. ### Claim 19 (depends on 17) The method of claim 17, the image comprising a frame of a video. ### Claim 20 A non-transitory machine-readable medium storing instructions which, when executed by one or more processors of a machine, cause the machine to perform operations comprising: receiving a request to add a three-dimensional (3D) caption to an image that depicts an object; identifying a graphical element that is associated with context of the 3D caption; displaying the 3D caption in the image; and selectively displaying the identified graphical element in the image based on whether the 3D caption is displayed proximate to the object or on a surface depicted in the image.