# US 12,118,603 B2 · Customized e-commerce tags in realtime multimedia content > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,603 B2 - Document ID: US12118603B2 - Status: Granted - Kind: B2 - Application number: 17/873,603 - Filed: 2022-07-26 - Published: 2023-01-19 - Granted: 2024-10-15 - Earliest claimed: 2020-09-08 - Assignees: Block, Inc. - Applicants: Block, Inc. - Inventors: Gabor Angeli; Marsal Gavalda; James Webb - Web page: https://patentgrep.com/patent/US12118603B2 ## CPC - G06Q 30/0631: {Recommending goods or services} - G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance - G06Q 10/087: Inventory or stock management, e.g. order filling, procurement or balancing against orders - G06Q 10/08726: {for replenishment processing, procedures, or recommendations using forecasting or optimisation} - G06Q 10/08741: {by picking of items from inventory for fulfillment} - G06Q 10/08774: {using consumer inventory reporting} - G06Q 30/0257: {User requested} - G06Q 30/0269: {based on user profile or attribute} - G06Q 30/0272: Period of advertisement exposure - G06Q 30/0633: {Managing shopping lists, e.g. compiling or processing purchase lists (shipping orders G06Q10/083; order filling G06Q10/087)} - G06Q 30/0641: {utilising user interfaces specially adapted for shopping} - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Techniques described herein are directed to receiving audio input during playback of multimedia content to trigger purchase and payment for an item represented in the multimedia content. In an example, multimedia content is sent to a user device. The multimedia content may contain a representation of an item and an indication that the item can be purchased. Based on receipt of an indication that trigger words were spoken during playback, a recording is initiated. A purchase intent of the user for the item is determined based on the recording. A graphical user interface is displayed via the user device to initiate payment by the user for the item. ## Claims ### Claim 1 A method comprising: sending, by at least one server computing device and to a user device of a user, multimedia content, wherein the multimedia content includes a representation of an item; receiving, by the at least one server computing device, audio data generated by the user device from a first time to a second time during playback of the multimedia content, wherein the first time is associated with display of the representation of the item by the user device; processing, by an automatic speech recognition component associated with the at least one server computing device, the audio data to generate text data representing one or more words spoken from the first time to the second time; determining, by a natural language component associated with the at least one server computing device via natural language processing of the text data, a purchase intent to purchase the item; responsive to determining the purchase intent to purchase the item, causing display, by the at least one server computing device and via the user device, of an interactive element selectable to purchase the item; detecting, by the at least one server computing device, an interaction with the interactive element; and based at least in part on detecting the interaction with the interactive element, causing display, by the at least one server computing device, of a graphical user interface comprising a selectable icon selectable to initiate payment for the item, wherein the graphical user interface is customized based at least in part on a user profile of the user. ### Claim 2 (depends on 1) The method as claim 1 recites, wherein the second time is at least one of an end of the playback or an offset from the first time. ### Claim 3 (depends on 1) The method as claim 1 recites, wherein the purchase intent comprises an intent to purchase an instance of the item with a particular attribute. ### Claim 4 (depends on 1) The method as claim 1 recites, wherein the one or more words spoken from the first time to the second time comprise one or more trigger words, and wherein determining the purchase intent comprises determining that the one or more trigger words were spoken. ### Claim 5 (depends on 4) The method as claim 4 recites, wherein the one or more trigger words are predefined by a merchant or at least one user of a plurality of users associated with the user device, and wherein the one or more trigger words are stored in respective user profiles of the plurality of users. ### Claim 6 (depends on 4) The method as claim 4 recites, wherein the one or more trigger words are associated with another platform. ### Claim 7 (depends on 1) The method as claim 1 recites, wherein the first time is further associated with an indication that audio input can be accepted, and wherein the indication is displayed at a point in the playback when the representation of the item is being displayed. ### Claim 8 (depends on 7) The method as claim 7 recites, further comprising: generating, by the at least one server computing device, the indication; and embedding, by the at least one server computing device, the indication in the multimedia content at a point in the playback when the representation of the item is being displayed. ### Claim 9 (depends on 1) The method as claim 1 recites, further comprising: identifying, by the at least one server computing device, the user profile based at least in part on at least one of a device identifier of the user device or access information used to access a platform associated with viewing the multimedia content. ### Claim 10 (depends on 1) The method as claim 1 recites, wherein the user profile includes a purchase history of the user, and wherein content of the graphical user interface is based at least in part on at least on the purchase history. ### Claim 11 A system comprising: one or more processors, and one or more non-transitory computer-readable media storing instructions executable by the one or more processors, wherein the instructions cause the one or more processors to perform acts comprising: sending, by at least one server computing device and to a user device of a user, multimedia content, wherein the multimedia content includes a representation of an item; receiving, by the at least one server computing device, audio data generated by the user device from a first time to a second time during playback of the multimedia content, wherein the first time is associated with display of the representation of the item by the user device; processing, by an automatic speech recognition component associated with the at least one server computing device, the audio data to generate text data representing one or more words spoken from the first time to the second time; determining, by a natural language component associated with the at least one server computing device via natural language processing of the text data, a purchase intent to purchase the item; responsive to determining the purchase intent to purchase the item, causing display, by the at least one server computing device and via the user device, of an interactive element selectable to purchase the item; detecting, by the at least one server computing device, an interaction with the interactive element; and based at least in part on detecting the interaction with the interactive element, causing display, by the at least one server computing device, of a graphical user interface comprising a selectable icon selectable to initiate payment for the item, wherein the graphical user interface is customized based at least in part on a user profile of the user. ### Claim 12 (depends on 11) The system as claim 11 recites, wherein the second time is at least one of an end of the playback or an offset from the first time. ### Claim 13 (depends on 11) The system as claim 11 recites, wherein the purchase intent comprises an intent to purchase an instance of the item with a particular attribute. ### Claim 14 (depends on 11) The system as claim 11 recites, wherein the first time is further associated with an indication that audio input can be accepted, and wherein the indication is displayed at a point in the playback when the representation of the item is being displayed. ### Claim 15 (depends on 14) The system as claim 14 recites, the acts further comprising: generating, by the at least one server computing device, the indication; and embedding, by the at least one server computing device, the indication in the multimedia content at a point in the playback when the representation of the item is being displayed. ### Claim 16 One or more non-transitory computer-readable media storing instructions executable by one or more processors that, when executed by the one or more processors, cause the one or more processors to perform acts comprising: sending, by at least one server computing device and to a user device of a user, multimedia content, wherein the multimedia content includes a representation of an item; receiving, by the at least one server computing device, audio data generated by the user device from a first time to a second time during playback of the multimedia content, wherein the first time is associated with display of the representation of the item by the user device; processing, by an automatic speech recognition component associated with the at least one server computing device, the audio data to generate text data representing one or more words spoken from the first time to the second time; determining, by a natural language component associated with the at least one server computing device via natural language processing of the text data, a purchase intent to purchase the item; responsive to determining the purchase intent to purchase the item, causing display, by the at least one server computing device and via the user device, of an interactive element selectable to purchase the item; detecting, by the at least one server computing device, an interaction with the interactive element; and based at least in part on detecting the interaction with the interactive element, causing display, by the at least one server computing device, of a graphical user interface comprising a selectable icon selectable to initiate payment for the item, wherein the graphical user interface is customized based at least in part on a user profile of the user. ### Claim 17 (depends on 16) The one or more non-transitory computer-readable media as claim 16 recites, wherein the second time is at least one of an end of the playback or an offset from the first time. ### Claim 18 (depends on 16) The one or more non-transitory computer-readable media as claim 16 recites, wherein the purchase intent comprises an intent to purchase an instance of the item with a particular attribute. ### Claim 19 (depends on 16) The one or more non-transitory computer-readable media as claim 16 recites, wherein the first time is further associated with an indication that audio input can be accepted, and wherein the indication is displayed at a point in the playback when the representation of the item is being displayed. ### Claim 20 (depends on 19) The one or more non-transitory computer-readable media as claim 19 recites, the acts further comprising: generating, by the at least one server computing device, the indication; and embedding, by the at least one server computing device, the indication in the multimedia content at a point in the playback when the representation of the item is being displayed.