# US 12,183,330 B2 · Theme detection for object-recognition-based notifications > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,183,330 B2 - Document ID: US12183330B2 - Status: Granted - Kind: B2 - Application number: 17/162,507 - Filed: 2021-01-29 - Published: 2021-07-01 - Granted: 2024-12-31 - Earliest claimed: 2019-12-30 - Assignees: Capital One Services, LLC - Applicants: Capital One Services, LLC - Inventors: Joshua Edwards; Michael Mossoba; Abdelkader Benkreira - Web page: https://patentgrep.com/patent/US12183330B2 ## CPC - G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning} - G10L 25/78: Detection of presence or absence of voice signals (switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10) - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G10L 15/1822: {Parsing for meaning understanding} - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 25/03: characterised by the type of extracted parameters - H04W 4/021: Services related to particular areas, e.g. point of interest [POI] services, venue services or geofences - G06T 2219/004: Annotating, labelling - G10L 15/16: using artificial neural networks - G10L 2015/228: {of application context} - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72) ## Abstract In certain embodiments, speech is converted to text for theme identification by natural language processing. Notification data is generated based on detected themes and the notification data may include rules for notification presentation on a client device. The notification data may include parameters for processing image data captured by an augmented reality device to detect one or more objects. The objects may be associated with the theme and detection thereof within captured image data, and in accordance with other rules, may cause the augmented reality device to present a notification with contextual relevance to a current environment of a user utilizing the augmented reality device. ## Claims ### Claim 1 A compute-power-constrained device for generating notifications related to objects in currently view on the compute-power-constrained device based on themes detected by natural language processing of speech, comprising: a camera, one or more processors, memory storing instructions that, when executed by the one or more processors, cause operations comprising: obtaining a natural language processing model configured to provide an output indicative of one or more themes identified in a set of chunks of input natural language text, wherein: the output indicative of the one or more themes is associated with theme information for at least one candidate theme based on scores of candidate themes identified in the set of chunks of input natural language text, the candidate themes are identified by the natural language processing model, based on the set of chunks of input natural language text, from a set of possible themes represented in training records identifying themes within corresponding ones of, or sets of, training chunks of natural language text, and at least some of the sets of training chunks of natural language text respectively include encoded links between training chunks in the set of chunks of input natural language text that are indicative of a shared theme in combination and not individually indicative of the shared theme; and in connection with receiving one or more audio requests of a conversation among two or more individuals, converting one or more segments of audio data of the one or more audio requests into corresponding chunks of natural language text to identify at least one theme for the one or more audio requests via the natural language processing model, and subsequently presenting, based on a subsequent detection of an object related to the at least one theme as being in view of the camera of the compute-power-constrained device, a reminder prompt that (i) indicates an action to be taken with respect to the at least one theme and (ii) is not presented coincident in time to the one or more audio requests being received. ### Claim 2 (depends on 1) The compute-power-constrained device of claim 1, wherein subsequently presenting the reminder prompt comprises subsequently presenting the reminder prompt that is presented to coincident in time to the object being in view of the camera. ### Claim 3 (depends on 1) The compute-power-constrained device of claim 1, wherein the reminder prompt comprises theme information associated with a shared theme identified within at least two corresponding chunks of natural language text, in combination and not individually, that are derived from the one or more audio requests. ### Claim 4 (depends on 1) The compute-power-constrained device of claim 1, wherein identifying the at least one theme comprises converting one or more segments of audio data of the one or more audio requests into corresponding chunks of natural language text, and wherein the compute-power-constrained device processes on the compute-power-constrained device, with the natural language processing model, the one or more audio requests, without transmitting over a network to a remote computing system the corresponding chunks of natural language text or the audio data from which the corresponding chunks of natural language text were converted. ### Claim 5 (depends on 1) The compute-power-constrained device of claim 1, the operations further comprising: labeling a first subset of the one or more segments of audio data as being from a first individual that is a party to the conversation; and labeling a second subset of the one or more segments of audio data as being from the second individual that is another party to the conversation. ### Claim 6 (depends on 5) The compute-power-constrained device of claim 5, wherein identifying the at least one theme comprises identifying, via the natural language processing model, a first theme indicating the first individual based on the labeling corresponding to the first and second individuals, the operations further comprising: in response to detecting the object based on the one or more audio requests, generating, coincident in time to the object appearing in view of the camera, at least one notification for the second individual that is related to the first theme indicating the first individual. ### Claim 7 A non-transitory computer readable medium storing instructions that, when executed by one or more processors, cause operations comprising: obtaining a natural language processing model configured to provide an output indicative of one or more themes identified in a set of chunks of input natural language text, wherein: the output indicative of the one or more themes is associated with theme information for at least one candidate theme based on scores of candidate themes identified in the set of chunks of input natural language text, the candidate themes are identified by the natural language processing model, based on the set of chunks of input natural language text, from a set of possible themes represented in training records identifying themes within corresponding ones of, or sets of, training chunks of natural language text, and at least some of the sets of training chunks of natural language text respectively include encoded links between training chunks in the set of chunks of input natural language text that are indicative of a shared theme in combination and not individually indicative of the shared theme; and in connection with receiving one or more audio requests, converting one or more segments of audio data of the one or more audio requests into corresponding chunks of natural language text to identify at least one theme for the one or more audio requests via the natural language processing model, and subsequently generating, based on a subsequent detection of an object related to the at least one theme as being in view of a camera, a notification that (i) indicates an action to be taken with respect to the at least one theme and (ii) is not presented coincident in time to the one or more audio requests being received. ### Claim 8 (depends on 7) The medium of claim 7, wherein converting the one or more segments of audio data to identify the at least one theme comprises: determining a sequence among the corresponding chunks of natural language text that are spoken by different individuals providing the one or more audio requests; and identifying the at least one theme for the one or more audio requests based on the sequence among the corresponding chunks of natural language text. ### Claim 9 (depends on 7) The medium of claim 7, the operations further comprising: in response to detecting the object based on the one or more audio requests, causing a user device to present the notification coincident in time to the object appearing in view of the camera, the notification not being presented coincident in time to the one or more audio requests being received. ### Claim 10 (depends on 9) The medium of claim 9, wherein the theme information is associated with a shared theme identified within at least two of the corresponding chunks of natural language text in combination and not individually. ### Claim 11 (depends on 7) The medium of claim 7, wherein a user device processes on the user device, with the natural language processing model, the one or more audio requests, without transmitting over a network to a remote computing system the corresponding chunks of natural language text or the audio data from which the corresponding chunks of natural language text were converted. ### Claim 12 (depends on 7) The medium of claim 7, the operations further comprising: labeling a first subset of the one or more segments of audio data as corresponding to a first individual that is a party to a conversation between the first individual and a second individual; and labeling a second subset of the one or more segments of audio data as corresponding to the second individual. ### Claim 13 (depends on 12) The medium of claim 12, the operations further comprising: identifying, via the natural language processing model, a first theme involving the first individual based on the labels corresponding to the first and second individuals; and in response to detecting the object based on the one or more audio requests, causing a user device of the second individual to present at least one notification, related to the first theme involving the first individual, coincident in time to the object appearing in view of the user device of the second individual. ### Claim 14 A method comprising: obtaining a natural language processing model configured to provide an output indicative of one or more themes identified in a set of chunks of input natural language text, wherein: the output indicative of the one or more themes is associated with theme information for at least one candidate theme based on scores of candidate themes identified in the set of chunks of input natural language text, and the candidate themes are identified by the natural language processing model, based on the set of chunks of input natural language text, from a set of possible themes represented in training records identifying themes within corresponding ones of, or sets of, training chunks of natural language text; receiving one or more audio requests; identifying, via the natural language processing model, at least one theme for the one or more audio requests via the natural language processing model; and subsequently presenting, based on detection of an object related to the at least one theme appears in image data captured by a camera, a notification that (i) indicates an action to be taken with respect to the at least one theme, (ii) is presented coincident in time to the object being detected in the image data, and (iii) is not presented coincident in time to the one or more audio requests being received. ### Claim 15 (depends on 14) The method of claim 14, wherein identifying the at least one theme comprises: determining a sequence among the corresponding chunks of natural language text that are spoken by different individuals providing the one or more audio requests; and identifying the at least one theme for the one or more audio requests based on the sequence among the corresponding chunks of natural language text. ### Claim 16 (depends on 14) The method of claim 14, further comprising: storing the at least one theme as part of a notification trigger rule after the at least one theme is identified for the one or more audio requests, wherein subsequently presenting the notification comprises subsequently presenting the notification related the at least one theme based on (i) the notification trigger rule and (ii) the detection of the object in the image data. ### Claim 17 (depends on 14) The method of claim 14, wherein identifying the at least one theme comprises converting one or more segments of audio data of the one or more audio requests into corresponding chunks of natural language text, and wherein the notification comprises theme information associated with a shared theme identified within at least two of the corresponding chunks of natural language text in combination and not individually. ### Claim 18 (depends on 14) The method of claim 14, wherein identifying the at least one theme comprises converting one or more segments of audio data of the one or more audio requests into corresponding chunks of natural language text, and wherein a user device processes on the user device, with the natural language processing model, the one or more audio requests, without transmitting over a network to a remote computing system the corresponding chunks of natural language text or the audio data from which the corresponding chunks of natural language text were converted. ### Claim 19 (depends on 14) The method of claim 14, further comprising: labeling a first subset of segments of audio data of the one or more audio requests as being from a first individual that is a party to a conversation between the first individual and a second individual; and labeling a second subset of the segments of audio data as being from the second individual that is another party to the conversation. ### Claim 20 (depends on 19) The method of claim 19, wherein identifying the at least one theme comprises identifying, via the natural language processing model, a first theme indicating the first individual based on the labeling corresponding to the first and second individuals, the method further comprising: in response to detecting the object based on the one or more audio requests, generating, coincident in time to the object appearing in view of the camera, a notification for the second individual that is related to the first theme indicating the first individual.