# US 12,602,901 B2 · Multimodal artificial intelligence-based communication systems > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,901 B2 - Document ID: US12602901B2 - Status: Granted - Kind: B2 - Application number: 18/504,758 - Filed: 2023-11-08 - Published: 2025-05-08 - Granted: 2026-04-14 - Earliest claimed: 2023-11-08 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Yuan Yuan Ding; Yang Liu; Jing Zhang; Yu Pan; Shi Yun Liang - Web page: https://patentgrep.com/patent/US12602901B2 ## CPC - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/82: using neural networks - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Methods, apparatus, and computer program products for multimodal artificial intelligence-based communication systems are provided herein. A computer-implemented method includes generating, using a first set of one or more artificial intelligence techniques, identifying information for one or more objects detected in image data associated with at least one user query; generating at least one updated version of the at least one user query by processing, using a second set of one or more artificial intelligence techniques, at least a portion of the at least one user query in conjunction with at least a portion of the identifying information for the one or more objects; generating at least one response to the at least one updated version of the at least one user query; and performing one or more automated actions based at least in part on the at least one response. ## Claims ### Claim 1 A system comprising: a memory configured to store program instructions; and a processor operatively coupled to the memory to execute the program instructions to: generate, using a first set of one or more artificial intelligence techniques, identifying information for one or more objects detected in image data associated with at least one user query; generate at least one updated version of the at least one user query by processing, using a second set of one or more artificial intelligence techniques, at least a portion of the at least one user query in conjunction with at least a portion of the identifying information for the one or more objects, wherein generating the at least one updated version of the at least one user query comprises processing, using at least one topic aware sequence-to-sequence (TA-Seq2Seg) model, the at least a portion of the at least one user query in conjunction with the at least a portion of the identifying information for the one or more objects; generate at least one response to the at least one updated version of the at least one user query; and perform one or more automated actions based at least in part on the at least one response. ### Claim 2 (depends on 1) The system of claim 1, wherein generating identifying information for one or more objects comprises processing at least a portion of the image data associated with the at least one user query using one or more neural networks. ### Claim 3 (depends on 2) The system of claim 2, wherein processing at least a portion of the image data associated with at least one user query using one or more neural networks comprises processing the at least a portion of the image data associated with the at least one user query using at least one fast region-based convolutional neural network (Fast R-CNN) model. ### Claim 4 (depends on 2) The system of claim 2, wherein processing at least a portion of the image data associated with at least one user query using one or more neural networks comprises processing the at least a portion of the image data associated with the at least one user query using at least one You Only Look Once (YOLO) model. ### Claim 5 (depends on 2) The system of claim 2, wherein processing at least a portion of the image data associated with at least one user query using one or more neural networks comprises using at least one pretrained neural network which has been additionally fine-tuned using one or more datasets relevant to one or more contexts related to the at least one user query. ### Claim 6 (depends on 1) The system of claim 1, wherein generating at least one updated version of the at least one user query comprises supplementing at least a portion of the at least one user query with at least one description of environmental information, derived from the at least a portion of the identifying information for the one or more objects, relevant to the at least one user query. ### Claim 7 (depends on 1) The system of claim 1, wherein performing the one or more automated actions comprises automatically outputting the at least one response to at least one user device associated with the at least one response. ### Claim 8 (depends on 1) The system of claim 1, wherein performing the one or more automated actions comprises automatically training, using feedback related to the at least one response, one or more of at least a portion of the first set of one or more artificial intelligence techniques and at least a portion of the second set of one or more artificial intelligence techniques. ### Claim 9 (depends on 1) The system of claim 1, wherein generating the at least one response comprises processing the at least one updated version of the at least one user query using one or more artificial intelligence-based chatbots. ### Claim 10 A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computing device to cause the computing device to: generate, using a first set of one or more artificial intelligence techniques, identifying information for one or more objects detected in image data associated with at least one user query; generate at least one updated version of the at least one user query by processing, using a second set of one or more artificial intelligence techniques, at least a portion of the at least one user query in conjunction with at least a portion of the identifying information for the one or more objects, wherein generating the at least one updated version of the at least one user query comprises processing, using at least one topic aware sequence-to-sequence (TA-Seq2Seq) model, the at least a portion of the at least one user query in conjunction with the at least a portion of the identifying information for the one or more objects; generate at least one response to the at least one updated version of the at least one user query; and perform one or more automated actions based at least in part on the at least one response. ### Claim 11 (depends on 10) The computer program product of claim 10, wherein generating identifying information for one or more objects comprises processing at least a portion of the image data associated with the at least one user query using one or more neural networks. ### Claim 12 (depends on 10) The computer program product of claim 10, wherein generating at least one updated version of the at least one user query comprises supplementing at least a portion of the at least one user query with at least one description of environmental information, derived from the at least a portion of the identifying information for the one or more objects, relevant to the at least one user query. ### Claim 13 (depends on 10) The computer program product of claim 10, wherein performing the one or more automated actions comprises automatically outputting the at least one response to at least one user device associated with the at least one response. ### Claim 14 A computer-implemented method comprising: generating, using a first set of one or more artificial intelligence techniques, identifying information for one or more objects detected in image data associated with at least one user query; generating at least one updated version of the at least one user query by processing, using a second set of one or more artificial intelligence techniques, at least a portion of the at least one user query in conjunction with at least a portion of the identifying information for the one or more objects, wherein generating the at least one updated version of the at least one user query comprises processing, using at least one topic aware sequence-to-sequence (TA-Seq2Seq) model, the at least a portion of the at least one user query in conjunction with the at least a portion of the identifying information for the one or more objects; generating at least one response to the at least one updated version of the at least one user query; and performing one or more automated actions based at least in part on the at least one response; wherein the method is carried out by at least one computing device. ### Claim 15 (depends on 14) The computer-implemented method of claim 14, wherein generating identifying information for one or more objects comprises processing at least a portion of the image data associated with the at least one user query using one or more neural networks. ### Claim 16 (depends on 14) The computer-implemented method of claim 14, wherein generating at least one updated version of the at least one user query comprises supplementing at least a portion of the at least one user query with at least one description of environmental information, derived from the at least a portion of the identifying information for the one or more objects, relevant to the at least one user query. ### Claim 17 (depends on 14) The computer-implemented method of claim 14, wherein software implementing the method is provided as a service in a cloud environment. ### Claim 18 (depends on 14) The computer-implemented method of claim 14, wherein performing the one or more automated actions comprises automatically outputting the at least one response to at least one user device associated with the at least one response. ### Claim 19 (depends on 14) The computer-implemented method of claim 14, wherein performing the one or more automated actions comprises automatically training, using feedback related to the at least one response, one or more of at least a portion of the first set of one or more artificial intelligence techniques and at least a portion of the second set of one or more artificial intelligence techniques. ### Claim 20 (depends on 14) The computer-implemented method of claim 14, wherein generating the at least one response comprises processing the at least one updated version of the at least one user query using one or more artificial intelligence-based chatbots.