# US 12,008,310 B2 · Automated clinical documentation system and method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,310 B2 - Document ID: US12008310B2 - Status: Granted - Kind: B2 - Application number: 17/846,355 - Filed: 2022-06-22 - Published: 2022-10-06 - Granted: 2024-06-11 - Earliest claimed: 2017-08-10 - Assignees: Microsoft Licensing Technology, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Donald E. Owen; Mehmet Mert Öz; Garret N. Erskine - Web page: https://patentgrep.com/patent/US12008310B2 ## CPC - G06F 40/174: Form filling; Merging - A61B 5/7405: {using sound} - G06F 3/16: Sound input; Sound output (speech processing G10L) - G06F 16/637: {Administration of user profiles, e.g. generation, initialization, adaptation or distribution} - G06F 16/685: {using automatically derived transcript of audio data, e.g. lyrics (speech recognition G10L15/00)} - G06F 16/904: Browsing; Visualisation therefor (for navigating the web G06F16/954; browsing optimisation for the web G06F16/957) - G06F 18/25: Fusion techniques - G06F 21/6245: {Protecting personal data, e.g. for financial or medical purposes} - G06F 40/30: Semantic analysis - G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) - G06K 19/07762: {the adhering arrangement making the record carrier wearable, e.g. having the form of a ring, watch, glove or bracelet (record carriers for insertion in the human body for medical purposes A61B90/00; record carriers adapted for attachment to animals A01K11/00)} - G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] - G06T 7/00: Image analysis - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition} - G06V 40/16: Human faces, e.g. facial parts, sketches or expressions - G06V 40/172: {Classification, e.g. identification} - G06V 40/23: {Recognition of whole body movements, e.g. for sport training} - G10L 15/08: Speech classification or search - G10L 17/00: Speaker identification or verification techniques - G10L 21/0232: Processing in the frequency domain - G10L 25/48: specially adapted for particular use - G11B 27/10: Indexing; Addressing; Timing or synchronising; Measuring tape travel - G16B 50/00: ICT programming tools or database systems specially adapted for bioinformatics - G16H 10/20: for electronic clinical trials or questionnaires - G16H 10/60: for patient-specific data, e.g. for electronic patient records - G16H 15/00: ICT specially adapted for medical reports, e.g. generation or transmission thereof - G16H 30/00: ICT specially adapted for the handling or processing of medical images (computed tomography A61B6/03) - G16H 30/20: for handling medical images, e.g. DICOM, HL7 or PACS - G16H 30/40: for processing medical images, e.g. editing - G16H 40/20: for the management or administration of healthcare resources or facilities, e.g. managing hospital staff or surgery rooms - G16H 40/60: for the operation of medical equipment or devices - G16H 40/63: for local operation - G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems - G16H 50/30: for calculating health indices; for individual health risk assessment - G16H 80/00: ICT specially adapted for facilitating communication between medical practitioners or patients, e.g. for collaborative diagnosis, therapy or health monitoring - G16Y 20/00: Information sensed or collected by the things - H04L 51/02: using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages - H04L 51/222: using geographical location information, e.g. messages transmitted or received in proximity of a certain spot or area - H04N 7/183: {for receiving images from a single remote source} - H04R 1/326: {for microphones (H04R1/34 and H04R1/40 take precedence)} - H04R 3/005: {for combining the signals of two or more microphones (specially adapted for hearing aids H04R25/407)} - H04R 3/12: for distributing signals to two or more loudspeakers {(specially adapted for hearing aids H04R25/407)} - G06N 20/00: Machine learning - G06T 2207/10024: Color image - G06T 2207/10044: Radar image - G06T 2207/10048: Infrared image - G06T 2207/10116: X-ray image - G06T 2207/10132: Ultrasound image - G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning} - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - G10L 21/0272: Voice signal separating - G10L 2021/02082: {the noise being echo, reverberation of the speech} - G10L 2021/02166: {Microphone arrays; Beamforming} - H04N 7/181: {for receiving images from a plurality of remote sources} - H04R 1/406: {microphones} - H04R 3/02: for preventing acoustic reaction {, i.e. acoustic oscillatory feedback (specially adapted for hearing aids H04R25/453)} - H04R 2420/07: Applications of wireless loudspeakers or wireless microphones - H04S 2400/15: Aspects of sound capture and related signal processing for recording or reproduction - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G06K: GRAPHICAL DATA READING (image or video recognition or understanding G06V); PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R) - G16B: BIOINFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR GENETIC OR PROTEIN-RELATED DATA PROCESSING IN COMPUTATIONAL MOLECULAR BIOLOGY - G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA - G16Y: INFORMATION AND COMMUNICATION TECHNOLOGY SPECIALLY ADAPTED FOR THE INTERNET OF THINGS [IoT] - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K) - H04S: STEREOPHONIC SYSTEMS ## Abstract A method, computer program product, and computing system for compartmentalizing a virtual assistant is executed on a computing device and includes obtaining encounter information via a compartmentalized virtual assistant during a patient encounter, wherein the compartmentalized virtual assistant includes a core functionality module. One or more additional functionalities are added to the compartmentalized virtual assistant on an as-needed basis. ## Claims ### Claim 1 A computer-implemented method for compartmentalizing a virtual assistant, executed on a computing device, comprising: obtaining encounter information via a compartmentalized virtual assistant during a patient encounter, wherein the compartmentalized virtual assistant includes a core functionality module that verbally interacts with an encounter participant of the patient encounter, wherein the core functionality module verbally interacts with at least one of a patient and a medical professional as the encounter participant during a portion of the patient encounter; determining, by the compartmentalized virtual assistant, that a medical treatment topic was mentioned by at least one of the patient and the medical professional while verbally interacting with the virtual assistant during the portion of the patient encounter; downloading and activating one or more additional functionalities associated with the medical treatment topic to the compartmentalized virtual assistant during the patient encounter based upon, at least in part, the compartmentalized virtual assistant determining that the medical treatment topic was mentioned during the portion of the patient encounter, wherein at least one of the one or more additional functionalities added and activated to the compartmentalized virtual assistant during the patient encounter includes an adverse interaction functionality configured to identify any potential adverse interactions associated with the medical treatment topic, wherein the one or more additional functionalities associated with the medical treatment topic has not been downloaded and activated prior to the compartmentalized virtual assistant determining that one of the medical treatment topic was mentioned, and wherein the one or more additional functionalities associated with the medical treatment topic is downloaded and activated after the compartmentalized virtual assistant determines that the medical treatment topic was mentioned; and providing feedback to the medical professional from the compartmentalized virtual assistant based upon the one or more additional functionalities associated with the medical treatment topic and the medical treatment topic being mentioned. ### Claim 2 (depends on 1) The computer-implemented method of claim 1 wherein downloading one or more additional functionalities to the compartmentalized virtual assistant on an as-needed basis includes: loading one or more additional functionality modules for the compartmentalized virtual assistant when needed to effectuate the additional functionalities. ### Claim 3 (depends on 1) The computer-implemented method of claim 1 further comprising: removing one or more existing functionalities from the compartmentalized virtual assistant on an as-needed basis. ### Claim 4 (depends on 3) The computer-implemented method of claim 3 wherein removing one or more existing functionalities from the compartmentalized virtual assistant on an as-needed basis includes: unloading one or more existing functionality modules of the compartmentalized virtual assistant when no longer needed to effectuate the existing functionalities. ### Claim 5 (depends on 1) The computer-implemented method of claim 1 further comprising: processing the encounter information to generate an encounter transcript. ### Claim 6 (depends on 5) The computer-implemented method of claim 5 further comprising: processing at least a portion of the encounter transcript to populate at least a portion of a medical record associated with the patient encounter. ### Claim 7 (depends on 1) The computer-implemented method of claim 1 wherein obtaining encounter information via a compartmentalized virtual assistant during a patient encounter includes: audibly obtaining encounter information via the compartmentalized virtual assistant during the patient encounter. ### Claim 8 A computer program product residing on a non-transitory computer readable medium having a plurality of instructions stored thereon which, when executed by a processor, cause the processor to perform operations comprising: obtaining encounter information via a compartmentalized virtual assistant during a patient encounter, wherein the compartmentalized virtual assistant includes a core functionality module that verbally interacts with an encounter participant of the patient encounter, wherein the core functionality module verbally interacts with at least one of a patient and a medical professional as the encounter participant during a portion of the patient encounter; determining, by the compartmentalized virtual assistant, that a medical treatment topic was mentioned by at least one of the patient and the medical professional while verbally interacting with the virtual assistant during the portion of the patient encounter; downloading and activating one or more additional functionalities associated with the medical treatment topic to the compartmentalized virtual assistant during the patient encounter based upon, at least in part, the compartmentalized virtual assistant determining that the medical treatment topic was mentioned during the portion of the patient encounter, wherein at least one of the one or more additional functionalities added and activated to the compartmentalized virtual assistant during the patient encounter includes an adverse interaction functionality configured to identify any potential adverse interactions associated with the medical treatment topic, wherein the one or more additional functionalities associated with the medical treatment topic has not been downloaded and activated prior to the compartmentalized virtual assistant determining that one of the medical treatment topic was mentioned, and wherein the one or more additional functionalities associated with the medical treatment topic is downloaded and activated after the compartmentalized virtual assistant determines that the medical treatment topic was mentioned; and providing feedback to the medical professional from the compartmentalized virtual assistant based upon the one or more additional functionalities associated with the medical treatment topic and the medical treatment topic being mentioned. ### Claim 9 (depends on 8) The computer program product of claim 8 wherein downloading one or more additional functionalities to the compartmentalized virtual assistant on an as-needed basis includes: loading one or more additional functionality modules for the compartmentalized virtual assistant when needed to effectuate the additional functionalities. ### Claim 10 (depends on 8) The computer program product of claim 8 further comprising: removing one or more existing functionalities from the compartmentalized virtual assistant on an as-needed basis. ### Claim 11 (depends on 10) The computer program product of claim 10 wherein removing one or more existing functionalities from the compartmentalized virtual assistant on an as-needed basis includes: unloading one or more existing functionality modules of the compartmentalized virtual assistant when no longer needed to effectuate the existing functionalities. ### Claim 12 (depends on 8) The computer program product of claim 8 further comprising: processing the encounter information to generate an encounter transcript. ### Claim 13 (depends on 12) The computer program product of claim 12 further comprising: processing at least a portion of the encounter transcript to populate at least a portion of a medical record associated with the patient encounter. ### Claim 14 (depends on 8) The computer program product of claim 8 wherein obtaining encounter information via a compartmentalized virtual assistant during a patient encounter includes: audibly obtaining encounter information via the compartmentalized virtual assistant during the patient encounter. ### Claim 15 A computing system including a processor and memory configured to perform operations comprising: obtaining encounter information via a compartmentalized virtual assistant during a patient encounter, wherein the compartmentalized virtual assistant includes a core functionality module that verbally interacts with an encounter participant of the patient encounter, wherein the core functionality module verbally interacts with at least one of a patient and a medical professional as the encounter participant during a portion of the patient encounter; determining, by the compartmentalized virtual assistant, that a medical treatment topic was mentioned by at least one of the patient and the medical professional while verbally interacting with the virtual assistant during the portion of the patient encounter; downloading and activating one or more additional functionalities associated with the medical treatment topic to the compartmentalized virtual assistant during the patient encounter based upon, at least in part, the compartmentalized virtual assistant determining that the medical treatment topic was mentioned during the portion of the patient encounter, wherein at least one of the one or more additional functionalities added and activated to the compartmentalized virtual assistant during the patient encounter includes an adverse interaction functionality configured to identify any potential adverse interactions associated with the medical treatment topic, wherein the one or more additional functionalities associated with the medical treatment topic has not been downloaded and activated prior to the compartmentalized virtual assistant determining that one of the medical treatment topic was mentioned, and wherein the one or more additional functionalities associated with the medical treatment topic is downloaded and activated after the compartmentalized virtual assistant determines that the medical treatment topic was mentioned; and providing feedback to the medical professional from the compartmentalized virtual assistant based upon the one or more additional functionalities associated with the medical treatment topic and the medical treatment topic being mentioned. ### Claim 16 (depends on 15) The computing system of claim 15 wherein downloading one or more additional functionalities to the compartmentalized virtual assistant on an as-needed basis includes: loading one or more additional functionality modules for the compartmentalized virtual assistant when needed to effectuate the additional functionalities. ### Claim 17 (depends on 15) The computing system of claim 15 further comprising: removing one or more existing functionalities from the compartmentalized virtual assistant on an as-needed basis. ### Claim 18 (depends on 17) The computing system of claim 17 wherein removing one or more existing functionalities from the compartmentalized virtual assistant on an as-needed basis includes: unloading one or more existing functionality modules of the compartmentalized virtual assistant when no longer needed to effectuate the existing functionalities. ### Claim 19 (depends on 15) The computing system of claim 15 further comprising: processing the encounter information to generate an encounter transcript. ### Claim 20 (depends on 19) The computing system of claim 19 further comprising: processing at least a portion of the encounter transcript to populate at least a portion of a medical record associated with the patient encounter. ### Claim 21 (depends on 15) The computing system of claim 15 wherein obtaining encounter information via a compartmentalized virtual assistant during a patient encounter includes: audibly obtaining encounter information via the compartmentalized virtual assistant during the patient encounter.