# US 2022/0036554 A1 · System and method for supporting the emotional and physical health of a user > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2022/0036554 A1 - Document ID: US20220036554A1 - Status: Published application - Kind: A1 - Application number: 17/390,712 - Filed: 2021-07-30 - Published: 2022-02-03 - Earliest claimed: 2020-08-03 - Assignees: Healthcare Integrated Technologies, Inc. - Applicants: Healthcare Integrated Technologies Inc. - Inventors: Kenneth M. GREENWOOD; Scott Michael BORUFF; Jurgen K. VOLLRATH - Web page: https://patentgrep.com/patent/US20220036554A1 ## CPC - G06T 7/0014: {using an image reference approach} - G06V 40/174: {Facial expression recognition} - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G10L 25/63: for estimating an emotional state - G10L 25/66: for extracting parameters related to health condition (detecting or measuring for diagnostic purposes A61B5/00) - G06T 2207/20081: Training; Learning - G10L 25/30: using neural networks - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract A method and system for improving the well-being of a user, comprises an artificial intelligence system coupled to a microphone and video camera for capturing verbal and visual input from the user, and a speaker for engaging the user in conversation to provide virtual companionship and to assess the user's physiological and emotional demeanor. ## Claims ### Claim 1 A system for improving the emotional and physical health of a user, comprising a microphone for capturing sound, a speaker for uttering verbal responses, a camera, a memory encoded with machine-readable code, and a processor connected with the memory and configured by logic defined by the machine-readable code to identify one or more emotional or physiological states of a user based on image data received from the camera and voice data received from the microphone, and comparing these to previously captured image data and voice prints, obtained from one or more of third parties, and the user, using an artificial intelligence (AI) network to perform analytics, wherein corroboration of a user's emotional or physiological state is provided by correlations between emotional and physiological states identified from the voice data, and emotional and physiological states identified from the images data. ### Claim 2 (depends on 1) The system of claim 1, wherein the AI network uses image data and voice data received from the user that have been corroborated against each other for correlations in the identified emotional or physiological states, to refine the captured image data and voice prints as learning data for the AI network. ### Claim 3 (depends on 1) The system of claim 1, wherein the voice data is analyzed for one or more of intonation, modulation, voice patterns, volume, pitch, pauses, speed of speech, slurring, time between words, choice of words, and non-verbal utterances. ### Claim 4 (depends on 1) The system of claim 1, wherein the image data is analyzed for body language and for facial expressions, which include one or more of the configuration of the mouth, the creases formed around the mouth, creases along the forehead and around the eyes, the state of the eyes, and the dilation of the pupils, compared to a relaxed facial expression of the user. ### Claim 5 (depends on 1) The system of claim 1, wherein the machine-readable code defines logic configured to generate verbal outputs via the speaker in response to emotional and physiological states identified from the captured voice data and image data, in order to capture additional verbal data and image data to further corroborate the identified emotional and physiological states. ### Claim 6 (depends on 1) The system of claim 1, wherein the machine-readable code includes logic that takes the identified emotional or physiological state of the user to engage the user in conversation or adjust the environment of the user, wherein the environment includes one or more of lighting, background sound or music, and temperature. ### Claim 7 (depends on 6) The system of claim 6, wherein engaging the user in conversation includes making suggestions or posing questions. ### Claim 8 (depends on 7) The system of claim 7, wherein the machine-readable code is arranged to define logic to initiate communication with at least one of, administrative staff responsible for the user, a physician or medical facility associated with the user, an emergency response entity, or a family member or emergency contact associated with the user. ### Claim 9 (depends on 8) The system of claim 8, further comprising a database for maintaining user information, and contact details of people associated with a user. ### Claim 10 (depends on 9) The system of claim 9, wherein the database includes the previously captured image data and voice prints. ### Claim 11 (depends on 10) The system of claim 10, wherein the database includes sound files of non-verbal data associated with emergencies that define trigger events, and the machine-readable code includes logic for comparing non-verbal sounds received from the microphone to the non-verbal sound files to identify an emergency and define a trigger event. ### Claim 12 (depends on 11) The system of claim 11, wherein making suggestions or posing questions includes the option to connect the user by voice or video to a third party. ### Claim 13 (depends on 12) The system of claim 12, wherein the making of suggestions or posing of questions for purposes of connecting the user to a third party includes making a suggestion to a user of a third-party to connect to, or receiving information from the user about which third-party to connect to. ### Claim 14 (depends on 1) The system of claim 1, further comprising a robot that includes the speaker and is connected to the processor to be responsive to the outputs of the processor, the robot being configured through outputs from the processor to empathize and provide support through movement and voice response. ### Claim 15 (depends on 14) The system of claim 14, wherein the robot is configured to move closer to the user in order to provide additional emotional support, capture low-volume verbal inputs, or capture a closer image of the user. ### Claim 16 A method of improving the emotional and physiological health of a user, comprising monitoring the user with sensors and responding to the user's needs based on verbal signals and images captured from the user via the sensors, wherein the verbal signals and images are compared in time to corroborate each other in order to identify and validate one or more emotional or physiological states of the user. ### Claim 17 (depends on 16) The method of claim 16, further comprising verbally interacting with the user in response to verbal signals and images captured and based on voice prompts uttered to the robot by the user. ### Claim 18 (depends on 17) The method of claim 17, comprising building up a physiological and emotional profile of the user based on the sensor data and interactions with the user. ### Claim 19 (depends on 15) The method of claim 15, comprising identifying and validating emotional or physiological emergencies and notifying at least one third party in the event of a validated emergency.