# US 2017/0286755 A1 · Facebot

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0286755 A1
- Document ID: US20170286755A1
- Status: Published application
- Kind: A1
- Application number: 15/402,956
- Filed: 2017-01-10
- Published: 2017-10-05
- Earliest claimed: 2016-03-30
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Michael M.O. Kaletsky
- Web page: https://patentgrep.com/patent/US20170286755A1

## CPC

- G06V 40/172: {Classification, e.g. identification}
- G06F 3/015: {Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection}
- G06F 40/166: Editing, e.g. inserting or deleting
- G06F 40/30: Semantic analysis
- G06F 40/56: Natural language generation
- G06V 40/161: {Detection; Localisation; Normalisation}
- G06V 40/174: {Facial expression recognition}
- G06V 40/175: {Static expression}
- G06V 40/40: Spoof detection, e.g. liveness detection
- H04L 51/02: using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
- H04L 51/10: Multimedia information
- G06F 2203/011: Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

A software agent configured to perform operations of: receiving from a human user an image in a message within a communication event established between a user terminal associated with the human user and the software agent; transmitting image data from the image to at least three image processing service components, including: (i) first image processing service component for detecting physical characteristic of a facial image and providing raw data pertaining to physical characteristics; (ii) a second image processing service component for detecting emotional characteristics of a facial image and providing raw data pertaining to emotional characteristics; and (iii) a third image processing service component to detect whether an image is a facial image or a non-facial image and providing a probability indication; processing the raw data from the first and second image processing service and the probability indication from the third image processing service to generate humanly readable text for incorporation in a response message; transmitting the response message in the communication event to the user terminal for display to a human user at the user terminal.

## Claims

### Claim 1

A computer program product comprising a software agent stored on computer readable storage, the software agent being configured when run on one or more processing devices to perform operations of:
receiving from a human user an image in a message within a communication event established between a user terminal associated with the human user and the software agent;
transmitting image data from the image to at least two image processing service components, including: (i) first image processing service component for detecting physical characteristics of a facial image and providing raw data pertaining to physical characteristics; and (ii) a second image processing service component for detecting emotional characteristics of a facial image and providing raw data pertaining to emotional characteristics;
processing the raw data from the first and second image processing service to generate humanly readable text for incorporation in a response message;
transmitting the response message in the communication event to the user terminal for display to a human user at the user terminal.

### Claim 2 (depends on 1)

A computer program product according to claim 1, wherein the method comprises supplying the image data to (iii) a third image processing service component to detect whether an image is a facial image or a non-facial image and receiving a probability indication from the third image processing service, wherein the image data is supplied to the first and second image processing services only if the probability indication indicates a probability above a threshold that the image is a facial image; and wherein said processing comprises processing the raw data from the first and second image processing service and the probability indication from the third image processing service to generate the humanly readable text for incorporation in the response message.

### Claim 3 (depends on 2)

A computer program product according to claim 2, wherein if the probability indication is below the threshold, the method comprises supplying the image data to a fourth image processing service which detects the nature of the image.

### Claim 4 (depends on 1)

A computer program product according to claim 1, wherein the step of processing the raw data comprises extracting characteristics from the raw data based on categories of expected characteristics, and accessing a library of characteristic terms to generate a humanly readable phrase or word matching the characteristic.

### Claim 5 (depends on 4)

A computer program product according to claim 4, wherein the method comprises accessing a semantics library of interconnecting phrases to generate the humanly readable text from the phrase or word.

### Claim 6 (depends on 1)

A computer program product according to claim 1, wherein the method comprises establishing the communication event in response to receiving a request from the user terminal.

### Claim 7 (depends on 1)

A computer program product according to claim 1, wherein the step of receiving the image data comprises extracting the image data from the received message.

### Claim 8 (depends on 1)

A computer program product according to claim 1, wherein the step of extracting the image data comprises extracting an indication from the message and using the indication to access a remote source of the image data.

### Claim 9

A computer device comprising a processor, a computer storage access component and a communications interface, the processor configured to run a computer program stored on storage accessed by the storage access component comprising a software agent, the software agent being configured when run on one or more processing devices to perform operations of:
receiving via the communications interface from a human user an image in a message within a communication event established between a user terminal associated with the human user and the software agent;
transmitting image data from the image to at least two image processing service components, including: (i) first image processing service component for detecting physical characteristics of a facial image and providing raw data pertaining to physical characteristics; and (ii) a second image processing service component for detecting emotional characteristics of a facial image and providing raw data pertaining to emotional characteristics;
processing the raw data from the first and second image processing service to generate humanly readable text for incorporation in a response message;
transmitting the response message in the communication event to the user terminal for display to a human user at the user terminal.

### Claim 10 (depends on 9)

A computer device according to claim 9, wherein the method comprises supplying the image data to (iii) a third image processing service component to detect whether an image is a facial image or a non-facial image and receiving a probability indication from the third image processing service, wherein the image data is supplied to the first and second image processing services only if the probability indication indicates a probability above a threshold that the image is a facial image; and wherein said processing comprises processing the raw data from the first and second image processing service and the probability indication from the third image processing service to generate the humanly readable text for incorporation in the response message.

### Claim 11 (depends on 10)

A computer device according to claim 10, wherein if the probability indication is below the threshold, the method comprises supplying the image data to a fourth image processing service which detects the nature of the image.

### Claim 12 (depends on 9)

A computer device according to claim 9, wherein the step of processing the raw data comprises extracting characteristics from the raw data based on categories of expected characteristics, and accessing a library of characteristic terms to generate a humanly readable phrase or word matching the characteristic, the library accessible via the storage access component.

### Claim 13 (depends on 12)

A computer device according to claim 12, wherein the method comprises accessing a semantics library of interconnecting phrases to generate the humanly readable text from the phrase or word.

### Claim 14 (depends on 9)

A computer device according to claim 9, wherein the method comprises establishing the communication event in response to receiving a request from the user terminal.

### Claim 15 (depends on 9)

A computer device according to claim 9, wherein the step of receiving the image data comprises extracting the image data from the received message.

### Claim 16 (depends on 9)

A computer device according to claim 9, wherein the step of extracting the image data comprises extracting an indication from the message and using the indication to access a remote source of the image data.

### Claim 17

A method implemented by a computer program product comprising a software agent stored on computer readable storage, the method comprising:
receiving from a human user an image in a message within a communication event established between a user terminal associated with the human user and the software agent;
transmitting image data from the image to at least two image processing service components, including: (i) first image processing service component for detecting physical characteristics of a facial image and providing raw data pertaining to physical characteristics; and (ii) a second image processing service component for detecting emotional characteristics of a facial image and providing raw data pertaining to emotional characteristics;
processing the raw data from the first and second image processing service to generate humanly readable text for incorporation in a response message;
transmitting the response message in the communication event to the user terminal for display to a human user at the user terminal.

### Claim 18 (depends on 17)

A method according to claim 17, wherein the method comprises supplying the image data to (iii) a third image processing service component to detect whether an image is a facial image or a non-facial image and receiving a probability indication from the third image processing service, wherein the image data is supplied to the first and second image processing services only if the probability indication indicates a probability above a threshold that the image is a facial image; and wherein said processing comprises processing the raw data from the first and second image processing service and the probability indication from the third image processing service to generate the humanly readable text for incorporation in the response message.

### Claim 19 (depends on 18)

A method according to claim 18, wherein if the probability indication is below the threshold, the method comprises supplying the image data to a fourth image processing service which detects the nature of the image.

### Claim 20 (depends on 17)

A method according to claim 17, wherein the step of processing the raw data comprises extracting characteristics from the raw data based on categories of expected characteristics, and accessing a library of characteristic terms to generate a humanly readable phrase or word matching the characteristic.
