# US 2019/0244012 A1 · Automatic image classification in electronic communications

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0244012 A1
- Document ID: US20190244012A1
- Status: Published application
- Kind: A1
- Application number: 15/887,869
- Filed: 2018-02-02
- Published: 2019-08-08
- Earliest claimed: 2018-02-02
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Kathleen Rae Thompson; Hayley Steplyk; Joseph Patrick Masterson
- Web page: https://patentgrep.com/patent/US20190244012A1

## CPC

- G06V 40/172: {Classification, e.g. identification}
- G06F 16/51: Indexing; Data structures therefor; Storage structures
- G06F 16/5838: {using colour}
- G06Q 10/107: Computer-aided management of electronic mailing [e-mailing]
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- H04L 51/08: Annexed information, e.g. attachments
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

Techniques of automatic image classification and modification in computing systems are disclosed herein. In one embodiment, a method includes scanning an inbox on email servers for emails containing image files. Upon detecting that an email in the inbox contains an image file, the method includes retrieving an identification photo of a user from a data store. The method also includes determining, via facial recognition, whether the image file in the email contains at least a partial image of the user based on the retrieved identification photo. In response to determining that the image file in the email contains at least a partial image of the user, a metadata value is inserted into the image file indicating that the image file contains at least a partial image of the user before the image file is stored in the inbox on the one or more email servers.

## Claims

### Claim 1

A method for automatic image classification in a computing system having one or more email servers interconnected to client devices by a computer network, the method comprising:
scanning an inbox on the one or more email servers for emails containing one or more image files; and
upon detecting that an email in the inbox contains an image file,
retrieving, via the computer network, an identification photo of a user from a data store containing entries of user identifications and corresponding identification photos;
determining, via facial recognition, whether the image file in the email contains at least a partial image of the user based on the retrieved identification photo of the user; and
in response to determining that the image file in the email contains at least a partial image of the user, inserting a metadata value into the image file indicating that the image file contains at least a partial image of the user and storing the image file along with the inserted metadata value in the inbox on the one or more email servers.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
in response to determining that the image file in the email contains at least a partial image of the user,
generating or modifying an image gallery containing image files individually having at least a partial image of the user; and
transmitting, via the computer network, the generated or modified image galley to another user having access to the inbox on the one or more email servers.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
in response to determining that the image file in the email contains at least a partial image of the user, transmitting, via the computer network, a copy of the image file along with the inserted metadata value to a client device corresponding to another user having access to the inbox on the one or more email servers.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
in response to determining that the image file in the email contains at least a partial image of the user, updating an image search index to indicate that the stored image file contains at least a partial image of the user.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
upon detecting that an email in the inbox contains an image file, retrieving, via the computer network, another image file known to contain an image of the user from the inbox on the one or more email servers; and
wherein determining whether the image file in the email contains at least a partial image of the user includes determining, via facial recognition, whether the image file in the email contains at least a partial image of the user based on the image of the user in the retrieved another image file.

### Claim 6 (depends on 1)

The method of claim 1 wherein determining, via facial recognition, whether the image file in the email contains at least a partial image of the user includes:
extracting one or more landmarks from the retrieved identification photo of the user's face; and
determining whether the extracted one or more landmarks are present in the image file.

### Claim 7 (depends on 1)

The method of claim 1 wherein determining, via facial recognition, whether the image file in the email contains at least a partial image of the user includes:
extracting one or more landmarks from the retrieved identification photo of the user's face;
determining whether the extracted one or more landmarks are present in the image file; and
indicating that the image file contains at least a partial image of the user in response to determining that the extracted one or more landmarks are present in the image file.

### Claim 8 (depends on 1)

The method of claim 1, further comprising in response to determining that the image file in the email contains at least a partial image of the user, repeating the retrieving and determining operations for additional identification photos of additional users.

### Claim 9 (depends on 1)

The method of claim 1 wherein scanning the inbox includes:
determining whether the email in the inbox contains an attachment;
in response to determining that the email in the inbox contains an attached, determining whether the attachment has a file extension indicating an image file; and
in response to determining that the attachment has a file extension indicating an image file, indicating that the email contains an image file.

### Claim 10

A computing device configured to be interconnected to one or more client devices by a computer network, the computing system comprising:
a processor; and
a memory operatively coupled to the processor, the memory containing instructions executable by the processor to cause the computing device to:
determining whether one or more image files are included as attachment to emails in an inbox corresponding to a user's email account; and
upon determining that one or more image files are included as attachment to the emails in the inbox,
determining, via facial recognition, whether the one or more image files individually contain at least a partial image of a person based on one or more facial landmarks of the person derived from an identification photo of the user; and
in response to determining that the one or more image file contain at least a partial image of the person, modify a metadata value of the one or more image files to indicate that the one or more image files contain at least a partial image of the person and storing the one or more image files along with the modified metadata value in the inbox corresponding to the user's email account.

### Claim 11 (depends on 10)

The computing device of claim 10 wherein the memory contains additional instructions executable by the processor to cause the computing device to:
in response to determining that the image file in the email contains at least a partial image of the user,
generate or modify an image gallery containing image files individually having at least a partial image of the user; and
transmit, via the computer network, the generated or modified image galley to another user having access to the inbox on the one or more email servers.

### Claim 12 (depends on 10)

The computing device of claim 10 wherein the memory contains additional instructions executable by the processor to cause the computing device to:
in response to determining that the image file in the email contains at least a partial image of the user, transmit, via the computer network, a copy of the image file along with the inserted metadata value to a client device corresponding to another user having access to the inbox on the one or more email servers.

### Claim 13 (depends on 10)

The computing device of claim 10 wherein the memory contains additional instructions executable by the processor to cause the computing device to:
in response to determining that the image file in the email contains at least a partial image of the user, update an image search index to indicate that the stored image file contains at least a partial image of the user.

### Claim 14 (depends on 10)

The computing device of claim 10 wherein the memory contains additional instructions executable by the processor to cause the computing device to:
upon detecting that an email in the inbox contains an image file, retrieve, via the computer network, another image file known to contain an image of the user from the inbox on the one or more email servers; and
wherein to determine whether the image file in the email contains at least a partial image of the user includes to determine, via facial recognition, whether the image file in the email contains at least a partial image of the user based on the image of the user in the retrieved another image file.

### Claim 15 (depends on 10)

The computing device of claim 10 wherein the memory contains additional instructions executable by the processor to cause the computing device to repeating the retrieving and determining operations for additional identification photos of additional users in response to determining that the image file in the email does not contain at least a partial image of the user.

### Claim 16

A method for automatic image classification in a computing system having one or more email servers interconnected to client devices by a computer network, the method comprising:
determining whether an image file is included as attachment to an email received in an inbox corresponding to a user's email account; and
in response to determining that an image file is included as attachment to the an email received in the inbox,
applying facial recognition to determine whether the image file contains at least a partial image of a person based on one or more features of the person derived from a profile or identification photo of the person or another image file previously identified as containing an image of the person; and
in response to determining that the image file contains at least a partial image of the person,
modifying a metadata value of the image file to indicate that the image file contains at least a partial image of the person; and
storing the image file along with the modified metadata value in the inbox corresponding to the user's email account.

### Claim 17 (depends on 16)

The method of claim 16, further comprising:
in response to determining that the image file in the email contains at least a partial image of the user,
generating or modifying an image gallery containing image files individually having at least a partial image of the user; and
transmitting, via the computer network, the generated or modified image galley to another user having access to the inbox on the one or more email servers.

### Claim 18 (depends on 16)

The method of claim 16, further comprising:
in response to determining that the image file in the email contains at least a partial image of the user, transmitting, via the computer network, a copy of the image file along with the inserted metadata value to a client device corresponding to another user having access to the inbox on the one or more email servers.

### Claim 19 (depends on 16)

The method of claim 16, further comprising:
in response to determining that the image file in the email contains at least a partial image of the user, updating an image search index to indicate that the stored image file contains at least a partial image of the user.

### Claim 20 (depends on 16)

The method of claim 16, further comprising:
upon detecting that an email in the inbox contains an image file, retrieving, via the computer network, another image file known to contain an image of the user from the inbox on the one or more email servers; and
wherein determining whether the image file in the email contains at least a partial image of the user includes determining, via facial recognition, whether the image file in the email contains at least a partial image of the user based on the image of the user in the retrieved another image file.