# US 2019/0244027 A1 · Photo processing method and apparatus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0244027 A1
- Document ID: US20190244027A1
- Status: Published application
- Kind: A1
- Application number: 16/386,152
- Filed: 2019-04-16
- Published: 2019-08-08
- Earliest claimed: 2015-11-25
- Assignees: Tencent Technology (Shenzhen) Company Limited
- Applicants: Tencent Technology (Shenzhen) Company Limited
- Inventors: Guofu TAN
- Web page: https://patentgrep.com/patent/US20190244027A1

## CPC

- G06F 16/51: Indexing; Data structures therefor; Storage structures
- G06F 3/14: Digital output to display device {; Cooperation and interconnection of the display device with other functional units}
- G06F 16/58: Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F 16/587: using geographical or spatial information, e.g. location
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 18/23211: with adaptive number of clusters
- G06V 10/763: {Non-hierarchical techniques, e.g. based on statistics of modelling distributions}
- G06V 20/35: {Categorising the entire scene, e.g. birthday party or wedding scene}
- G06V 40/161: {Detection; Localisation; Normalisation}
- G06V 40/164: {using holistic features}
- G06V 40/168: {Feature extraction; Face representation}
- G06V 40/173: {face re-identification, e.g. recognising unknown faces across different face tracks}
- H04N 1/00196: {Creation of a photo-montage, e.g. photoalbum (H04N1/0019 takes precedence)}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

The present disclosure discloses a photo processing method and an apparatus for grouping photos into photo albums based on facial recognition results. The method includes: performing face detection on multiple photos, to obtain a face image feature set, each face image feature in the face image feature set corresponding to one of the multiple photos; determining a face-level similarity for each pair of face image features in the face image feature set; determining a photo-level similarity between each pair of photos in the multiple photos in accordance with their associated face-level similarities; generating a photo set for each target photo in the multiple photos, wherein any photo-level similarity between the target photo and another photo in the photo set exceeds a predefined photo-level threshold; and generating a label for each photo set using photographing location and photographing time information associated with the photos in the photo set.

## Claims

### Claim 1

A method for generating a plurality of photo albums performed at a computing device having one or more processors and memory storing one or more programs to be executed by the one or more processors, the method comprising:
performing face detection on multiple photos to obtain a face image feature set, each face image feature in the face image feature set corresponding to one of the multiple photos;
determining a face-level similarity for each pair of face image features in the face image feature set;
determining a photo-level similarity between each pair of photos in the multiple photos in accordance with their associated face-level similarities, wherein the photo-level similarity between a respective pair of photos is a summation of the face-level similarity between the face image features of the respective pair of photos;
selecting, from the multiple photos, a photo set for each target photo, wherein any photo-level similarity between the target photo and another photo in the photo set exceeds a predefined photo-level threshold;
determining a photo set-level similarity between each pair of photo sets associated with the multiple photos, wherein a respective photo set-level similarity between two photo sets is determined according to face-level similarities corresponding to the two photo sets respectively;
combining at least two photo sets into one photo album in accordance with a determination that a face-level similarity between two face image features of two photos from the two photo sets is greater than a preset face-level similarity threshold and any photo set-level similarities of the two photo sets exceeds a predefined photo set-level threshold, respectively; and
generating one or more labels for each of a plurality of photo albums using photographing location and photographing time information associated with the photos within the photo album accordingly.

### Claim 2 (depends on 1)

The method according to claim 1, further comprising:
selecting labels of the at least two photo sets to be combined into a combined photo album; and
combining the identified labels into one label for the combined photo album.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the photo set-level similarity between two photo sets is defined in the following expression:
s
_
=
{
1
n
i

n
j

∑


s
ij

(
f
i
∈
A
i
,
f
j
∈
A
j
)
}
,
s representing the photo set-level similarity, Ai represents one of the two photo sets, Aj represents the other of the two photo sets, fi represents a face image feature that corresponds to a photo in the photo set Ai, fj represents a face image feature that corresponds to a photo in the photo set Aj, sij represents a similarity that corresponds to a pair of face image features formed by the face image feature fi and the face image feature fj, ni represents a quantity of photos in the photo set Ai, and nj represents a quantity of photos in the photo set Aj, i and j being two unequal natural numbers.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the photo-level similarity between a pair of photo sets is determined according to respective numbers of photos included in each photo set of the pair of photo sets.

### Claim 5 (depends on 1)

The method according to claim 1, further comprising:
displaying the one or more labels for each photo album to an end user; and
prompting the user to update the one or more labels for the photo album.

### Claim 6 (depends on 1)

The method according to claim 1, further comprising:
sorting photos in each output photo album according to an order of photographing time of the photos.

### Claim 7 (depends on 6)

The method according to claim 6, further comprising:
dividing photos in each output photo album into different photo subsets according to the order of photographing time of the photos; and
generating a sub-label for each photo subset according to the photographing time of the photos in the photo subset.

### Claim 8

A computing device, comprising:
one or more processors;
memory coupled to the one or more processors; and
a plurality of program instructions stored in the memory that, when executed by the one or more processors, cause the computing device to perform a plurality of operations including:
performing face detection on multiple photos to obtain a face image feature set, each face image feature in the face image feature set corresponding to one of the multiple photos;
determining a face-level similarity for each pair of face image features in the face image feature set;
determining a photo-level similarity between each pair of photos in the multiple photos in accordance with their associated face-level similarities, wherein the photo-level similarity between a respective pair of photos is a summation of the face-level similarity between the face image features of the respective pair of photos;
selecting, from the multiple photos, a photo set for each target photo, wherein any photo-level similarity between the target photo and another photo in the photo set exceeds a predefined photo-level threshold;
determining a photo set-level similarity between each pair of photo sets associated with the multiple photos, wherein a respective photo set-level similarity between two photo sets is determined according to face-level similarities corresponding to the two photo sets respectively;
combining at least two photo sets into one photo album in accordance with a determination that a face-level similarity between two face image features of two photos from the two photo sets is greater than a preset face-level similarity threshold and any photo set-level similarities of the two photo sets exceeds a predefined photo set-level threshold, respectively; and
generating one or more labels for each of a plurality of photo albums using photographing location and photographing time information associated with the photos within the photo album accordingly.

### Claim 9 (depends on 8)

The computing device according to claim 8, wherein the plurality of operations further comprise:
selecting labels of the at least two photo sets to be combined into a combined photo album; and
combining the identified labels into one label for the combined photo album.

### Claim 10 (depends on 8)

The computing device according to claim 8, wherein the photo set-level similarity between two photo sets is defined in the following expression:
s
_
=
{
1
n
i

n
j

∑


s
ij

(
f
i
∈
A
i
,
f
j
∈
A
j
)
}
,
s representing the photo set-level similarity, Ai represents one of the two photo sets, Aj represents the other of the two photo sets, fi represents a face image feature that corresponds to a photo in the photo set Ai, fj represents a face image feature that corresponds to a photo in the photo set Aj, sij represents a similarity that corresponds to a pair of face image features formed by the face image feature fi and the face image feature fj, ni represents a quantity of photos in the photo set Ai, and nj represents a quantity of photos in the photo set Aj, i and j being two unequal natural numbers.

### Claim 11 (depends on 8)

The computing device according to claim 8, wherein the photo-level similarity between a pair of photo sets is determined according to respective numbers of photos included in each photo set of the pair of photo sets.

### Claim 12 (depends on 8)

The computing device according to claim 8, wherein the plurality of operations further comprise:
displaying the one or more labels for each photo album to an end user; and
prompting the user to update the one or more labels for the photo album.

### Claim 13 (depends on 8)

The computing device according to claim 8, wherein the plurality of operations further comprise:
sorting photos in each output photo album according to an order of photographing time of the photos.

### Claim 14 (depends on 13)

The computing device according to claim 13, wherein the plurality of operations further comprise:
dividing photos in each output photo album into different photo subsets according to the order of photographing time of the photos; and
generating a sub-label for each photo subset according to the photographing time of the photos in the photo subset.

### Claim 15

A non-transitory computer readable storage medium, wherein the storage medium stores a plurality of program instructions that, when executed by one or more processors of a computing device, cause the computing device to perform a plurality of operations including:
performing face detection on multiple photos to obtain a face image feature set, each face image feature in the face image feature set corresponding to one of the multiple photos;
determining a face-level similarity for each pair of face image features in the face image feature set;
determining a photo-level similarity between each pair of photos in the multiple photos in accordance with their associated face-level similarities, wherein the photo-level similarity between a respective pair of photos is a summation of the face-level similarity between the face image features of the respective pair of photos;
selecting, from the multiple photos, a photo set for each target photo, wherein any photo-level similarity between the target photo and another photo in the photo set exceeds a predefined photo-level threshold;
determining a photo set-level similarity between each pair of photo sets associated with the multiple photos, wherein a respective photo set-level similarity between two photo sets is determined according to face-level similarities corresponding to the two photo sets respectively;
combining at least two photo sets into one photo album in accordance with a determination that a face-level similarity between two face image features of two photos from the two photo sets is greater than a preset face-level similarity threshold and any photo set-level similarities of the two photo sets exceeds a predefined photo set-level threshold, respectively; and
generating one or more labels for each of a plurality of photo albums using photographing location and photographing time information associated with the photos within the photo album accordingly.

### Claim 16 (depends on 15)

The non-transitory computer readable storage medium according to claim 15, wherein the plurality of operations further comprise:
selecting labels of the at least two photo sets to be combined into a combined photo album; and
combining the identified labels into one label for the combined photo album.

### Claim 17 (depends on 15)

The non-transitory computer readable storage medium according to claim 15, wherein the photo set-level similarity between two photo sets is defined in the following expression:
s
_
=
{
1
n
i

n
j

∑


s
ij

(
f
i
∈
A
i
,
f
j
∈
A
j
)
}
,
s representing the photo set-level similarity, Ai represents one of the two photo sets, Aj represents the other of the two photo sets, fi represents a face image feature that corresponds to a photo in the photo set Ai, fj represents a face image feature that corresponds to a photo in the photo set Aj, sij represents a similarity that corresponds to a pair of face image features formed by the face image feature fi and the face image feature fj, ni represents a quantity of photos in the photo set Ai, and nj represents a quantity of photos in the photo set Aj, i and j being two unequal natural numbers.

### Claim 18 (depends on 15)

The non-transitory computer readable storage medium according to claim 15, wherein the photo-level similarity between a pair of photo sets is determined according to respective numbers of photos included in each photo set of the pair of photo sets.

### Claim 19 (depends on 15)

The non-transitory computer readable storage medium according to claim 15, wherein the plurality of operations further comprise:
displaying the one or more labels for each photo album to an end user; and
prompting the user to update the one or more labels for the photo album.

### Claim 20 (depends on 15)

The non-transitory computer readable storage medium according to claim 15, wherein the plurality of operations further comprise:
sorting photos in each output photo album according to an order of photographing time of the photos;
dividing photos in the output photo album into different photo subsets according to the order of photographing time of the photos; and
generating a sub-label for each photo subset according to the photographing time of the photos in the photo subset.
