# US 2007/0183663 A1 · Intra-mode region-of-interest video object segmentation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2007/0183663 A1
- Document ID: US20070183663A1
- Status: Published application
- Kind: A1
- Application number: 11/350,376
- Filed: 2006-02-07
- Published: 2007-08-09
- Earliest claimed: 2006-02-07
- Assignees: QUALCOMM Incorporated
- Inventors: Haohong Wang; Khaled El-Maleh
- Web page: https://patentgrep.com/patent/US20070183663A1

## CPC

- H04N 19/17: the unit being an image region, e.g. an object
- G06V 40/162: {using pixel segmentation or colour matching}
- G06V 40/165: {using facial parts and geometric relationships}
- G06V 40/167: {using comparisons between temporally consecutive images}
- G06V 40/19: Sensors therefor
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

The disclosure is directed to techniques for automatic segmentation of a region-of-interest (ROI) video object from a video sequence. ROI object segmentation enables selected ROI or “foreground” objects of a video sequence that may be of interest to a viewer to be extracted from non-ROI or “background” areas of the video sequence. Examples of a ROI object are a human face or a head and shoulder area of a human body. The disclosed techniques include a hybrid technique that combines ROI feature detection, region segmentation, and background subtraction. In this way, the disclosed techniques may provide accurate foreground object generation and low-complexity extraction of the foreground object from the video sequence. A ROI object segmentation system may implement the techniques described herein. In addition, ROI object segmentation may be useful in a wide range of multimedia applications that utilize video sequences, such as video telephony applications and video surveillance applications.

## Claims

### Claim 1

A method comprising:
receiving a video frame of a video sequence;
detecting region of interest (ROI) features within the video frame;
approximating a ROI object shape within the video frame based on the detected ROI features;
segmenting the video frame into multiple candidate regions based on the detected ROI features; and
selecting one or more of the candidate regions located within the ROI object shape of the video frame as an ROI object.

### Claim 2 (depends on 1)

The method of claim 1, further comprising generating an output image that identifies the ROI object, and preferentially encoding the ROI object relative to non-ROI regions of the video frame.

### Claim 3 (depends on 1)

The method of claim 1, wherein selecting one or more candidate regions comprises selecting one or more of the candidate regions as the ROI object without reference to motion information for the video frame in the video sequence.

### Claim 4 (depends on 1)

The method of claim 1, wherein detecting ROI features comprises:
detecting a face mask within the video frame based on chrominance values of skin-color regions within the video frame;
detecting eye features within the video frame based on both chrominance values and luminance values of pixels within the detected face mask; and
detecting mouth features within the video frame based on chrominance values of pixels within the detected face mask.

### Claim 5 (depends on 1)

The method of claim 1, wherein the detected ROI features comprise detected facial feature candidates within the video frame, the method further comprising verifying the detected facial feature candidates to select correct facial features for the ROI object and remove false facial features from the set of facial feature candidates.

### Claim 6 (depends on 5)

The method of claim 5, wherein verifying the detected facial feature candidates comprises:
detecting valley regions within the video frame;
comparing locations of the detected facial feature candidates within the video frame to locations of the valley regions within the video frame;
removing a facial feature from the set of facial feature candidates when the facial feature does not at least partially overlap one of the detected valley regions within the video frame.

### Claim 7 (depends on 5)

The method of claim 5, wherein verifying the detected facial feature candidates comprises:
comparing each of the detected facial feature candidates to a plurality of criteria based on symmetric and geometric properties of eye features;
assigning a score to each of the detected facial feature candidates based on a level of correspondence with the plurality of criteria; and
removing a facial feature from the set of facial feature candidates when the score assigned to the facial feature is less than a threshold value.

### Claim 8 (depends on 5)

The method of claim 5, wherein verifying the detected facial feature candidates comprises:
combining eye feature candidates and mouth feature candidates into eye-mouth triangles; and
verifying the eye-mouth triangles based on orientation of the eye-mouth triangles within the video frame and gradient characteristics of the ROI object within the eye-mouth triangles.

### Claim 9 (depends on 1)

The method of claim 1, wherein the video frame includes more than one ROI object, the method further comprising separating the detected ROI features into groups for the individual ROI objects, and applying maximum matching graph theory to the detected ROI features within the video frame.

### Claim 10 (depends on 1)

The method of claim 1, wherein approximating a ROI object shape comprises approximating the ROI object shape based on a geometric model defined by a location of the detected ROI features, wherein the approximate ROI object shape comprises a human head and shoulder object shape within the video frame.

### Claim 11 (depends on 1)

The method of claim 1, wherein segmenting the video frame comprises:
classifying each pixel within the video frame based on similarity to neighboring pixels; and
merging the similar pixels into a minimum number of the candidate regions within the video frame.

### Claim 12 (depends on 1)

The method of claim 1, wherein selecting one or more of the candidate regions comprises selecting one or more of the candidate regions that have a predetermined percentage of pixels located within the ROI object shape of the video frame, the method further comprising generating a foreground object by merging the selected one or more of the candidate regions.

### Claim 13 (depends on 12)

The method of claim 12, wherein the video frame includes more than one foreground object, the method further comprising merging the more than one foreground objects to form an output image and sending the output image to a multimedia application.

### Claim 14

A computer-readable medium comprising instructions that cause a programmable processor to:
receive a video frame of a video sequence;
detect region of interest (ROI) features within the video frame;
approximate a ROI object shape within the video frame based on the detected ROI features;
segment the video frame into multiple candidate regions based on the detected ROI features; and
select one or more of the candidate regions located within the ROI object shape of the video frame as an ROI object.

### Claim 15 (depends on 14)

The computer-readable medium of claim 14, wherein the instructions cause the processor to generate an output image that identifies the ROI object, and preferentially encode the ROI object relative to non-ROI regions of the video frame.

### Claim 16 (depends on 14)

The computer-readable medium of claim 14, further comprising instructions to cause the processor to: select one or more of the candidate regions as the ROI object without reference to motion information for the video frame and a different video frame in the video sequence.

### Claim 17 (depends on 14)

The computer-readable medium of claim 14, wherein the instructions that cause the programmable processor to detect ROI features cause the programmable processor to:
detect a face mask within the video frame based on chrominance values of skin-color regions within the video frame;
detect eye features within the video frame based on both chrominance values and luminance values of pixels within the detected face mask; and
detect mouth features within the video frame based on chrominance values of pixels within the detected face mask.

### Claim 18 (depends on 14)

The computer-readable medium of claim 14, wherein the detected ROI features comprise detected facial feature candidates within the video frame, further comprising instructions that cause the programmable processor to verify the detected facial feature candidates to select correct facial features for the ROI object and remove false facial features from the set of facial feature candidates.

### Claim 19 (depends on 18)

The computer-readable medium of claim 18, wherein the instructions cause the programmable processor to:
detect valley regions within the video frame;
compare locations of the detected facial feature candidates within the video frame to locations of the valley regions within the video frame;
remove a facial feature from the set of facial feature candidates when the facial feature does not at least partially overlap one of the detected valley regions within the video frame.

### Claim 20 (depends on 18)

The computer-readable medium of claim 18, wherein the instructions cause the programmable processor to:
compare each of the detected facial feature candidates to a plurality of criteria based on symmetric and geometric properties of eye features;
assign a score to each of the detected facial feature candidates based on a level of correspondence with the plurality of criteria; and
remove a facial feature from the set of facial feature candidates when the score assigned to the facial feature is less than a threshold value.

### Claim 21 (depends on 18)

The computer-readable medium of claim 18, wherein the instructions cause the programmable processor to:
combine eye feature candidates and mouth feature candidates into eye-mouth triangles; and
verify the eye-mouth triangles based on orientation of the eye-mouth triangles within the video frame and gradient characteristics of the ROI object within the eye-mouth triangles.

### Claim 22 (depends on 14)

The computer-readable medium of claim 14, wherein the video frame includes more than one ROI object, further comprising instructions that cause the programmable processor to separate the detected ROI features into groups for the individual ROI objects, and apply maximum matching graph theory to the detected ROI features within the video frame.

### Claim 23 (depends on 14)

The computer-readable medium of claim 14, wherein the instructions cause the programmable processor to approximate the ROI object shape based on a geometric model defined by a location of the detected ROI features.

### Claim 24 (depends on 14)

The computer-readable medium of claim 14, wherein the instructions that cause the programmable processor to segment the video frame cause the programmable processor to:
classify each pixel within the video frame based on similarity to neighboring pixels; and
merge the similar pixels into a minimum number of candidate regions within the first frame.

### Claim 25 (depends on 17)

The computer-readable medium of claim 17, wherein the instructions cause the programmable processor to select one or more of the candidate regions that have a predetermined percentage of pixels located within the ROI object shape of the video frame, further comprising instructions that cause the programmable processor to generate a foreground object by merging the selected one or more of the candidate regions.

### Claim 26 (depends on 25)

The computer-readable medium of claim 25, wherein the video frame includes more than one foreground object, further comprising instructions that cause the programmable processor to merge the more than one foreground objects to form the output image and send the output image to a multimedia application.

### Claim 27

An automatic region of interest (ROI) object segmentation system comprising:
a ROI feature detector that detects ROI features within a video frame of a video sequence;
an object shape approximation module that approximates a ROI object shape within the video frame based on the detected ROI features;
a region segmentation module that segments the video frame into multiple candidate regions based on the detected ROI features; and
a object generation module that selects one or more of the candidate regions located within the ROI object shape of the video frame as an ROI object.

### Claim 28 (depends on 27)

The system of claim 27, further comprising:
an object fusion module that generates an output image that identifies the ROI object; and
an encoder that preferentially encodes the ROI object relative to non-ROI regions of the video frame.

### Claim 29 (depends on 27)

The system of claim 27, wherein the object generation module selects the one or more of the candidate regions as the ROI object without reference to motion information for the video frame in the video sequence.

### Claim 30 (depends on 27)

The system of claim 27, wherein the ROI feature detector includes a face mask detector that detects a face mask within the video frame based on chrominance values of skin-color regions within the video frame, an eye detector that detects eye features within the video frame based on both chrominance values and luminance values of pixels within the detected face mask, and a mouth detector that detects mouth features within the video frame based on chrominance values of pixels within the detected face mask.

### Claim 31 (depends on 27)

The system of claim 27, wherein the detected ROI features comprise detected facial feature candidates within the video frame, the system further comprising a feature verification module that verifies the detected facial feature candidates to select correct facial features for the ROI object and remove false facial features from the set of facial feature candidates, and wherein the feature verification module:
detects valley regions within the video frame;
compares locations of the detected facial feature candidates within the video frame to locations of the valley regions within the video frame;
removes a facial feature from the set of facial feature candidates when the facial feature does not at least partially overlap one of the detected valley regions within the video frame.

### Claim 32 (depends on 31)

The system of claim 31, wherein the feature verification module:
compares each of the detected facial feature candidates to a plurality of criteria based on symmetric and geometric properties of eye features;
assigns a score to each of the detected facial feature candidates based on a level of correspondence with the plurality of criteria; and
removes a facial feature from the set of facial feature candidates when the score assigned to the facial feature is less than a threshold value.

### Claim 33 (depends on 31)

The system of claim 31, wherein the feature verification module:
combines eye feature candidates and mouth feature candidates into eye-mouth triangles; and
verifies the eye-mouth triangles based on orientation of the eye-mouth triangles within the video frame and gradient characteristics of the ROI object within the eye-mouth triangles.

### Claim 34 (depends on 27)

The system of claim 27, wherein the video frame includes more than one ROI object, further comprising a multi-face separation module that separates the detected ROI features into groups for the individual ROI objects, and applies maximum matching graph theory to the detected ROI features within the video frame to separate the detected ROI features.

### Claim 35 (depends on 27)

The system of claim 27, wherein the object shape approximation module approximates the ROI object shape based on a geometric model defined by a location of the detected ROI features, wherein the approximate ROI object shape comprises a human head and shoulder object shape within the video frame.

### Claim 36 (depends on 27)

The system of claim 27, wherein the region segmentation module:
classifies each pixel within the video frame based on similarity to neighboring pixels; and
merges the similar pixels into a minimum number of candidate regions within the first frame.

### Claim 37 (depends on 27)

The system of claim 27, further comprising an object generation module that selects one or more of the candidate regions that have a predetermined percentage of pixels located within the ROI object shape of the video frame, wherein the object generation module generates a foreground object by merging the selected one or more of the candidate regions.

### Claim 38 (depends on 37)

The system of claim 37, wherein the video frame includes more than one foreground objects, wherein the object fusing module merges the more than one foreground objects to form the output image and sends the output image to a multimedia application.

### Claim 39

A method comprising:
receiving a video frame of a video sequence;
detecting region of interest (ROI) features within the video frame, wherein the ROI features include a face mask including eye features and mouth features;
segmenting the video frame into multiple candidate regions based on the detected ROI features;
combining eye feature candidates and mouth feature candidates into eye-mouth triangles;
verifying the eye-mouth triangles based on orientation of the eye-mouth triangles within the video frame; and
selecting one or more of the candidate regions as an ROI object based on the verification.

### Claim 40 (depends on 39)

The method of claim 39, wherein detecting ROI features comprises detecting the face mask within the video frame based on both chrominance values and luminance values of pixels within the detected face mask.

### Claim 41 (depends on 39)

The method of claim 39, wherein verifying the detected facial feature candidates comprises:
detecting valley regions within the video frame;
comparing locations of the detected facial feature candidates within the video frame to locations of the valley regions within the video frame;
removing a facial feature from the set of facial feature candidates when the facial feature does not at least partially overlap one of the detected valley regions within the video frame.

### Claim 42 (depends on 39)

The method of claim 39, wherein verifying the detected facial feature candidates comprises:
comparing each of the detected facial feature candidates to a plurality of criteria based on symmetric and geometric properties of eye features;
assigning a score to each of the detected facial feature candidates based on a level of correspondence with the plurality of criteria; and
removing a facial feature from the set of facial feature candidates when the score assigned to the facial feature is less than a threshold value.

### Claim 43

A computer-readable medium comprising instructions to cause a programmable processor to:
receive a video frame of a video sequence;
detect region of interest (ROI) features within the video frame, wherein the ROI features include a face mask including eye features and mouth features;
segment the video frame into multiple candidate regions based on the detected ROI features;
combine eye feature candidates and mouth feature candidates into eye-mouth triangles;
verify the eye-mouth triangles based on orientation of the eye-mouth triangles within the video frame; and
select one or more of the candidate regions as an ROI object based on the verification.

### Claim 44 (depends on 43)

The computer-readable medium of claim 43, wherein the instructions cause the processor to detect the face mask within the video frame based on both chrominance values and luminance values of pixels within the detected face mask.

### Claim 45 (depends on 43)

The computer-readable medium of claim 43, wherein verification of the detected facial feature candidates includes:
detecting valley regions within the video frame;
comparing locations of the detected facial feature candidates within the video frame to locations of the valley regions within the video frame;
removing a facial feature from the set of facial feature candidates when the facial feature does not at least partially overlap one of the detected valley regions within the video frame.

### Claim 46 (depends on 43)

The computer-readable medium of claim 43, wherein verification of the detected facial feature candidates comprises:
comparing each of the detected facial feature candidates to a plurality of criteria based on symmetric and geometric properties of eye features;
assigning a score to each of the detected facial feature candidates based on a level of correspondence with the plurality of criteria; and
removing a facial feature from the set of facial feature candidates when the score assigned to the facial feature is less than a threshold value.

### Claim 47

A system comprising:
a region of interest (ROI) feature detector that detects ROI features within a video frame of a video sequence, wherein the ROI features include a face mask including eye features and mouth features;
a region segmentation module that segments the video frame into multiple candidate regions based on the detected ROI features;
a feature verification module that combines eye feature candidates and mouth feature candidates into eye-mouth triangles, and verifies the eye-mouth triangles based on orientation of the eye-mouth triangles within the video frame; and
an object generation module that selects one or more of the candidate regions as an ROI object based on the verification.

### Claim 48 (depends on 47)

The system of claim 47, wherein the ROI feature detector detects the face mask within the video frame based on both chrominance values and luminance values of pixels within the detected face mask.

### Claim 49 (depends on 47)

The system of claim 47, wherein the feature verification module:
detects valley regions within the video frame;
compares locations of the detected facial feature candidates within the video frame to locations of the valley regions within the video frame;
removes a facial feature from the set of facial feature candidates when the facial feature does not at least partially overlap one of the detected valley regions within the video frame.

### Claim 50 (depends on 49)

The system of claim 49, wherein the feature verification module:
compares each of the detected facial feature candidates to a plurality of criteria based on symmetric and geometric properties of eye features;
assigns a score to each of the detected facial feature candidates based on a level of correspondence with the plurality of criteria; and
removes a facial feature from the set of facial feature candidates when the score assigned to the facial feature is less than a threshold value.
