# US 6,732,016 B2 · Robot, identifying environment determining method, and program thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 6,732,016 B2
- Document ID: US6732016B2
- Status: Granted
- Kind: B2
- Application number: 10/308,155
- Filed: 2002-12-03
- Published: 2003-05-01
- Granted: 2004-05-04
- Earliest claimed: 2001-01-30
- Assignees: NEC Corporation
- Inventors: Junichi Funada
- Web page: https://patentgrep.com/patent/US6732016B2

## CPC

- H04N 23/70: Circuitry for compensating brightness variation in the scene
- G06V 40/172: {Classification, e.g. identification}
- H04N 23/611: where the recognised objects include parts of the human body
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An identification environment determining method includes converting a captured image into digital information, detecting a face area of a person from the digital image to store the face area, and identifying the person using a stored image and face image information that is referred for identification, informing a user about a result of identification acquired, inputting information obtained by having the user judge whether the result of identification is correct or incorrect, determining whether a present lighting environment is suitable for identification, controlling a robot, and moving the robot.

## Claims

### Claim 1

An identification environment determining method comprising:
converting a captured image into digital information;
detecting a face area of a person from the digital information to store the face area, and identifying the person using a stored image and face image information;
informing a user about a result of said identifying said person;
inputting information obtained by having the user judge whether the result of identification is correct or incorrect;
determining whether a present lighting environment is suitable for identification using information obtained in said converting said captured image;
controlling a robot; and
moving the robot.

### Claim 2 (depends on 1)

The identification environment determining method as claimed in claim 1, wherein whether the lighting environment is suitable is determined using at least one of a variance value of pixel values in the face area a ratio of an average pixel value in the face area to an average pixel value in the vicinity of an outside of the detected face area, and an average pixel value of the whole image.

### Claim 3 (depends on 1)

The identification environment determining method as claimed in claim 1, further comprising at least one of:
observing whether a lighting device in a room is turned on;
observing whether curtains in a room are drawn;
retaining information about time; and
retaining positional information obtained by observing a position of the robot in a room,
wherein whether the lighting environment is suitable is determined using at least one of information about whether the lighting device is turned on, information about whether the curtains are drawn, information about time just before executing identification, and the positional information of the robot.

### Claim 4 (depends on 1)

The identification environment determining method as claimed in claim 1, wherein:
when determining to shift to a state for identification whether the present lighting environment is suitable for identification is determined
when the present lighting environment is suitable for identification, the person is identified; and
when that the present lighting environment is not suitable for identification, a control is conducted not to identify the person.

### Claim 5 (depends on 1)

The identification environment determining method as claimed in claim 1, wherein:
when determining to shift to a state for identification whether the present lighting environment is suitable for identification is determined
when that the present lighting environment is suitable for identification, the person is identified;
when that the present lighting environment is not suitable for identification, a control is conducted not to identify the person; and
the lighting environment is determined using at least one of a variance value of pixel values in the face area, a ratio of an average pixel value in the face area to an average pixel value in the vicinity of an outside of the detected face area, and an average pixel value of the whole image.

### Claim 6 (depends on 1)

The identification environment determining method as claimed in claim 1, further comprising at least one of:
observing whether a lighting device in a room is turned on;
observing whether curtains in a room are drawn;
retaining information about time; and
retaining positional information obtained by observing a position of the robot in a room, wherein:
when determining to shift to a state for identification whether the present lighting environment is suitable for identification is determined; and
when that the present lighting environment is suitable for identification, the person is identified;
when the present lighting environment is not suitable for identification, a control is conducted not to identify the person; and
the lighting environment is determined at using at least one of information about whether or the lighting device is turned on, information about whether the curtains are drawn, information about time just before executing identification, and the positional information of the robot.

### Claim 7 (depends on 1)

The identification environment determining method as claimed in claim 1, wherein:
when determining to shift to a state for identification whether the present lighting environment is suitable for identification is determined;
when the present lighting environment is suitable for identification the person is identified; and
when that the present lighting environment is not suitable for identification, a control is conducted not to identify the person, and the robot is moved.

### Claim 8 (depends on 1)

The identification environment determining method as claimed in claim 1, wherein:
when determining to shift to a state for identification, it is determined whether the present lighting environment is suitable for identification is determined; and
when the present lighting environment is suitable for identification, the person is identified; or
when the present lighting environment is not suitable for identification, a control is conducted not to identify the person, and the robot is moved; and
the lighting environment is determined using at least one of a variance value of pixel values in the face area, a ratio of an average pixel value in the face area to an average pixel value in the vicinity of an outside of the detected face area, and an average pixel value of the whole image.

### Claim 9 (depends on 1)

The identification environment determining method as claimed in claim 1, further comprising at least one step selected from:
observing whether a lighting device in a room is turned on;
observing whether or not curtains in a room are drawn;
retaining information about time; and
retaining positional information obtained by observing a position of the robot in a room, wherein:
when determining to shift to a state for identification, whether the present lighting environment is suitable for identification is determined;
when the present lighting environment is suitable for identification, the person is identified; and
when the present lighting environment is not suitable for identification, a control is conducted not to identify the person, and the robot is moved; and
the lighting environment is determined using at least one information selected from information about whether the lighting device is turned on, information about whether or not the curtains are drawn, information about time just before executing identification, and the positional information of the robot.

### Claim 10

An identification environment determining method comprising:
converting a captured image into digital information;
detecting a face area of a person from the digital information to store the face area, and identifying the person using a stored image and face image information that is referred to for identification;
informing a user about a result of identification;
inputting information obtained by having the user judge whether the result of identification is correct or incorrect;
determining a lighting environment on the basis of the image information acquired in said converting said captured image, and classifying the lighting environment into a corresponding lighting category;
recording, with respect to each lighting category, identifying performance information including a number of times where face identification has succeeded and a number of times where face identification has been executed;
determining whether a present lighting environment is suitable for identification on the basis of the identifying performance information;
controlling a robot; and
moving the robot.

### Claim 11 (depends on 10)

The identification environment determining method as claimed in claim 10, wherein the lighting environment is categorized into the corresponding lighting category using at least one of a variance value of pixel values in the face area, a ratio of an average pixel value in the face area to an average pixel value in the vicinity of an outside of the detected face area, and an average pixel value of the whole image.

### Claim 12 (depends on 10)

The identification environment determining method as claimed in claim 10, further comprising at least one of:
observing whether a lighting device in a room is turned on;
observing whether or not curtains in a room are drawn;
retaining information about time; and
retaining positional information obtained by observing a position of the robot in a room, wherein:
the lighting environment is categorized into the corresponding lighting category using at least one of information about whether the lighting device is turned on, information about whether the curtains are drawn, information about time just before executing identification, and the positional information of the robot.

### Claim 13 (depends on 10)

The identification environment determining method as claimed in claim 10, wherein:
the identifying performance information is recorded each time the information whether result of identification is correct is acquired from the user;
when determining to shift to a state for identification, whether the present lighting environment is suitable for identification is determined;
when that the present lighting environment is suitable for identification, the person is identified; and
when that the present lighting environment is not suitable for identification, a control is conducted not to identify the person.

### Claim 14 (depends on 10)

The identification environment determining method as claimed in claim 10, wherein:
the identifying performance information is recorded each time the information whether a result of identification is correct is acquired from the user;
when determining to shift to a state for identification, whether the present lighting environment is suitable for identification is determined;
when the present lighting environment is suitable for identification, the person is identified;
when the present lighting environment is not suitable for identification, a control is conducted not to identify the person; and
the lighting environment is categorized into the corresponding lighting category using at least one of a variance value of pixel values in the face area, a ratio of an average pixel value in the face area to an average pixel value in the vicinity of an outside of the detected face area, and an average pixel value of the whole image.

### Claim 15 (depends on 10)

The identification environment determining method as claimed in claim 10, further comprising at least one of:
observing whether or not a lighting device in a room is turned on;
observing whether or not curtains in a room are drawn;
retaining information about time; and
retaining positional information obtained by observing a position of the robot in a room, wherein:
the identifying performance information is recorded each time the information whether a result of identification is correct is acquired from the user;
when determining to shift to a state for identification, whether the present lighting environment is suitable for identification is determined at; and
when that the present lighting environment is suitable for identification, the person is identified; or
when that the present lighting environment is not suitable for identification, a control is conducted not to identify the person at; and
the lighting environment is categorized into the corresponding lighting category using at least one of information about whether the lighting device is turned on, information about whether the curtains are drawn, information about time just before executing identification, and the positional information of the robot.

### Claim 16 (depends on 10)

The identification environment determining method as Claimed in claim 10, wherein:
the identifying performance information is recorded each time the information about whether a result of identification is correct is acquired from the user;
when determining to shift to a state for identification, whether the present lighting environment is suitable for identification is determined; and
when the present lighting environment is suitable for identification, the person is identified; or
when the present lighting environment is not suitable for identification, a control is conducted not to identify the person, and the robot is moved.

### Claim 17 (depends on 10)

The identification environment determining method as claimed in claim 10, wherein:
the identifying performance information is recorded each time the information about whether a result of identification is correct is acquired from the user;
when determining to shift to a state for identification, whether the present lighting environment is suitable for identification is determined at;
when the present lighting environment is suitable for identification, the person is identified; and
when the present lighting environment is not suitable for identification, a control is conducted not to identify the person, and the robot is moved; and
the lighting environment is categorized into the corresponding lighting category using at least one of a variance value of pixel values in the face area, a ratio of an average pixel value in the face area to an average pixel value in the vicinity of an outside of the detected face area, and an average pixel value of the whole image.

### Claim 18 (depends on 10)

The identification environment determining method as claimed in claim 10, further comprising at least one of:
observing whether a lighting device in a room is turned on;
observing whether curtains in a room are drawn;
retaining information about time; and
retaining positional information obtained by observing a position of the robot in a room, wherein:
the identifying performance information is recorded each time the information about whether a result of identification is correct is acquired from the user;
when determining to shift to a state for identification, whether the present lighting environment is suitable for identification is determined at;
when that the present lighting environment is suitable for identification, the person is identified; and
when the present lighting environment is not suitable for identification, a control is conducted not to identify the person, and the robot is moved; and
the lighting environment is categorized into the corresponding lighting category using at least one of information about whether the lighting device is turned on, information about whether the curtains are drawn, information about time just before executing identification, and the positional information of the robot.

### Claim 19

An identification environment determining method comprising:
converting a captured image into digital information;
detecting a face area of a person from the digital information to store the face area, and identifying the person using a stored image and face image information that is referred to for identification;
informing a user about a result of said identifying said person;
inputting information obtained by having the user judge whether the result of identification is correct;
determining a lighting environment on the basis of the image information acquired in said converting said captured image, and classifying the lighting environment into a corresponding lighting category;
recording, with respect to each lighting category, identifying performance information including a number of times where face identification has succeeded and a number of times where face identification has been executed;
determining whether a present lighting environment is suitable for identification on the basis of the identifying performance information;
controlling a robot;
and
moving the robot to a position where identifying performance is higher on the basis of the identifying performance information.

### Claim 20 (depends on 19)

The identification environment determining method as claimed in claim 19, wherein the lighting environment is categorized into the corresponding lighting category using at least one of a variance value of pixel values in the face area, a ratio of an average pixel value in the face area to an average pixel value in the vicinity of an outside of the detected face area, and an average pixel value of the whole image.

### Claim 21 (depends on 19)

The identification environment determining method as claimed in claim 19, further comprising at least one of:
a lighting device state observing step for observing whether a lighting device in a room is turned on;
a curtain state observing step for observing whether curtains in a room are drawn;
retaining information about time; and
retaining positional information obtained by observing a position of the robot in a room, wherein:
the lighting environment is categorized into the corresponding lighting category using at least one of information about whether the lighting device is turned on, information about whether the curtains are drawn, information about time just before executing identification, and the positional information of the robot.

### Claim 22 (depends on 19)

The identification environment determining method as claimed in claim 19, wherein:
the identifying performance information is recorded each time the information about whether a result of identification is correct is acquired from the user;
when determining to shift to a state for identification, whether the present lighting environment is suitable for identification is determined;
when that the present lighting environment is suitable for identification, the person is identified; and
when the present lighting environment is not suitable for identification, a control is conducted not to identify the person.

### Claim 23 (depends on 19)

The identification environment determining method as claimed in claim 19, wherein:
the identifying performance information is recorded each time the information about whether a result of identification is correct is acquired from the user;
when determining to shift to a state for identification, whether the present lighting environment is suitable for identification is determined; and
when the present lighting environment is suitable for identification, the person is identified; or
when the present lighting environment is not suitable for identification, a control is conducted not to identify the person; and
the lighting environment is categorized into the corresponding lighting category using at least one of a variance value of pixel values in the face area, a ratio of an average pixel value in the face area to an average pixel value in the vicinity of an outside of the detected face area, and an average pixel value of the whole image.

### Claim 24 (depends on 19)

The identification environment determining method as claimed in claim 19, further comprising at least one of:
observing whether a lighting device in a room is turned on;
observing whether curtains in a room are drawn;
retaining information about time; and
retaining positional information obtained by observing a position of the robot in a room, wherein:
the identifying performance information is recorded each time the information about whether a result of identification is correct is acquired from the user;
when determining to shift to a state for identification, whether the present lighting environment is suitable for identification is determined; and
when the present lighting environment is suitable for identification, the person is identified; or
when the present lighting environment is not suitable for identification, a control is conducted not to identify the person; and
the lighting environment is categorized into the corresponding lighting category using at least one of information about whether the lighting device is turned on, information about whether the curtains are drawn, information about time just before executing identification, and the positional information of the robot.

### Claim 25 (depends on 19)

The identification environment determining method as claimed in claim 19, wherein:
the identifying performance information is recorded each time the information about whether a result of identification is correct or is acquired from the user;
when determining to shift to a state for identification, whether the present lighting environment is suitable for identification is determined;
when that the present lighting environment is suitable for identification, the person is identified; and
when the present lighting environment is not suitable for identification, a control is conducted not to identify the person, and the robot is moved.

### Claim 26 (depends on 19)

The identification environment determining method as claimed in claim 19, wherein:
the identifying performance information is recorded each time the information about whether a result of identification is correct is acquired from the user;
when determining to shift to a state for identification, whether the present lighting environment is suitable for identification is determined;
when that the present lighting environment is suitable for identification, the person is identified; and
when the present lighting environment is not suitable for identification, a control is conducted not to identify the person, and the robot is moved; and
the lighting environment is categorized into the corresponding lighting category using at least one of a variance value of pixel values in the face area, a ratio of an average pixel value in the face area to an average pixel value in the vicinity of an outside of the detected face area, and an average pixel value of the whole image.

### Claim 27 (depends on 19)

The identification environment determining method as claimed in claim 19, further comprising at least one of:
observing whether a lighting device in a room is turned on;
observing whether curtains in a room are drawn;
retaining information about time; and
retaining positional information obtained by observing a position of the robot in a room, wherein:
the identifying performance information is recorded each time the information about whether a result of identification is correct incorrect is acquired from the user;
when determining to shift to a state for identification, whether the present lighting environment is suitable for identification is determined; and
when the present lighting environment is suitable for identification, the person is identified; or
when the present lighting environment is not suitable for identification, a control is conducted not to identify the person, and the robot is moved; and
the lighting environment is categorized into the corresponding lighting category using at least one of information about whether the lighting device is turned on, information about whether the curtains are drawn, information about time just before executing identification, and the positional information of the robot.

### Claim 28

An identification environment determining method comprising:
converting a captured image into digital information;
detecting a face area of a person from the digital image to store the face area, and identifying the person using a stored image and face image information that is referred to for identification;
informing a user about a result of said identifying said person;
inputting information obtained by having the user judge whether the result of identification is correct;
determining a lighting environment on the basis of the image information acquired in said converting said captured image, and classifying the lighting environment into a corresponding lighting category;
recording a lighting environment at registration for registering the captured face image and recording a lighting category for the registered face image with respect to each user;
comparing a lighting environment at registration for determining, on the basis of information recorded in said recording said lighting environment at registration, whether a face image of a user communicating with a robot has been registered in a same lighting environment as a present lighting environment;
controlling the robot; and
moving the robot.

### Claim 29 (depends on 28)

The identification environment determining method as claimed in claim 28, wherein the lighting environment is categorized into the corresponding lighting category at the lighting environment determining step using at least one of a variance value of pixel values in the face area, a ratio of an average pixel value in the face area to an average pixel value in the vicinity of an outside of the detected face area, and an average pixel value of the whole image.

### Claim 30 (depends on 28)

The identification environment determining method as claimed in claim 28, further comprising at least one of:
observing whether a lighting device in a room is turned on;
observing whether curtains in a room are drawn;
retaining information about time; and
retaining positional information obtained by observing a position of the robot in a room, wherein:
the lighting environment is categorized into the corresponding lighting category using at least one of information about whether the lighting device is turned on, information about whether the curtains are drawn, information about time just before executing identification, and the positional information of the robot.

### Claim 31 (depends on 28)

The identification environment determining method as claimed in claim 28, wherein:
when determining to shift to a state for identification, the person is identified; and
when the face image of the user communicating with the robot has not been registered in the lighting environment in said comparing said a lighting environment at registration, an instruction is to register the face image of the user.

### Claim 32 (depends on 28)

The identification environment determining method as claimed in claim 28, wherein:
when determining to shift to a state for identification, the person is identified;
the lighting environment is categorized into the corresponding lighting category using at least one of a variance value of pixel values in the face area, a ratio of an average pixel value in the face area to an average pixel value in the vicinity of an outside of the detected face area, and an average pixel value of whole image; and
when the face image of the user communicating with the robot has not been registered in the lighting environment, an instruction is issued to register the face image of the user.

### Claim 33 (depends on 28)

The identification environment determining method as claimed in claim 28, further comprising at least one of:
observing whether a lighting device in a room is turned on;
observing whether curtains in a room are drawn;
retaining information about time; and
retaining positional information obtained by observing a position of the robot in a room, wherein:
when determining to shift to a state for identification, the person is identified;
the lighting environment is categorized into the corresponding lighting category using at least one of information about whether the lighting device is turned on, information about whether the curtains are drawn, information about time just before executing identification, and the positional information of the robot; and
when determining that the face image of the user communicating with the robot has not been registered in the lighting environment at the step of comparing a lighting environment at registration, an instruction is issued to register the face image of the user.

### Claim 34

An identification environment determining method comprising:
converting a captured image into digital information;
detecting a face area of a person from the digital information to store the face area, and identifying the person using a stored image and face image information that is referred to for identification;
informing a user about a result of identification;
inputting information obtained by having the user judge whether the result of identification is correct;
determining a lighting environment on the basis of the image information acquired in said converting said captured image, and classifying the lighting environment into a corresponding lighting category;
detecting and extracting a face area from the image and converting the face area into characteristic data to execute identification;
storing, when acquiring a characteristic data group of a face per person, the characteristic data group in each lighting category;
compiling a recognition dictionary using characteristic data per lighting category;
storing, per lighting category, the recognition dictionary compiled at the dictionary compiling step;
selecting, from recognition dictionaries stored at the dictionary storing step, a recognition dictionary of a same lighting category as the lighting category determined at the lighting environment determining step;
a face identification step for executing identification of characteristic data to determine a user's name on the basis of the recognition dictionary selected at the dictionary selecting step;
controlling a robot; and
moving the robot.

### Claim 35 (depends on 34)

The identification environment determining method as claimed in claim 34, wherein the lighting environment is categorized into the corresponding lighting category using at least one of a variance value of pixel values in the face area, a ratio of an average pixel value in the face area to an average pixel value in the vicinity of an outside of the detected face area, and an average pixel value of the whole image.

### Claim 36 (depends on 34)

The identification environment determining method as claimed in claim 34, further comprising at least one of:
observing whether a lighting device in a room is turned on;
observing whether curtains in a room are drawn;
retaining information about time; and
retaining positional information obtained by observing a position of the robot in a room, wherein:
the lighting environment is categorized into the corresponding lighting category using at least one of information about whether the lighting device is turned on, information about whether the curtains are drawn, information about time just before executing identification, and the positional information of the robot.

### Claim 37 (depends on 34)

The identification environment determining method as claimed in claim 34 wherein when determining to shift to a state for identification, the person is identified.

### Claim 38 (depends on 34)

The identification environment determining method as claimed in claim 34, wherein:
when determining to shift to a state for identification, the person is identified; and
the lighting environment is categorized into the corresponding lighting category using at least one of a variance value of pixel values in the face area, a ratio of an average pixel value in the face area to an average pixel value in the vicinity of an outside of the detected face area, and an average pixel value of the whole image.

### Claim 39 (depends on 34)

The identification environment determining method as claimed in claim 34, further comprising at least one of:
observing whether a lighting device in a room is turned on;
observing whether curtains in a room are drawn;
retaining information about time; and
retaining positional information obtained by observing a position of the robot in a room, wherein:
when determining to shift to a state for identification, the person is identified; and
the lighting environment is categorized into the corresponding lighting category using at least one of information about whether the lighting device is turned on, information about whether the curtains are drawn, information about time just before executing identification, and the positional information of the robot.

### Claim 40

An identification environment determining method comprising:
receiving images;
identifying a person using said images by recognizing a face of said person;
informing a user about a result of said identifying said person;
inputting information obtained by having the user judge whether the result of identification is correct or incorrect; and
determining whether a present lighting environment is suitable for identification using information obtained in said receiving said images.

### Claim 41

An identification environment determining method comprising:
receiving images;
identifying a person using said images by recognizing its face;
informing a user about a result of identification;
inputting information obtained by having the user judge whether the result of identification is correct or incorrect;
determining a lighting environment on the basis of the image information acquired in said receiving said images, and classifying the lighting environment into a corresponding lighting category;
recording, with respect to each lighting category, identifying performance information including the number of times where face identification has succeeded and the number of times where face identification has been executed; and
determining whether a present lighting environment is suitable for identification on the basis of the identifying performance information.

### Claim 42

An identification environment determining method comprising:
receiving images;
identifying a person using said images by recognizing a face of said person;
informing a user about a result of said identifying said person;
inputting information obtained by having the user judge whether the result of identification is correct;
determining a lighting environment on the basis of the image information acquired in said receiving said images, and classifying the lighting environment into a corresponding lighting category;
recording a lighting environment at registration for registering the captured face image and recording a lighting category for the registered face image with respect to each user; and
comparing a lighting environment at registration for determining, on the basis of information recorded in said at said recording said lighting environment at registration, whether a face image of a user communicating with a robot has been registered in a same lighting environment as a present lighting environment.

### Claim 43

An identification environment determining method comprising:
receiving images;
identifying a person using said images by recognizing a face of said person;
informing a user about a result of identification;
inputting information obtained by having the user judge whether the result of identification is correct;
determining a lighting environment on the basis of the image information acquired in said receiving said images, and classifying the lighting environment into a corresponding lighting category;
detecting and extracting a face area from the image acquired at said receiving said images, and converting the face area into characteristic data to execute identification;
storing, when acquiring a characteristic data group of a face per person, the characteristic data group in each lighting category;
compiling a recognition dictionary using characteristic data per lighting category; and
storing, per lighting category, the recognition dictionary compiled in said compiling said recognition dictionary;
selecting, from recognition dictionaries stored at the dictionary storing step, a recognition dictionary of a same lighting category as the lighting category determined in said determining said lighting environment; and
executing identification of characteristic data to determine a user name on the basis of the recognition dictionary selected in said selecting said recognition dictionary.

### Claim 44

An identification environment determining method comprising:
receiving images;
identifying a person using said images by recognizing a face of said person;
informing a user about a result of said identifying said person;
inputting information obtained by having the user judge whether the result of identification is correct;
determining a lighting environment on the basis of the image information acquired in said receiving said images, and classifying the lighting environment into a corresponding lighting category;
recording, with respect to each lighting category, identifying performance information including the number of times where face identification has succeeded and the number of times where face identification has been executed;
determining whether a present lighting environment is suitable for identification on the basis of the identifying performance information;
controlling a robot; and
moving the robot to a position where identifying performance is higher on the basis of the identifying performance information.
