# US 2021/0174095 A1 · Interacting with visitors of a connected home environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174095 A1
- Document ID: US20210174095A1
- Status: Published application
- Kind: A1
- Application number: 17/116,662
- Filed: 2020-12-09
- Published: 2021-06-10
- Earliest claimed: 2019-12-09
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Liana Kong; John Jordan Nold; Sahana Mysore; Ruiyi Song
- Web page: https://patentgrep.com/patent/US20210174095A1

## CPC

- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06F 3/165: {Management of the audio stream, e.g. setting of volume, audio stream path}
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V 40/172: {Classification, e.g. identification}
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G08B 3/10: using electric transmission; using electromagnetic transmission
- G08B 13/196: using television cameras
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS

## Abstract

The disclosure describes techniques for interacting with visitors at a visitor interaction system. A method includes obtaining video data captured by the visitor interaction system, identifying an approach of a person, and analyzing the video data to determine an identity of the person. The method includes automatically and without user intervention determining that the person belongs to a visitor groups of a plurality of visitor groups. Each of the visitor groups corresponds to a response model that includes a plurality of response actions. At least one of the response actions includes an autonomous response action. The method includes identifying a response model corresponding to the visitor groups and initiating an autonomous response action associated with the response model. Initiating the autonomous response action including presenting a message to the person, monitoring a response to the message, and sending a report message to the user via a client device.

## Claims

### Claim 1

A method comprising:
obtaining video data captured in a field of view by a camera of a visitor interaction system;
identifying, based on the video data, an approach of a person within the field of view of the camera;
analyzing the video data to determine an identity of the person; and
automatically and without user intervention:
determining, based on the identity of the person, that the person belongs to one of a plurality of predefined visitor groups, each of the predefined visitor groups corresponding to a respective predefined response model that includes a plurality of respective response actions, wherein the plurality of respective response actions include an autonomous response action that is executable by the visitor interaction system autonomously, without requiring intervention of a user of the visitor interaction system;
identifying a first predefined response model corresponding to the one of the plurality of predefined visitor groups; and
initiating, by the visitor interaction system, a first autonomous response action of the plurality of respective response actions associated with the first predefined response model, including:
outputting a first message via a component of the visitor interaction system;
monitoring a first response of the person to the first message; and
sending a first report message to the user via a client device registered to the user, the first report message including a representation of the video data, approach data for the approach, and a summary of the first message and the first response.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
after providing the first autonomous response action, receiving from the person a first reply and context information related to the first reply;
determining a second response action, wherein the second response action is based on the first reply, the context information, and the first predefined response model;
initiating the second response action at the camera, including providing a second response message; and
sending a second report message to the user via the client device registered to the user, the second report message including at least the second response action.

### Claim 3 (depends on 1)

The method of claim 1, wherein the respective predefined response model for each of the predefined visitor groups is modified using one or more previous response actions for the respective predefined response model, and wherein the one or more previous response actions for the respective predefined response model include one or more user selected response actions, one or more user rejected response actions, one or more user defined response actions, and one or more repeated response actions.

### Claim 4 (depends on 1)

The method of claim 1, wherein the first autonomous response action includes an audible alert, a spoken message, a request to record a message from the person, a text notification to one or more devices, a call to the police, and an alert to other devices of the visitor interaction system.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
prior to providing the first message:
receiving an inquiry from the person;
determining whether the first message includes confidential information based on the identity of the person; and
responsive to determining that the first message includes confidential information, refraining from providing the first message,
wherein the first message is output in response to determining that the first message does not include confidential information.

### Claim 6 (depends on 1)

The method of claim 1, wherein the first message includes a first question to the person, the method further comprising:
receiving an answer to the first question;
generating, based on the first predefined response model and the answer, a follow-up question; and
providing the follow-up question.

### Claim 7 (depends on 1)

The method of claim 1,
wherein the person is determined to belong to an expected person group, and
wherein the first autonomous response action includes at least one of a group of operations consisting of:
unlocking a door lock; and
identifying a personal message based on the identity of the person and enabling the camera to broadcast the personal message to the person.

### Claim 8 (depends on 1)

The method of claim 1,
wherein the person is determined to belong to a deliveryman group, and
wherein the method further comprises, in response to the first report message, receiving a user input for electronically signing for a package delivery.

### Claim 9 (depends on 1)

The method of claim 1, wherein analyzing the video data to determine the identity of the person in the video data includes determining that the identity of the person is an intruder based on one or more suspicious behaviors of the person, the method further comprising:
responsive to determining that the person is the intruder, determining that the person belongs to an intruder group;
identifying the first predefined response model corresponding to the intruder group; and
updating, based on the one or more suspicious behaviors of the person, the first autonomous response action corresponding to the first predefined response model.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
collecting context information related to the approach, wherein the context information related to the approach of the person includes an identified delivery package carried by the person in the video data, and
wherein the first message includes a prerecorded computer-generated instruction, the method further comprising:
determining, based on the identified delivery package, that the person belongs to a deliveryman group of the plurality of predefined visitor groups; and
enabling broadcasting of the prerecorded computer-generated instruction to the person via a speaker of the camera.

### Claim 11 (depends on 1)

The method of claim 1, further comprising:
collecting context information related to the approach, wherein the context information related to the approach of the person includes an identified product being sold;
determining, based on one or more products being sold, that the identity of the person belongs to a salesperson group of the predefined visitor groups; and
customizing, based on the one or more product types being sold, the first message.

### Claim 12

A visitor interaction system comprising:
a camera that captures video data in a field of view of the camera;
one or more processors; and
a memory coupled to the one or more processors, the memory storing one or more programs that, when executed by the one or more processors, cause the one or more processors to:
identify, using the video data, an approach of a person within the field of view of the camera;
analyze the video data to determine an identity of the person; and
automatically and without user intervention:
determine, based on the identity of the person, that the person belongs to one of a plurality of predefined visitor groups, each of the predefined visitor groups corresponding to a respective predefined response model that includes a plurality of respective response actions, wherein the plurality of respective response actions include an autonomous response action that is executable by the visitor interaction system autonomously, without requiring intervention of a user of the visitor interaction system;
identify a first predefined response model corresponding to the one of the plurality of predefined visitor groups; and
initiate a first autonomous response action of the plurality of respective response actions associated with the first predefined response model, including:
presenting a first message to the person via a component of the visitor interaction system;
monitoring a first response of the person to the first message; and
sending a first report message to the user via a client device registered to the user, the first report message including a representation of the video data, approach data for the approach, and a summary of the first message and the first response.

### Claim 13 (depends on 12)

The visitor interaction system of claim 12, wherein the one or more programs further cause the one or more processors to:
after providing the first autonomous response action, receive a first reply and context information related to the first reply;
determine a second response action, wherein the second response action is based on the first reply, the context information, and the first predefined response model;
initiate the second response action at the camera, including providing a second response message; and
send a second report message to the user via the client device registered to the user, the second report message including at least the second response action.

### Claim 14 (depends on 12)

The visitor interaction system of claim 12, wherein the respective predefined response model for each of the predefined visitor groups is modified using one or more previous response actions for the respective predefined response model, and wherein the one or more previous response actions for the respective predefined response model include one or more user selected response actions, one or more user rejected response actions, one or more user defined response actions, and one or more repeated response actions.

### Claim 15 (depends on 12)

The visitor interaction system of claim 12, wherein the first message includes a first question to the person, and wherein the one or more programs further cause the one or more processors to:
receive an answer to the first question;
generate, based on the first predefined response model and the answer, a follow-up question; and
output the follow-up question.

### Claim 16 (depends on 12)

The visitor interaction system of claim 12,
wherein the person is determined to belong to an expected person group, and
wherein the first autonomous response action includes at least one of a group of operations consisting of:
unlocking a door lock; and
identifying a personal message based on the identity of the person and enabling the camera to broadcast the personal message to the person.

### Claim 17 (depends on 12)

The visitor interaction system of claim 12,
wherein the person is determined to belong to a deliveryman group, and
wherein the one or more programs further cause the one or more processors to, in response to the first report message, receive a user input for electronically signing for a package delivery.

### Claim 18

A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors of a visitor interaction system, cause the one or more processors to:
obtain video data captured in a field of view by a camera of the visitor interaction system;
identify, using the video data, an approach of a person within the field of view of the camera;
analyze the video data to determine an identity of the person; and
automatically and without user intervention:
determine, based on the identity of the person, that the person belongs to one of a plurality of predefined visitor groups, each of the predefined visitor groups corresponding to a respective predefined response model that includes a plurality of respective response actions, wherein the plurality of respective response actions include an autonomous response action that is executable by the visitor interaction system autonomously, without requiring intervention of a user of the visitor interaction system;
identify a first predefined response model corresponding to the one of the plurality of predefined visitor groups; and
initiate a first autonomous response action of the plurality of respective response actions associated with the first predefined response model, including:
presenting a first message to the person via a component of the visitor interaction system;
monitoring a first response of the person to the first message; and
sending a first report message to the user via a client device registered to the user, the first report message including a representation of the video data, approach data for the approach, and a summary of the first message and the first response.

### Claim 19 (depends on 18)

non-transitory computer-readable storage medium of claim 18, wherein the instructions further cause the one or more processors to:
after providing the first autonomous response action, receive from the person a first reply and context information related to the first reply;
determine a second response action, wherein the second response action is based on the first reply, the context information, and the first predefined response model;
initiate the second response action at the camera, including providing a second response message; and
send a second report message to the user via the client device registered to the user, the second report message including at least the second response action.

### Claim 20 (depends on 18)

The non-transitory computer-readable storage medium of claim 18, wherein the first message includes a first question to the person, wherein the instructions further cause the one or more processors to:
receive an answer to the first question;
generate, based on the first predefined response model and the answer, a follow-up question; and
output the follow-up question.
