# US 2009/0276802 A1 · Avatars in social interactive television

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2009/0276802 A1
- Document ID: US20090276802A1
- Status: Published application
- Kind: A1
- Application number: 12/113,704
- Filed: 2008-05-01
- Published: 2009-11-05
- Earliest claimed: 2008-05-01
- Assignees: At&T Knowledge Ventures, L.P.
- Inventors: Brian Scott Amento; Christopher Harrison; Larry Stead
- Web page: https://patentgrep.com/patent/US20090276802A1

## CPC

- H04N 7/17318: {Direct or substantially direct transmission and handling of requests}
- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- H04N 21/23412: {for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects}
- H04N 21/42203: {sound input device, e.g. microphone}
- H04N 21/4223: Cameras (H04N23/00 takes precedence)
- H04N 21/44012: {involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs}
- H04N 21/4788: communicating with other users, e.g. chatting {(arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104)}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Virtual environments are presented on displays along with multimedia programs to permit viewers to participate in a social interactive television environment. The virtual environments include avatars that are created and maintained in part using continually updated animation data that may be captured from cameras that monitor viewing areas in a plurality of sites. User input from the viewers may be processed in determining which viewers are presented in instances of the virtual environment. Continually updating the animation data results in avatars accurately depicting a viewer's facial expressions and other characteristics. Presence data may be collected and used to determine when to capture background images from a viewing area that may later be subtracted during the capture of animation data. Speech recognition technology may be employed to provide callouts within a virtual environment.

## Claims

### Claim 1

A method of presenting avatars with a multimedia program, the method comprising:
continually monitoring a viewer to result in animation input data;
displaying an avatar corresponding to the viewer in a virtual environment that includes further avatars corresponding to further viewers; and
continually updating the avatar in response to the animation input data.

### Claim 2 (depends on 1)

The method of claim 1, wherein continually monitoring the viewer includes capturing facial expressions, wherein the animation input data includes facial expression data.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
detecting laughter from the viewer; and wherein the animation input data includes laughter data.

### Claim 4 (depends on 1)

The method of claim 1, wherein the virtual environment is superimposed on the multimedia program for presentation on a display.

### Claim 5 (depends on 1)

The method of claim 1, wherein the virtual environment is placed within a perimeter outside the presentation of a multimedia program on a display.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
subtracting background data to result in animation input data that corresponds substantially to the monitored viewer and not a viewing area around the monitored viewer.

### Claim 7 (depends on 6)

The method of claim 6, further comprising:
detecting a level of motion in the viewing area around the monitored viewer;
in response to a low level of motion, capturing background data from the viewing area; and
in response to a high level of motion, continually monitoring the viewer to result in animation input data achieved in part by excluding the captured background data.

### Claim 8 (depends on 6)

The method of claim 6, further comprising:
detecting the viewer entering the viewing area; and
simulating the avatar entering the virtual environment in response to the detecting the viewer entering.

### Claim 9 (depends on 8)

The method of claim 8, further comprising:
detecting the viewer leaving the viewing area; and
simulating the avatar leaving the virtual environment in response to the detecting the viewer leaving.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
receiving updated animation data corresponding to the further viewers; and
updating the further avatars in response to the updated animation data.

### Claim 11 (depends on 10)

The method of claim 10, wherein receiving updated animation data occurs continually during a viewing session.

### Claim 12

A computer program product stored on one or more computer readable media for simultaneously presenting avatars and a multimedia program, the computer program product having instructions operable for:
during a viewing session, receiving animation input data harvested by monitoring a viewer;
displaying an avatar corresponding to the viewer in a virtual environment that includes further avatars corresponding to further viewers; and
during the animation session, continually updating the avatar in response to the animation input data.

### Claim 13 (depends on 12)

The computer program product of claim 12, wherein the received animation input data includes captured facial expression data.

### Claim 14 (depends on 13)

The computer program product of claim 13, further comprising instructions operable for:
detecting laughter from the viewer, and wherein the animation input data includes laughter data.

### Claim 15 (depends on 12)

The computer program product of claim 12, wherein the virtual environment is superimposed on the multimedia program for presentation on a display.

### Claim 16 (depends on 12)

The computer program product of claim 12, wherein the virtual environment is placed within a perimeter outside the presentation of a multimedia program on a display.

### Claim 17 (depends on 12)

The computer program product of claim 12, further comprising instructions operable for:
subtracting background data to result in animation input data that corresponds substantially to the monitored viewer and not a viewing area around the monitored viewer.

### Claim 18 (depends on 17)

The computer program product of claim 17, further comprising instructions operable for:
detecting a first level of motion in the viewing area for the monitored person;
in response to the first level of motion, capturing background data from the viewing area;
detecting a second level of motion in the viewing area to determine a viewing session is occurring, wherein continually monitoring the person to result in animation input data is accomplished in part by excluding the captured background data.

### Claim 19 (depends on 17)

The computer program product of claim 17, further comprising instructions operable for:
detecting the viewer entering the viewing area; and
simulating the avatar entering the virtual environment in response to the detecting the viewer entering.

### Claim 20 (depends on 19)

The computer program product of claim 19, further comprising instructions operable for:
detecting the viewer leaving the viewing area; and
simulating the avatar leaving the virtual environment in response to the detecting the viewer leaving.

### Claim 21 (depends on 12)

The computer program product of claim 12, further comprising instructions operable for:
receiving updated animation data corresponding to the further viewers; and
updating the further avatars in response to the updated animation data.

### Claim 22 (depends on 16)

The computer program product of claim 16, wherein receiving updated animation data occurs continually during a viewing session.

### Claim 23

A service for presenting a virtual environment containing a plurality of avatars with the presentation of a multimedia program, the service comprising:
receiving from a plurality of viewers a plurality of requests to be presented with instances of a virtual environment that includes a plurality of avatars that correspond to the plurality of viewers;
translating animation data generated from individual viewers of the plurality of viewers into a plurality of synthetic avatar actions taken by corresponding avatars of the plurality of avatars, wherein the synthetic avatar actions are intended to mimic actions by corresponding viewers of the plurality of viewers; and
presenting the virtual environment for display, wherein the virtual environment includes the plurality of avatars that are animated with the synthetic avatar actions.

### Claim 24 (depends on 23)

The service of claim 23, wherein the animation data includes facial animation data.
