# US 8,276,056 B1 · Systems and methods for playing, browsing and interacting with MPEG-4 coded audio-visual objects

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 8,276,056 B1
- Document ID: US8276056B1
- Status: Granted
- Kind: B1
- Application number: 11/849,399
- Filed: 2007-09-04
- Granted: 2012-09-25
- Earliest claimed: 1998-01-28
- Assignees: AT&T Intellectual Property II, L.P.
- Inventors: Barin Geoffry Haskell; Atul Puri; Robert Lewis Schmidt
- Web page: https://patentgrep.com/patent/US8276056B1

## CPC

- H04N 21/47205: {for manipulating displayed content, e.g. interacting with MPEG-4 objects, editing locally}
- H04N 19/25: with scene description coding, e.g. binary format for scenes [BIFS] compression
- H04N 19/90: using coding techniques not provided for in groups H04N19/10-H04N19/85, e.g. fractals
- H04N 21/23412: {for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects}
- H04N 21/234318: {by decomposing into objects, e.g. MPEG-4 objects}
- H04N 21/43072: {of multiple content streams on the same device}
- H04N 21/44012: {involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs}
- H04N 21/8547: involving timestamps for synchronizing content
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A number of novel configurations for MPEG-4 playback, browsing and user interaction are disclosed. MPEG-4 playback systems are not simple extensions of MPEG-2 playback systems, but, due to object based nature of MPEG-4, present new opportunities and challenges in synchronized management of independent coded objects as well as scene composition and presentation. Therefore, these configurations allow significantly new and enhanced multimedia services and systems. In addition, MPEG-4 aims for an advanced functionality, called Adaptive Audio Visual Session (AAVS) or MPEG-J. Adaptive Audio Visual Session (AAVS) (i.e., MPEG-AAVS, MPEG-Java or MPEG-J) requires, in addition to the definition of configurations, a definition of an application programming interface (API) and its organization into Java packages. Also disclosed are concepts leading to definition of such a framework.

## Claims

### Claim 1

A device that processes MPEG-4 data having a scene description graph and data related to at least one object, the device comprising:
means for interacting with a user configured for basic user interaction via control signals from at least one user input device;
a binary format of scene description graph interpreter connected to the means for interacting with the user for interpreting the MPEG-4 data to yield interpreted MPEG-4 data; and
a media decoder, compositor and renderer that receives the interpreted MPEG-4 data and presents at least one object on the means for interacting with the user,
wherein at least one of the binary format of scene description graph interpreter and the media decoder, compositor and renderer comprises at least one programmer interface accessible via the means for interacting with the user, and
wherein at least one of the means for interacting with the user and the binary format of scene description graph interpreter is further configured for local user interaction via at least one script obtained from a JavaScript interpreter.

### Claim 2 (depends on 1)

The device of claim 1, wherein the means for interacting with the user comprises an audio visual object demultiplexer format of a scene browser.

### Claim 3 (depends on 1)

The device of claim 1, wherein the MPEG-4 data having a scene description graph and data related to at least one object comprises MPEG-4 coded content, the MPEG-4 coded content including at least one of audio media, visual media and synthetic media.

### Claim 4 (depends on 1)

The device of claim 1, wherein the binary format of scene description graph interpreter invokes at least one media decoder based on the scene description graph.

### Claim 5 (depends on 1)

The device of claim 1, wherein the binary format of scene description graph interpreter further comprises:
at least one object node; and
at least one corresponding object programmer interface, each object programmer interface connected to a corresponding one of the at least one object node.

### Claim 6 (depends on 5)

The device of claim 5, wherein the media decoders, compositor and renderer further comprises:
at least one decoder, each decoder connected to a corresponding one the at least one object programmer interface; and
a scene compositor connected to the at least one decoder.

### Claim 7 (depends on 6)

The device of claim 6, wherein the scene compositor is connected to the audiovisual object demultiplexer and binary format of scene browser and the audiovisual object demultiplexer and binary format of scene browser is connected to the scene compositor.

### Claim 8 (depends on 1)

The device of claim 1, wherein the binary format of scene description graph interpreter further comprises:
a VideoObject2D node or a MovieTexture node connected to the audiovisual object demultiplexer and binary format of scene browser;
a video object programmer interface connected to the VideoObject2D or MovieTexture node;
an AudioSource node connected to the means for interacting with the user;
an audio object programmer interface connected to the AudioSource node;
an ImageTexture node connected to the means for interacting with the user; and
an image object programmer interface connected to the ImageTexture node.

### Claim 9 (depends on 8)

The device of claim 8, wherein the media decoders, compositor and renderer further comprise:
a video object decoder connected to the video object programmer interface;
an audio object decoder connected to the audio object programmer interface;
an image object decoder connected to the image object programmer interface; and
a scene compositor connected to each of the video object decoder, the audio object decoder, and the image object decoder.

### Claim 10 (depends on 9)

The device of claim 9, wherein the scene compositor is connected to the means for interacting with the user and the means for interacting with the user is connected to the scene compositor.

### Claim 11 (depends on 8)

The device of claim 8, wherein the binary format of scene description graph interpreter further comprises:
a proto node; and
a native proto programmer interface connected to the proto node.

### Claim 12 (depends on 11)

The device of claim 11, wherein the media decoders, compositor and renderer further comprise:
a video object decoder connected to the video object programmer interface;
an audio object decoder connected to the audio object programmer interface;
an image object decoder connected to the image object programmer interface;
a native proto implementation connected to the native proto programmer interface; and
a scene compositor connected to each of the native proto implementation, the video object decoder, the audio object decoder, and the image object decoder.

### Claim 13 (depends on 12)

The device of claim 12, wherein the scene compositor is connected to the means for interacting with the user and the means for interacting with the user is connected to the scene compositor.

### Claim 14 (depends on 8)

The device of claim 8, wherein the binary format of scene description graph interpreter further comprises:
a script node;
an interpreter programmer interface connected to the script node; and
a scripting interface.

### Claim 15 (depends on 14)

The device of claim 14, wherein the media decoders, compositor and renderer further comprise:
a video object decoder connected to the video object programmer interface;
an audio object decoder connected to the audio object programmer interface;
an image object decoder connected to the image object programmer interface;
a native proto implementation connected to the native proto programmer interface;
a JavaScript interpreter connected to the interpreter programmer interface;
a Java interpreter connected to the interpreter programmer interface; and
a scene compositor connected to each of the scripting interface, the native proto implementation, the video object decoder, the audio object decoder, and the image object decoder.

### Claim 16 (depends on 15)

The device of claim 15, wherein the scene compositor is connected to the means for interacting with the user and the means for interacting with the user is connected to the scene compositor.

### Claim 17 (depends on 15)

The device of claim 15, wherein the JavaScript interpreter and the Java interpreter are connected to the scripting interface.

### Claim 18 (depends on 17)

The device of claim 17, wherein the multimedia browser supports programmatic behavior and interaction via the JavaScript interpreter and the Java interpreter to modify the scene internally.

### Claim 19 (depends on 14)

The device of claim 14, wherein the scripting interface is connected to the means for interacting with the user.

### Claim 20 (depends on 1)

The device of claim 1, wherein at least one control signal from at least one user input device is provided to the audiovisual object demultiplexer and binary format of scene browser and to the binary format of scene description graph interpreter.

### Claim 21 (depends on 1)

The device of claim 1, further comprising an adaptive audio visual session connected to the means for interacting with the user.

### Claim 22 (depends on 21)

The device of claim 21, wherein at least one control signal from at least one user input device is provided to the adaptive audio visual session.

### Claim 23 (depends on 21)

The device of claim 21, wherein the adaptive audio visual session comprises:
an adaptive audio visual session external interface;
a browser specific binding connected to the adaptive audio visual session external interface; and
a browser programmer interface connected to the browser specific binding.

### Claim 24 (depends on 23)

The device of claim 23, wherein the means for interacting with the user is connected to the browser programmer interface.

### Claim 25 (depends on 23)

The device of claim 23, wherein at least one control signal is provided to the adaptive audio visual session external interface.

### Claim 26 (depends on 25)

The device of claim 25, wherein the at least one control signal is an adaptive audio visual session external script or applet.

### Claim 27 (depends on 1)

The device of claim 1, wherein the multimedia browser supports programmatic behavior and interaction via at least one of Java and JavaScript to modify the scene internally.

### Claim 28 (depends on 1)

The device of claim 1, wherein the multimedia browser supports external interface for BIFS player control in response to changing resources and support of user interaction.

### Claim 29

A method of providing an audio-visual object browser that operates on MPEG-4 data having a scene description graph and data related to at least one object, the method comprising:
providing an audiovisual object demultiplexer and binary format of scene browser for basic user interaction with a user via control signals from at least one user input device;
interpreting a binary format of scene description graph associated with the MPEG-4 data from the audiovisual object demultiplexer and binary format of scene browser;
decoding, composing and rendering the interpreted binary format of scene description graph on the audiovisual object demultiplexer and binary format of scene browser;
providing at least one programmer interface accessible via the audiovisual object demultiplexer and binary format of scene browser for adjusting at least one of the interpreting and the decoding, composing and rendering; and
configuring at least one of the audiovisual object demultiplexer and binary format of scene browser and the binary format of scene description graph interpreter for local user interaction via at least one script obtained from a JavaScript interpreter.

### Claim 30

A method of rendering MPEG-4 data having a scene description graph and data related to at least one object, the method comprising:
interacting with a user through an audiovisual object demultiplexer and binary format of scene browser via a basic user interaction using control signals from at least one input device;
interpreting a binary format of scene description graph associated with the MPEG-4 data from the audiovisual object demultiplexer and binary format of scene browser;
decoding, composing and rendering the interpreted binary format of scene description graph on the audiovisual object demultiplexer and binary format of scene browser;
providing at least one programmer interface accessible via the audiovisual object demultiplexer and binary format of scene browser for adjusting at least one of the interpreting and the decoding, composing and rendering; and
interacting with at least one of the audiovisual object demultiplexer and binary format of scene browser and the binary format of scene description graph interpreter via a local user interaction using at least one script obtained from a JavaScript interpreter.