# US 2004/0025115 A1 · Method, terminal, browser application, and mark-up language for multimodal interaction between a user and a terminal

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2004/0025115 A1
- Document ID: US20040025115A1
- Status: Published application
- Kind: A1
- Application number: 10/603,687
- Filed: 2003-06-26
- Published: 2004-02-05
- Earliest claimed: 2002-08-05
- Assignees: Alcatel
- Inventors: Jurgen Sienel; Dieter Kopp
- Web page: https://patentgrep.com/patent/US20040025115A1

## CPC

- G06F 16/986: {Document structures and storage, e.g. HTML extensions}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A multimodal interaction between a user and a terminal takes place by using at least an input unit belonging to an input modality and an output unit belonging to a output modality, a mark-up language, extended with multimodal meta tags, and a browser application, which is capable of interpreting the mark-up language. The input/output unit and the browser application are provided at a terminal. According to a method, terminal and browser application of invention at least one input and/or output processing application is provided for processing data related to the browser application, wherein multimodal meta tags from the mark-up language are interpreted for controlling the multimodal interactions and processing data of the multimodal interaction with respect to the multimodal meta tags.

## Claims

### Claim 1

Method for multimodal interaction between an user and a terminal comprising an input unit, an output unit and a browser application, which is capable of interpreting a mark-up language, wherein the interaction takes place over the input unit, and/or the output unit and by using the browser application and the mark-up language, and wherein, the mark-up language comprises an extension of multimodal meta tags for multimodal interactions,
and wherein the method comprises the steps of:
interpreting multimodal meta tags from the mark-up language for controlling the multimodal interactions and
processing data of the multimodal interaction with respect to the multimodal meta tags by using at least one input and/or output processing application.

### Claim 2 (depends on 1)

Method according to claim 1, wherein the browser application is controlled corresponding to said multimodal meta tags and/or a communication between the browser application and the input and/or output processing applications is controlled corresponding to said multimodal meta tags.

### Claim 3 (depends on 1)

Method according to claim 1, wherein said at least one input and/or output processing application is one of the following applications:
handwriting recognition application
speech recognition application
eye movement recognition application
speech generation application
pointing recognition application.

### Claim 4 (depends on 1)

Method according to claim 1, wherein said input and/or output processing application is provided at the terminal.

### Claim 5 (depends on 1)

Method according to claim 1, wherein said input and/or output processing application is realised as an application having a distributed architecture.

### Claim 6 (depends on 1)

Method according to claim 1, comprising by rules determining the handling of a plurality of multimodal interactions being related to each other.

### Claim 7

Terminal for multimodal interaction between an user and said terminal comprising an input unit, an output unit and a browser application, which is capable of interpreting a mark-up language, wherein the interaction takes place over the input unit, and/or the output unit and by using the browser application and the mark-up language, and wherein said used mark-up language comprises an extension of multimodal meta tags for multimodal interactions,
said multimodal meta tags from the mark-up language are interpreted at the terminal for controlling the multimodal interactions, and said terminal comprises at least one input and/or output processing application for processing data of the multimodal interaction with respect to the multimodal meta tags.

### Claim 8 (depends on 7)

Terminal according to claim 7, wherein means are provided for determining and evaluating time information of said multimodal interaction.

### Claim 9

Browser application used for multimodal interaction between an user and a terminal comprising an input unit, an output unit, at least one input and/or output processing application for processing data of the multimodal interaction, and said browser application,
wherein the multimodal interaction takes place over the input unit, and/or the output unit and by using said browser application and a mark-up language comprising an extension of multimodal meta tags for multimodal interactions,
and wherein said browser application interprets the mark-up language with said multimodal meta tags,
said browser application is controlled corresponding to said multimodal meta tags and/or a communication between the browser application and the input and/or output processing applications is controlled corresponding to said multimodal meta tags.

### Claim 10

Mark-up language, in particular used for representing of information of a mark-up language document in connection with a browser application, wherein the mark-up language comprises meta tags specifying properties and values of said properties of the mark-up language document, and wherein said mark-up languages further comprises multimodal meta tags for controlling the multimodal interactions and processing data of the multimodal interaction with respect to the multimodal meta tags by using at least one input and/or output processing application.
