# US 2004/0080507 A1 · Freely specifiable real-time control

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2004/0080507 A1
- Document ID: US20040080507A1
- Status: Published application
- Kind: A1
- Application number: 10/386,693
- Filed: 2003-03-13
- Published: 2004-04-29
- Earliest claimed: 2000-09-13
- Assignees: 3DConnexion GmbH
- Inventors: Bernd Von Prittwitz
- Web page: https://patentgrep.com/patent/US20040080507A1

## CPC

- G10H 1/0008: {Associated control or indicating means}
- G06F 3/033: Pointing devices displaced or positioned by the user {, e.g. mice, trackballs, pens or joysticks}; Accessories therefor (digitisers characterised by the transducing means G06F3/041)
- G10H 2220/096: using a touch screen
- G10H 2220/315: for joystick-like proportional control of musical input; Videogame input devices used for musical input or control, e.g. gamepad, joysticks
- G10H 2220/321: Garment sensors, i.e. musical control means with trigger surfaces or joint angle sensors, worn as a garment by the player, e.g. bracelet, intelligent clothing
- G10H 2220/401: 3D sensing, i.e. three-dimensional (x, y, z) position or movement sensing
- G10H: ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Freely specifiable real-time control of animated graphics, video and/or audio sequences with the aid of controllable virtual objects. The invention relates to a method for graphic visualisation and real-time control of virtual objects for generating and/or influencing image sequences or sound sequences. Said method allows for the manipulation, control and influencing, of objects represented on at least one screen via interactive remote control of a user in the properties and/or actions of said objects, in a comfortable and reliable manner. The invention more specifically relates to a method for animating and controlling the movements of an articulated object in real-time; a method for controlling, in a freely specifiable manner, animated graphics, video or audio data with the help of optical or acoustic parameters; a computer software product for carrying out such a method, in addition to a system for controlling the movement of virtual objects in real-time. In order to control the movement of a virtual articular object in real-time, a model of a moving parameter (106), in addition to a surface model (108) defined for said object control signals (116) are inputted, using a 3D input device, in real-time. Additionally, the skeletal modified data is calculated on the basis of the parameterised control signals (116) and at least one parameterised moving model (106). The surface is calculated on the basis of the skeletal modified data and at least one surface model (108). Cyclic repetition of said steps enables a real-time movement control to be obtained.

## Claims

### Claim 1

A process for real-time motion control of an articulated object, wherein the following steps are carried out:
a) definition of at least one parametrised motion pattern (106) for the articulated object,
b) definition of at least one parametrised surface pattern (108) for the articulated object,
c) input of parametrised control signals (116) by means of a 3D input device (104),
d) computation of skeletal amendment data on the basis of the parametrised control signals (116) and also on the basis of at least one parametrised motion pattern (106), whereby the skeletal amendment data reproduce translatory and/or rotatory amendments to the position and/or orientation of the articulations of the articulated object,
e) computation of the surface of the articulated object on the basis of the skeletal amendment data and also on the basis of at least one surface pattern (108), steps c), d) and e) being repeated cyclically, in order to obtain a real-time motion control of the articulated object in a manner dependent on the control signals (116).

### Claim 2 (depends on 1)

Process according to claim 1,
characterised in that
furthermore sound signals (224) are generated by linkage of stored digital, parametrised sound patterns (110) with parametrised control signals (116).

### Claim 3

Process according to one of the preceding claims,
characterised in that
time derivatives of the parametrised control signals (116) can be ascertained and processed.

### Claim 4

Process according to one of the preceding claims,
characterised in that
the assignment of the parametrised control signals (116) or their derivatives to the motion patterns (106), surface patterns (108) or sound patterns (110) can be freely specified.

### Claim 5

Process according to one of the preceding claims,
characterised in that
furthermore parametrised illumination patterns (112) are saved which are used for generating illumination effects by means of appropriate linkages with parametrised control signals (116).

### Claim 6

A process for freely specifiable real-time control of animated graphics, video sequences and/or audio sequences,
characterised in that
for the purpose of generating motion data, surface data and/or sound data at least one predefined parametrised motion pattern (106), surface pattern (108) and/or sound pattern (110) is saved,
parametrised control signals (116) having various degrees of freedom are evaluated by a 3D input device (104) and used for generating motion data (310), surface data (312) and/or sound data (314) with the aid of the saved motion patterns (106), surface patterns (108) and/or sound patterns (110) and
the assignment of parametrised control signals (116) each having one degree of freedom to a particular motion pattern (106), surface pattern (108) and/or sound pattern (110) is freely specifiable.

### Claim 7 (depends on 6)

Process according to claim 6,
characterised in that
furthermore time derivatives of the parametrised control signals (116) can be assigned to a particular motion pattern (106), surface pattern (108) and/or sound pattern (110).

### Claim 8 (depends on 6, 7)

Process according to claim 6 or 7,
characterised in that
coupling parameters of the assignment of parametrised control signals (116), such as, for example, attenuation and amplification, are freely specifiable.

### Claim 9

A computer-software product,
characterised in that
it implements a process according to one of the exemplary embodiments described above when it is loaded in a memory of a computer.

### Claim 10

A system for real-time motion control of virtual or real articulated objects, said system being provided with a memory in which the following are saved:
at least one parametrised motion pattern (106) and
at least one parametrised surface pattern (108) of an articulated object,
a 3D input device (104) for the input of parametrised control signals (116) for real-time motion control,
a motion generator (204) for computing skeletal amendment data on the basis of the parametrised control signals (116) and on the basis of at least one parametrised motion pattern (106), whereby the skeletal amendment data reproduce translatory and/or rotatory amendments to the position and/or orientation of the articulations of the articulated object, and also
a surface generator (206) for computing the surface of the articulated object on the basis of the skeletal amendment data and the parametrised surface pattern (108).

### Claim 11 (depends on 10)

System according to claim 10,
characterised by
a sound generator (210) for generating sound signals by linkage of at least one digital parametrised sound pattern (110) with parametrised control signals (116).

### Claim 12 (depends on 10, 11)

System according to one of claims 10 or 11,
characterised by
a free-specification module (202) for ascertaining and processing time derivatives of the parametrised control signals (116).

### Claim 13 (depends on 10, 11, 12)

System according to one of claims 10 to 12,
characterised by
a free-specification module (202) for optional assignment of the parametrised control signals (116) or their derivatives for the motion data (310), surface data (312) and/or sound data (314).

### Claim 14

A system for freely specifiable real-time control of animated graphics, video sequences and/or audio sequences, comprising
a memory in which for the generation of motion data (310), surface data (312), sound data (314) and/or illumination data (316) at least one predefined parametrised motion pattern (106), surface pattern (108), sound pattern (110) and/or illumination pattern (112) is saved, and also
an arithmetic unit for evaluating parametrised control signals (116) having various degrees of freedom from a 3D input device (104) for generating motion data (310), surface data (312), sound data (314) and/or illumination data (316) with the aid of the saved motion patterns (106), surface patterns (108), sound patterns (110) and/or illumination patterns (112),
furthermore comprising a free-specification module (202) for freely specifiable assignment of parametrised control signals (116) each having one degree of freedom to a particular motion pattern (106), surface pattern (108), sound pattern (110) and/or illumination pattern (112).

### Claim 15 (depends on 14)

System according to claim 14,
characterised in that
the free-specification module (202) furthermore enables the assignment of time derivatives of the parametrised control signals (116) to a particular motion pattern (106), surface pattern (108), sound pattern (110) and/or illumination pattern (112).

### Claim 16 (depends on 14, 15)

System according to claim 14 or 15,
characterised in that
the free-specification module (202) enables a freely specifiable setting of coupling parameters of the assignment of parametrised control signals (116) such as, for example, attenuation and amplification.

### Claim 17

A system for real-time control of acoustic parameters of digitised sound patterns (410), comprising
a computer, the control unit (401) of which is conductively connected to at least one electronic musical instrument (402) via a MIDI interface (404a+b) and
a 3D input device (104) for the input of parametrised control signals (116)
wherein the digitised sound patterns (410)
are stored digitally in the memory of at least one electronic musical instrument (402) and
can be generated and played back on at least one electronic musical instrument (402),
characterised in that
individual or several acoustic parameters of the digitally stored sound patterns (410) can be generated or influenced in real time by means of a movement of the 3D input device (104).

### Claim 18 (depends on 21)

System according to claim 21,
characterised in that
control elements, keys, switches, knobs and/or sliding controls of a virtual mixing console are provided for the purpose of generating or influencing individual or several acoustic parameters of the digitally stored sound patterns (410).

### Claim 19 (depends on 18)

System according to claim 18,
characterised in that
the controlled acoustic parameters of a particular digitised sound pattern (410) are pitches, note-values or rest-values, loudness-levels, tempi, articulation instructions, timbres, pedal effects, vibrato effects, chorus effects, echo effects, overtone effects and/or other special effects.

### Claim 20 (depends on 17, 18, 19)

System according to one of claims 17 to 19,
characterised in that
time derivatives of the parametrised control signals (116) can be ascertained and processed.

### Claim 21 (depends on 17, 18, 19, 20)

System according to one of claims 17 to 20,
characterised in that
the assignment of parametrised control signals (116) each having one degree of freedom to one or more parameters of a particular digitised sound pattern (410) is freely specifiable.

### Claim 22

A system for real-time control of acoustic parameters of digitised sound patterns (512), comprising
a computer, the control unit (501) of which is connected to at least one sound card (510) and also to at least one tone generator (502) and
a 3D input device (104) for the input of parametrised control signals (116) for the real-time motion control of objects,
wherein the digitised sound patterns (512)
are stored digitally in the memory of at least one sound card (510) of the computer and
can be generated and played back on at least one tone generator (502) of the same computer,
characterised in that
individual or several acoustic parameters of the digitally stored sound patterns (512) can be generated or influenced in real time by means of a movement of the 3D input device (104).

### Claim 23 (depends on 22)

System according to claim 22,
characterised in that
furthermore time derivatives of the parametrised control signals (116) can be assigned to one or more parameters of a particular digitised sound pattern (512).

### Claim 24 (depends on 22, 23)

System according to one of claims 22 or 23,
characterised in that
coupling parameters of the assignment of parametrised control signals (116) are freely specifiable.

### Claim 25

A process for real-time control of acoustic parameters of digitally recorded sound patterns (410 or 512) which
are digitally stored in accordance with the MIDI standard in the memory of at least one sound card (510) of a computer and/or of at least one electronic musical instrument (402) and
can be played back in accordance with the MIDI standard on at least one tone generator (502) of the same computer or on at least one electronic musical instrument (402),
wherein the control unit (401 or 501) of the computer
is conductively connected to at least one sound card (510) and to at least one tone generator (502) or to at least one electronic musical instrument (402) via a MIDI interface (404a+b or 504a+b) and
is conductively connected to a 3D input device (104) for the input of parametrised control signals (116) for real-time motion control of virtual objects,
characterised in that
individual or several acoustic parameters of the digitally stored sound patterns (410 or 512) can be generated or influenced in real time by a movement of the 3D input device (104).

### Claim 26 (depends on 25)

Process according to claim 25,
characterised in that
the virtual objects which are moved for the purpose of generating or influencing individual or several acoustic parameters of the digitally stored sound patterns (410 or 512) are the control elements, keys, switches, knobs and/or sliding controls of a virtual mixing console.

### Claim 27 (depends on 25, 26)

Process according to one of claims 25 or 26,
characterised in that
the controlled acoustic parameters of a particular digitised sound pattern (410 or 512) are pitches, note-values or rest-values, loudness-levels, tempi, articulation instructions, timbres, pedal effects, vibrato effects, chorus effects, echo effects, overtone effects and/or other special effects.

### Claim 28 (depends on 25, 26, 27)

Process according to one of claims 25 to 27,
characterised in that
coupling parameters of the assignment of parametrised control signals (116) are freely specifiable.
