# US 2016/0292131 A1 · System, method and apparatus for generating hand gesture animation determined on dialogue length and emotion

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2016/0292131 A1
- Document ID: US20160292131A1
- Status: Published application
- Kind: A1
- Application number: 14/901,013
- Filed: 2014-06-27
- Published: 2016-10-06
- Earliest claimed: 2013-06-27
- Assignees: Plotagon AB Corporation
- Applicants: Plotagon AB
- Inventors: Linus LANGELS; Philip EDNER; Jonas LOFGREN
- Web page: https://patentgrep.com/patent/US20160292131A1

## CPC

- G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/04817: using icons (graphical or visual programming using iconic symbols G06F8/34)
- G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 40/103: Formatting, i.e. changing of presentation of documents (automatic justification G06F40/189; automatic line break hyphenation G06F40/191)
- G06F 40/12: Use of codes for handling textual entities
- G06F 40/151: Transformation
- G06F 40/166: Editing, e.g. inserting or deleting
- G06F 40/289: Phrasal analysis, e.g. finite state techniques or chunking
- G06F 40/55: Rule-based translation
- G06T 13/20: Three-dimensional [3D] animation
- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- G06T 19/003: {Navigation within 3D models or images}
- G11B 27/34: Indicating arrangements {(indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D)}
- G06F 40/174: Form filling; Merging
- G06T 2213/04: Animation description language
- G06T 2213/08: Animation software package
- G06T 2213/12: Rule based animation
- H04N 5/222: Studio circuitry; Studio devices; Studio equipment (cameras or camera modules comprising electronic image sensors, or control thereof H04N23/00)
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

System, method and apparatuses directed to a paradigm of manuscript generation and manipulation combined with contemporaneous or simultaneous visualization of the text or other media being entered by the creator with emotion and mood of the characters being conveyed graphically through rendering. Through real time calculations, respective characters are graphically depicted speaking and interacting physically with other characters, pursuant to directive found in a manuscript text.

## Claims

### Claim 1

A rendering tool for manuscript visualization comprising:
an input interface, a user using said input interface to input at least one instruction for a manuscript, said manuscript describing a virtual environment representative of at least one scene in said manuscript with at least one character therein, said at least one instruction containing a character command; and
a synchronizer, said synchronizer synchronizing said character command in said at least one instruction to a rendering of a particular character in said virtual environment,
whereby a visual feature of said particular character is altered by said character command.

### Claim 2 (depends on 1)

The rendering tool according to claim 1, wherein said at least one instruction for said manuscript is selected from the group consisting of scenes, characters, verbal and non-verbal dialogue, human and animal movement, inanimate objects, action, interactions, audio, visuals, music, animation, body movement, lip movement, facial movement and combinations thereof.

### Claim 3 (depends on 1)

The rendering tool according to claim 1, wherein said character command is to speak dialogue, and said synchronizer synchronizes text of said dialogue to lip movement of said particular character in said virtual environment.

### Claim 4 (depends on 3)

The rendering tool according to claim 3, wherein said synchronizer passes said text of said dialogue through a text-to-speech system, generating an audio file and metadata associated therewith.

### Claim 5 (depends on 4)

The rendering tool according to claim 4, wherein said text-to-speech system generates phoneme data, said phoneme data being divided into a plurality of respective channels.

### Claim 6 (depends on 5)

The rendering tool according to claim 5, wherein said phoneme data comprises phonemes selected from the group consisting of WQ, I, Upper Lip up/down, Lower Lip up/down, and jawbone movement governing mouth opening.

### Claim 7 (depends on 5)

The rendering tool according to claim 5, wherein said synchronizer maps said phoneme data from said plurality of respective channels to lip movement of said particular character.

### Claim 8 (depends on 7)

The rendering tool according to claim 7, wherein said synchronizer employs an algorithm based on a Facial Action Coding System (FACS) in the mapping, wherein the length of the audio file is employed to map lip animation to said phoneme data.

### Claim 9 (depends on 7)

The rendering tool according to claim 7, wherein said synchronizer employs a plurality of blendshapes to control facial features of said particular character, and mimic speaking said dialogue thereby.

### Claim 10 (depends on 1)

The rendering tool according to claim 1, wherein said character command is to alter body position, and said synchronizer synchronizes text of said manuscript to alter at least one body position of said particular character.

### Claim 11 (depends on 10)

The rendering tool according to claim 10, wherein said character command to alter body position includes a body part selected from the group of consisting of head, upper body, lower body and combinations thereof.

### Claim 12 (depends on 10)

The rendering tool according to claim 10, wherein said character command is to gesture, and said synchronizer synchronizes text of said manuscript to alter at least one body position of said particular character to visualize said gesture.

### Claim 13 (depends on 12)

The rendering tool according to claim 12, wherein said gesture is a head gesture.

### Claim 14 (depends on 13)

The rendering tool according to claim 13, wherein said head gesture is selected from the group consisting of nods, winks, smirks, shaking of head no, disgust, alarm, terror and combinations thereof.

### Claim 15 (depends on 12)

The rendering tool according to claim 12, wherein said gesture is an arm or leg gesture.

### Claim 16 (depends on 12)

The rendering tool according to claim 12, wherein said gesture is a hand gesture.

### Claim 17 (depends on 12)

The rendering tool according to claim 12, wherein said gesture is of at least one finger.

### Claim 18 (depends on 1)

The rendering tool according to claim 1, wherein said character command is to alter a facial expression, and said synchronizer synchronizes text of said manuscript to alter at least one facial expression of said particular character.

### Claim 19 (depends on 1)

The rendering tool according to claim 1, wherein said character command is to alter an emotion or mood of a particular character, and said synchronizer synchronizes at least one facial expression of said particular character with an emotion or mood sample stored on a database.

### Claim 20 (depends on 1)

The rendering tool according to claim 1, wherein said synchronizer is connected to an animation database, said animation database containing a plurality of standard character attributes therein.

### Claim 21 (depends on 20)

The rendering tool according to claim 20, wherein a plurality of retargetable characters in said virtual environment have said standard character attributes.

### Claim 22 (depends on 1)

The rendering tool according to claim 1, wherein said particular character in a particular scene in said manuscript has a first animation state associated therewith, said particular character having a second animation state in a subsequent scene in said manuscript, and
wherein said first animation state of said particular character is transformed and blended into said second animation state.

### Claim 23

A method for rendering images in a visualization tool comprising:
inputting, by a user into an input interface, at least one instruction for a manuscript, said manuscript describing a virtual environment representative of at least one scene in said manuscript with at least one character therein, said at least one instruction containing a character command;
synchronizing, by a synchronizer, said character command in said at least one instruction to a rendering of a particular character in said virtual environment; and
visualizing, in a visualization interface, said particular character altered by said character command.
