# US 2026/0065805 A1 · Cascade dual-decoder based sign language producing device, method, and recording medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0065805 A1
- Document ID: US20260065805A1
- Status: Published application
- Kind: A1
- Application number: 19/319,509
- Filed: 2025-09-04
- Published: 2026-03-05
- Earliest claimed: 2024-09-04
- Assignees: Ajou University Industry-Academic Cooperation Foundation
- Applicants: Ajou University Industry-Academic Cooperation Foundation
- Inventors: Tae-Sun CHUNG; Xiaohan MA; Rize JIN
- Web page: https://patentgrep.com/patent/US20260065805A1

## CPC

- G09B 21/009: {Teaching or communicating with deaf persons (G09B21/04 takes precedence)}
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)

## Abstract

Provided is a cascade dual-decoder based sign language producing device and method, and the device includes a text encoder configured to input a text sequence prepared in advance into at least one encoder block to output contextual features, a hand pose decoder configured to input the contextual features output from the text encoder and a hand pose sequence prepared in advance into at least one attention layer to output a hand-channel sign pose feature that aligns text and a hand motion, and a sign pose decoder configured to input the contextual features output from the text encoder and the hand pose decoder, the hand-channel sign pose feature, and a sign pose sequence prepared in advance into at least one attention layer to output a full-channel sign pose sequence in which the sign language is implemented as a hand element and a non-hand element.

## Claims

### Claim 1

A cascade dual-decoder based sign language producing device comprising:
a text encoder configured to input a text sequence prepared in advance into at least one encoder block to output a contextual feature;
a hand pose decoder configured to input the contextual feature output from the text encoder and a hand pose sequence prepared in advance into at least one attention layer to output a hand-channel sign pose feature that aligns text and hand motions; and
a sign pose decoder configured to input the contextual feature output from the text encoder and the hand pose decoder, the hand-channel sign pose feature, and a sign pose sequence prepared in advance into the at least one attention layer to output a full-channel sign pose sequence, wherein a sign language is implemented as a hand element and a non-hand element.

### Claim 2 (depends on 1)

The cascade dual-decoder based sign language producing device of claim 1, wherein
the text encoder is configured to generate a text sequence representation by performing a word-embedding on the text sequence and adding a Positional Encoding (PE) corresponding to the word-embedded text sequence, and
the PE is derived from a predefined sinusoidal function.

### Claim 3 (depends on 1)

The cascade dual-decoder based sign language producing device of claim 1, wherein
the hand pose decoder is configured to generate a hand pose representation by performing a hand-channel sign embedding on the hand pose sequence and adding a Counter Encoding (CE) to the embedded hand pose sequence, and
the CE represents time information for a hand-channel sign pose inference.

### Claim 4 (depends on 3)

The cascade dual-decoder based sign language producing device of claim 3, wherein the hand pose decoder is further configured to:
input the hand pose representation into a masked hand attention layer to model the hand pose representation,
input the modeled hand pose representation and the contextual feature from the text encoder into a text-hand attention layer to model dependencies between the text sequence and the hand pose sequence, and
input the modeled dependencies and the contextual feature into a feed forward layer to generate the hand-channel sign pose feature.

### Claim 5 (depends on 1)

The cascade dual-decoder based sign language producing device of claim 1, wherein
the sign pose decoder is configured to generate a sign pose representation by performing a full-channel sign embedding on the sign pose sequence and adding a Counter Encoding (CE) to the embedded sign pose sequence, and
the CE represents time information for a full-channel sign pose inference.

### Claim 6 (depends on 5)

The cascade dual-decoder based sign language producing device of claim 5, wherein the sign pose decoder is configured to:
model the sign pose representation by inputting the sign pose representation into a masked sign attention layer,
model dependencies between the text sequence and the sign pose sequence by inputting the modeled sign pose representation and the contextual feature from the text encoder into a text-sign attention layer,
align the hand-channel sign pose feature and a full-channel sign pose feature by inputting the modeled dependencies and the hand-channel sign pose feature from the hand pose decoder into a hand-sign attention layer, and
generate the full-channel sign pose sequence by inputting the aligned hand-channel sign pose feature and the full-channel sign pose feature into a feed forward layer and a linear layer.

### Claim 7 (depends on 1)

The cascade dual-decoder based sign language producing device of claim 1,
wherein the cascade dual-decoder based sign language producing device is trained through a space-time loss function, and
wherein the space-time loss function is derived from a sum of a spatial regression loss and a temporal continuity loss for each of the hand pose decoder and the sign pose decoder.

### Claim 8

A sign language producing method by a cascade dual-decoder-based sign language producing device, the sign language producing method comprising:
inputting, by a text encoder, a text sequence prepared in advance into at least one encoder block to output contextual feature;
inputting, by a hand pose decoder, the contextual feature output from the text encoder and a hand pose sequence prepared in advance into at least one attention layer to output a hand-channel sign pose feature that aligns text and hand motions; and
inputting, by a sign pose decoder, the contextual feature output from the text encoder and the hand pose decoder, the hand-channel sign pose feature, and a sign pose sequence prepared in advance into the at least one attention layer to output a full-channel sign pose sequence, a sign language is implemented as hand element and a non-hand element.

### Claim 9 (depends on 8)

The sign language producing method of claim 8, wherein the outputting of the contextual feature comprises: generating a text sequence representation by performing a word-embedding on the text sequence and adding a Positional Encoding (PE) corresponding to the word-embedded text sequence, and
wherein the PE is derived from a predefined sinusoidal function.

### Claim 10 (depends on 8)

The sign language producing method of claim 8, wherein the outputting of the hand-channel sign pose feature comprises: generating a hand pose representation by performing a hand-channel sign embedding on the hand pose sequence and adding a Counter Encoding (CE) to the embedded hand pose sequence, and
wherein the CE represents time information for a hand-channel sign pose inference.

### Claim 11 (depends on 10)

The sign language producing method of claim 10, wherein the outputting of the hand-channel sign pose feature further comprises:
inputting the hand pose representation into a masked hand attention layer to model the hand pose representation,
inputting the modeled hand pose representation and the contextual feature from the text encoder into a text-hand attention layer to model dependencies between the text sequence and the hand pose sequence, and
inputting the modeled dependencies and the contextual feature into a feed forward layer to generate the hand-channel sign pose feature.

### Claim 12 (depends on 8)

The sign language producing method of claim 8, wherein the outputting of the full-channel sign pose sequence comprises: generating a sign pose representation by performing a full-channel sign embedding on the sign pose sequence and adding a Counter Encoding (CE) to the embedded sign pose sequence, and
wherein the CE represents time information for a full-channel sign pose inference.

### Claim 13 (depends on 12)

The sign language producing method of claim 12, wherein the outputting of the full-channel sign pose sequence further comprises:
modeling the sign pose representation by inputting the sign pose representation into a masked sign attention layer,
modeling dependencies between the text sequence and the sign pose sequence by inputting the modeled sign pose representation and the contextual feature from the text encoder into a text-sign attention layer,
aligning the hand-channel sign pose feature and a full-channel sign pose feature by inputting the modeled dependencies and the hand-channel sign pose feature from the hand pose decoder into a hand-sign attention layer, and
generating the full-channel sign pose sequence by inputting the aligned hand-channel sign pose feature and the full-channel sign pose feature into a feed forward layer and a linear layer.

### Claim 14 (depends on 8)

The sign language producing method of claim 8,
wherein the cascade dual-decoder-based sign language producing device is trained through a space-time loss function, and
wherein the space-time loss function is derived from a sum of a spatial regression loss and a temporal continuity loss for each of the hand pose decoder and the sign pose decoder.

### Claim 15

A recording medium having recorded thereon a computer program for performing a sign language producing method by a cascade dual-decoder-based sign language producing device, wherein the sign language producing method comprises:
inputting, by a text encoder, a text sequence prepared in advance into at least one encoder block to output contextual feature;
inputting, by a hand pose decoder, the contextual feature output from the text encoder and a hand pose sequence prepared in advance into at least one attention layer to output a hand-channel sign pose feature that aligns text and hand motions; and
inputting, by a sign pose decoder, the contextual feature output from the text encoder and the hand pose decoder, the hand-channel sign pose feature, and a sign pose sequence prepared in advance into the at least one attention layer to output a full-channel sign pose sequence, a sign language is expressed by a hand element and a non-hand element.
