# US 2022/0035443 A1 · Systems and methods for generating complementary data for visual display

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0035443 A1
- Document ID: US20220035443A1
- Status: Published application
- Kind: A1
- Application number: 17/276,548
- Filed: 2019-07-22
- Published: 2022-02-03
- Earliest claimed: 2018-09-21
- Assignees: Penumbra, Inc.
- Applicants: Penumbra, Inc.
- Inventors: Hans Winold; Joseph White; Ryan Corniel; Mark Samuel Gutentag; John Lockhart
- Web page: https://patentgrep.com/patent/US20220035443A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- A61B 5/1123: {Discriminating type of movement, e.g. walking or running (A61B5/1116, A61B5/112 take precedence)}
- A61B 5/1128: {using image analysis (A61B5/1127 takes precedence)}
- A61B 5/744: {Displaying an avatar, e.g. an animated cartoon character}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06V 20/20: in augmented reality scenes
- G06V 40/23: {Recognition of whole body movements, e.g. for sport training}
- G09B 19/00: Teaching not covered by other main groups of this subclass (teaching or practice apparatus for gun-aiming or gun-laying F41G3/26)
- G16H 20/30: relating to physical therapies or activities, e.g. physiotherapy, acupressure or exercising
- G16H 40/63: for local operation
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G16H 50/50: for simulation or modelling of medical disorders
- G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

A system for generating complementary data for a visual display that includes one or plurality of wearable sensors that collect tracking data for a users position, orientation, and movement. The sensor(s) are in communication with at least one processor that may be configured to receive tracking data, identify missing tracking data, generate complementary data to substitute for missing tracking data, generate a 3D model comprised of tracking data and complementary data, and communicate the 3D model to a display. Complementary tracking data may be generated by comparison to a key pose library, by comparison to past tracking data, or by inverse kinematics.

## Claims

### Claim 1

A system for generating complementary tracking data, the system comprising:
one or a plurality of wearable sensors configured to be disposed on a subject, wherein the wearable sensors are configured to collect and transmit tracking data;
a processor in communication with a memory that includes processor executable instructions, wherein the execution of the instructions cause the system to:
(a) receive tracking data from the one or the plurality of wearable sensors;
(b) map the tracking data onto a 3D model, wherein the 3D model is a virtual representation of an extent and a motion of the subject;
(c) identify an incomplete portion of the 3D model, the incomplete portion comprising a portion of the model not mapped to the tracking data;
(d) generate complementary tracking data to substitute for at least a portion of the incomplete portion; and
(e) include the complementary tracking data in the 3D model.

### Claim 2 (depends on 1)

The system of claim 1, wherein the complementary data is generated by comparing available tracking data to a key pose library.

### Claim 3 (depends on 3)

The system of claim 3, wherein tracking data is compared to blend spaces between two or more key poses in the key pose library.

### Claim 4 (depends on 3)

The system of claim 3, wherein tracking data is compared to key poses of the library of key poses to determine a strongest match.

### Claim 5 (depends on 4)

The system of claim 4, wherein the determination of a match weights similarities of joints and body parts closest to the incomplete portion more heavily than joints and body parts distant from the incomplete portion.

### Claim 6 (depends on 1)

The system of claim 1, wherein the memory includes processor executable instructions that cause the system to access one or a series of prior 3D models, wherein the prior 3D models were previously communicated for display.

### Claim 7 (depends on 6)

The system of claim 6, wherein the complementary tracking data is generated by identifying a cluster of repetitive tracking data among a series of prior 3D models, matching tracking data to the cluster, and generating complementary data similar to the matched portion of the cluster.

### Claim 8 (depends on 6)

The system of claim 6, wherein the complementary tracking data is generated by identifying a cluster of repetitive tracking data among a series of prior 3D models, determining where available tracking data best fits in the cluster of repetitive tracking data, and generating complementary tracking data that mimics the cluster of repetitive tracking data.

### Claim 9 (depends on 6)

The system of claim 6, wherein the processor is configured to analyze the one or a series of prior 3D models for gesture triggers, wherein the identification of a gesture trigger causes the system to communicate a series of updated 3D models that are at least partially blended with a pre-recorded gesture animation.

### Claim 10 (depends on 1)

The system of claim 1, wherein complementary tracking data is generated with a FABRIK solver when tracking data for an end effector is available and with a cluster function or key pose match when tracking data for an end effector is unavailable.

### Claim at position 11

-20. (canceled)

### Claim 21

A system for generating complementary tracking data, the system comprising a processor in communication with a memory that includes processor executable instructions, the instructions comprising:
a set of tracking data, wherein the tracking data relates to the position or motion of one or more parts of the user;
a 3D model, wherein the 3D model is a virtual representation of an extent and a motion of a user;
a library, wherein the library comprises a set of poses, gestures, or both;
a set of complementary tracking data, wherein the set of complementary tracking data comprises a least a portion of at least one selected pose or gesture from the library;
a combined model, the combined model comprising the set of tracking data and the set of complementary data.

### Claim at position 13

-26. (canceled)

### Claim 27 (depends on 21)

The system of claim 21, wherein the complementary data is generated by a comparison of the set of tracking data to a key pose library.

### Claim 28 (depends on 27)

The system of claim 27, wherein the comparison comprises spaces between two or more key poses in the key pose library and the complementary data is generated to blend between those spaces.

### Claim at position 16

-34. (canceled)

### Claim 35 (depends on 21)

The system of claim 21, wherein complementary tracking data is generated with a FABRIK solver when tracking data for an end effector is available and with a cluster function or key pose match when tracking data for an end effector is unavailable.

### Claim at position 18

-39. (canceled)

### Claim 40

A system for generating complementary data for a visual display, the system comprising:
(a) one or more electromagnetic emitters and one or more electromagnetic sensors configured to be selectively placed on one or more tracked body parts; and
(b) at least one processor in communication with the visual display, the one or more electromagnetic emitters, and the one or more electromagnetic sensors and configured to receive tracking data from the one or more electromagnetic emitters and the one or more electromagnetic sensors, and to generate complementary display data comprising projected motion not within the tracking data and based upon a library of potential motions.

### Claim at position 20

-57. (canceled)
