# US 2018/0322680 A1 · Creating a mixed-reality video based upon tracked skeletal features

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0322680 A1
- Document ID: US20180322680A1
- Status: Published application
- Kind: A1
- Application number: 15/589,632
- Filed: 2017-05-08
- Published: 2018-11-08
- Earliest claimed: 2017-05-08
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: John MCELMURRAY; Gustaf PETTERSSON; Karim BENCHEMSI
- Web page: https://patentgrep.com/patent/US20180322680A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06T 11/60: Creating or editing images; Combining images with text
- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44)
- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G06V 40/166: {using acquisition arrangements}
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- G06V 40/176: {Dynamic expression}
- G06V 40/23: {Recognition of whole body movements, e.g. for sport training}
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Technologies related to generating mixed reality video are described herein. Video frames capture a human, and positions of skeletal features of the human are estimated based solely upon the video frames. A mixed reality video is generated, wherein the mixed reality video includes the video combined with an animation, wherein the animation is based upon movement of the human in the video as represented by the estimated positions of the skeletal features of the human.

## Claims

### Claim 1

A computing system comprising:
a processor; and
memory storing instructions that, when executed by the processor, cause the processor to perform acts comprising:
estimating positions of skeletal features of a human body captured in frames of video, wherein the positions of the skeletal features of the human body in the frames are estimated based solely upon the frames;
generating mixed reality video frames based upon the estimated positions of the skeletal features of the human body, wherein the mixed reality video frames include the frames combined with an animation; and
storing the mixed reality video frames in a computer-readable storage device.

### Claim 2 (depends on 1)

The computing system of claim 1 being a mobile telephone, the mobile telephone further comprises a camera that generates the frames.

### Claim 3 (depends on 1)

The computing system of claim 1, the acts further comprising:
estimating expressions of a face of the human in the frames, wherein the expressions of the face of the human are estimated based solely upon the frames; and
generating the mixed reality video based upon the estimated expressions of the face of the human in the frames, wherein the animation comprises a character with facial expressions that correspond to the estimated expressions of the face of the human.

### Claim 4 (depends on 1)

The computing system of claim 1, the acts further comprising:
receiving a video file, wherein the video file includes the frames; and
including the mixed reality video frames in a mixed reality video files.

### Claim 5 (depends on 1)

The computing system of claim 1, further comprising:
prior to generating the mixed reality video frames, receiving a selection of the animation from amongst several possible animations in a library, wherein the mixed reality frames are generated to include the selected animation.

### Claim 6 (depends on 5)

The computing system of claim 5, wherein the selected animation corresponds only to a portion of the human body, and wherein the animation, when included in the mixed reality video frames, covers only the portion of the human body.

### Claim 7 (depends on 5)

The computing system of claim 5, wherein the selected animation corresponds to an entirety of the human body, and wherein the animation, when included in the mixed reality video frames, covers the entirety of the human body.

### Claim 8 (depends on 5)

The computing system of claim 5, wherein the selected animation contacts the human body while leaving the human body visible in the mixed reality video frames.

### Claim 9 (depends on 1)

The computing system of claim 1, wherein estimating the positions of the skeletal features of the human body captured in the frames comprises:
in a frame in the frames, generating an initial estimate of a position of a skeletal feature of the human body;
determining that the initial estimate is incorrect based upon the initial estimate of the position of the skeletal feature in the frame relative to an estimated position of another skeletal feature in the frame; and
updating the initial estimate of the skeletal feature in the frame responsive to determining that the initial estimate is incorrect.

### Claim 10 (depends on 9)

The computing system of claim 9, wherein updating the initial estimate of the skeletal feature comprises interpolating between estimated positions of the skeletal feature in frames that are adjacent to the frame in the video.

### Claim 11 (depends on 1)

The computing system of claim 1, wherein estimating the positions of the skeletal features of the human body captured in the video frame comprises:
in a frame in the frames, generating an initial estimate of a position of a skeletal feature of the human body;
determining that the initial estimate is incorrect based upon an estimated position of the feature in a second frame that is adjacent the frame in the frames; and
updating the initial estimate responsive to determining that the initial estimate is incorrect.

### Claim 12 (depends on 11)

The computing system of claim 11, wherein determining that the initial estimate is incorrect comprises:
identifying a distance between the initial estimate and the estimated position of the skeletal feature in the second frame;
comparing the distance with a threshold; and
determining that the distance is greater than the threshold.

### Claim 13

A method executed on a computing device, the method comprising:
estimating positions of skeletal features of a human in frames of video, wherein the positions of the skeletal features are estimated based solely upon the frames of the video; and
generating a mixed reality video based upon the frames of the video and the estimated positions of the skeletal features of the human, wherein the mixed reality video includes an animation combined with the frames of the video, wherein the animation corresponds to movement of the human as represented by the estimated positions of the skeletal features of the human in the frames of the video.

### Claim 14 (depends on 13)

The method of claim 13, wherein the computing device is a mobile telephone, the method further comprising:
receiving the video from a camera of the mobile telephone; and
receiving a selection of the animation from a user of the mobile telephone, wherein the positions of the skeletal features are estimated responsive to receiving the selection of the animation.

### Claim 15 (depends on 13)

The method of claim 13, further comprising:
receiving the video from a client computing device by way of a network connection; and
responsive to generating the mixed reality video, transmitting the mixed reality video to the client computing device.

### Claim 16 (depends on 13)

The method of claim 13, further comprising:
receiving a selection of the animation from a library of selectable animations; and
generating the mixed reality video responsive to the animation being selected from the library of selectable animations.

### Claim 17 (depends on 16)

The method of claim 16, wherein the animation solely covers an extremity of the human body in the mixed reality video.

### Claim 18 (depends on 16)

The method of claim 16, wherein the animation covers an entirety of the human body in the mixed reality video.

### Claim 19

A computer-readable storage medium comprising instructions that, when executed by a processor, cause the processor to perform acts comprising:
receiving a video file, wherein the video file comprises frames, and further wherein the frames include a human body;
estimating positions of skeletal features in the frames based solely upon the frames;
receiving a selection of an animation; and
based upon the selection of the animation, the frames, and the estimated positions of the skeletal features in the frames, generating a mixed reality video file, wherein the mixed reality video file includes the frames combined with the animation, wherein the animation reflects movement of the human body as represented by the estimated positions of the skeletal features in the frames.

### Claim 20 (depends on 19)

The computer-readable storage of claim 19 being included in a mobile telephone that includes a camera that is used to generate the video file.
