# US 2021/0345016 A1 · Computer vision based extraction and overlay for instructional augmented reality

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0345016 A1
- Document ID: US20210345016A1
- Status: Published application
- Kind: A1
- Application number: 15/929,472
- Filed: 2020-05-04
- Published: 2021-11-04
- Earliest claimed: 2020-05-04
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Aiko Nakano; Diane Wang
- Web page: https://patentgrep.com/patent/US20210345016A1

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06F 16/7328: {Query by example, e.g. a complete video frame or video sequence (graphical querying G06F16/7335)}
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 18/2413: based on distances to training or reference patterns
- G06Q 30/0643: {graphically representing goods, e.g. 3D product representation}
- G06T 7/11: Region-based segmentation
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 40/168: {Feature extraction; Face representation}
- G09B 5/065: {Combinations of audio and video presentations, e.g. videotapes, videodiscs, television systems}
- H04N 21/4307: {Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen}
- H04N 21/8153: {comprising still images, e.g. texture, background image}
- H04N 21/8456: {by decomposing the content in the time domain, e.g. in time segments}
- G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Systems and methods are described that utilize one or more processors to obtain a plurality of segments of a first media content item, extract, from a first segment in the plurality of segments, a plurality of image frames associated with a plurality of tracked movements of at least one object represented in the extracted image frames, compare, objects represented in the image frames extracted from the first segment to tracked objects in a second media content item. In response to detecting that at least one of the tracked objects is similar to at least one object in the plurality of extracted image frames, generating virtual content depicting the plurality of tracked movements from the first segment being performed on the at least one tracked object in the second media content item and triggering rendering of the virtual content as an overlay on the at least one tracked object.

## Claims

### Claim 1

A computer-implemented method carried out by at least one processor, the method comprising:
obtaining a plurality of segments of a first media content item;
extracting, from a first segment in the plurality of segments, a plurality of image frames, the plurality of image frames being associated with a plurality of tracked movements of at least one object represented in the extracted image frames;
comparing, objects represented in the image frames extracted from the first segment to tracked objects in a second media content item;
in response to detecting that at least one of the tracked objects in the second media content item is similar to at least one object in the plurality of extracted image frames, generating, based on the extracted plurality of image frames, virtual content depicting the plurality of tracked movements from the first segment being performed on the at least one tracked object in the second media content item; and
triggering rendering of the virtual content as an overlay on the at least one tracked object in the second media content item.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
extracting, from the plurality of segments, a second segment from the first media content item, the second segment having a timestamp after the first segment; and
generating, using the extracted at least one image frame from the second segment of the first media content item, virtual content that depicts the at least one image frame from the second segment on the at least one tracked object in the second media content item.

### Claim 3 (depends on 2)

The method of claim 2, wherein the at least one image frame from the second segment depicts a visual result associated with the at least one object in the extracted image frames.

### Claim 4 (depends on 1)

The method of claim 1, wherein a computer vision system is employed by the at least one processor to:
analyze the first media content item to determine which of the plurality of segments to extract and which of the plurality of image frames to extract; and
analyze the second media content item to determine which object corresponds to the at least one object in the plurality of extracted image frames of the first media content item.

### Claim 5 (depends on 1)

The method of claim 1, wherein the detecting that the at least one tracked object in the second media content item is similar to the at least one object in the plurality of extracted image frames includes comparing a shape of the at least one tracked object to the shape of the at least one object in the plurality of extracted image frames, and wherein the generated virtual content is depicted on the at least one tracked object according to the shape of the at least one object in the plurality of extracted image frames.

### Claim 6 (depends on 1)

The method of claim 1, wherein triggering rendering of the virtual content as an overlay on the at least one tracked object in the second media content item includes synchronizing the rendering of the virtual content on the second media content item with a timestamp associated with the first segment.

### Claim 7 (depends on 1)

The method of claim 1, wherein:
the plurality of tracked movements correspond to instructional content in the first media content item; and
the plurality of tracked movements are depicted as the virtual content, the virtual content illustrating performance of the plurality of tracked movements on the at least one object in the plurality of extracted image frames in the second media content item.

### Claim 8

A system comprising:
an image capture device associated with a computing device;
at least one processor; and
memory storing instructions that, when executed by the at least one processor, cause the system to:
obtain a plurality of segments of a first media content item;
extract, from a first segment in the plurality of segments, a plurality of image frames, the plurality of image frames being associated with a plurality of tracked movements of at least one object represented in the extracted image frames;
compare, objects represented in the image frames extracted from the first segment to tracked objects in a second media content item;
in response to detecting that at least one of the tracked objects in the second media content item is similar to at least one object in the plurality of extracted image frames, generating, based on the extracted plurality of image frames, virtual content depicting the plurality of tracked movements from the first segment being performed on the at least one tracked object in the second media content item; and
trigger rendering of the virtual content as an overlay on the at least one tracked object in the second media content item.

### Claim 9 (depends on 8)

The system of claim 8, further comprising:
extracting, from the plurality of segments, a second segment from the first media content item, the second segment having a timestamp after the first segment; and
generating, using the extracted at least one image frame from the second segment of the first media content item, virtual content that depicts the at least one image frame from the second segment on the at least one tracked object in the second media content item.

### Claim 10 (depends on 9)

The system of claim 9, wherein the at least one image frame from the second segment depicts a visual result associated with the at least one object in the extracted image frames.

### Claim 11 (depends on 8)

The system of claim 8, wherein the system further includes a computer vision system employed by the at least one processor to:
analyze the first media content item to determine which of the plurality of segments to extract and which of the plurality of image frames to extract; and
analyze the second media content item to determine which object corresponds to the at least one object in the plurality of extracted image frames of the first media content item.

### Claim 12 (depends on 8)

The system of claim 8, wherein the detecting that the at least one tracked object in the second media content item is similar to the at least one object in the plurality of extracted image frames includes comparing a shape of the at least one tracked object to the shape of the at least one object in the plurality of extracted image frames, and wherein the generated virtual content is depicted on the at least one tracked object according to the shape of the at least one object in the plurality of extracted image frames.

### Claim 13 (depends on 8)

The system of claim 8, wherein triggering rendering of the virtual content as an overlay on the at least one tracked object in the second media content item includes synchronizing the rendering of the virtual content on the second media content item with a timestamp associated with the first segment.

### Claim 14 (depends on 8)

The system of claim 8, wherein:
the plurality of tracked movements correspond to instructional content in the first media content item, and
the plurality of tracked movements are depicted as the virtual content, the virtual content illustrating performance of the plurality of tracked movements on the at least one object in the plurality of extracted image frames in the second media content item.

### Claim 15

A computer readable medium tangibly embodied on a non-transitory computer-readable medium and comprising instructions that, when executed, are configured to cause at least one processor to:
obtain a plurality of segments of a first media content item;
extract, from a first segment in the plurality of segments, a plurality of image frames, the plurality of image frames being associated with a plurality of tracked movements of at least one object represented in the extracted image frames;
compare, objects represented in the image frames extracted from the first segment to tracked objects in a second media content item;
in response to detecting that at least one of the tracked objects in the second media content item is similar to at least one object in the plurality of extracted image frames, generating, based on the extracted plurality of image frames, virtual content depicting the plurality of tracked movements from the first segment being performed on the at least one tracked object in the second media content item; and
trigger rendering of the virtual content as an overlay on the at least one tracked object in the second media content item.

### Claim 16 (depends on 15)

The computer readable medium of claim 15, wherein the instructions, when executed, are configured to cause the at least one processor to perform the steps of claim 15 for each of the obtained plurality of segments of the first media content item.

### Claim 17 (depends on 15)

The computer readable medium of claim 15, wherein a computer vision system is employed by the at least one processor to:
analyze the first media content item to determine which of the plurality of segments to extract and which of the plurality of image frames to extract; and
analyze the second media content item to determine which object corresponds to the at least one object in the plurality of extracted image frames of the first media content item.

### Claim 18 (depends on 15)

The computer readable medium of claim 15, wherein the detecting that the at least one tracked object in the second media content item is similar to the at least one object in the plurality of extracted image frames includes comparing a shape of the at least one tracked object to the shape of the at least one object in the plurality of extracted image frames, and wherein the generated virtual content is depicted on the at least one tracked object according to the shape of the at least one object in the plurality of extracted image frames.

### Claim 19 (depends on 15)

The computer readable medium of claim 15, wherein triggering rendering of the virtual content as an overlay on the at least one tracked object in the second media content item includes synchronizing the rendering of the virtual content on the second media content item with a timestamp associated with the first segment.

### Claim 20 (depends on 15)

The computer readable medium of claim 15, wherein:
the plurality of tracked movements correspond to instructional content in the first media content item; and
the plurality of tracked movements are depicted as the virtual content, the virtual content illustrating performance of the plurality of tracked movements on the at least one object in the plurality of extracted image frames in the second media content item.
