# US 2015/0105155 A1 · System and Method for Visualizing Synthetic Objects Within Real-World Video Clip

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2015/0105155 A1
- Document ID: US20150105155A1
- Status: Published application
- Kind: A1
- Application number: 14/579,535
- Filed: 2014-12-22
- Published: 2015-04-16
- Earliest claimed: 2011-09-08
- Assignees: Paofit Holdings Pte. Ltd.
- Applicants: Paofit Holdings Pte Ltd.
- Inventors: Marc Scott Hardy; Andrei Richard Frank
- Web page: https://patentgrep.com/patent/US20150105155A1

## CPC

- A63F 13/655: by importing photos, e.g. of the player
- A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence)
- A63F 13/245: specially adapted to a particular type of game, e.g. steering wheels
- A63F 13/335: using Internet
- A63F 13/35: Details of game servers
- A63F 13/52: involving aspects of the displayed game scene
- A63F 13/53: involving additional visual information provided to the game scene, e.g. by overlay to simulate a head-up display [HUD] or displaying a laser sight in a shooting game
- A63F 13/65: automatically by game devices or servers from real world data, e.g. measurement in live racing competition
- A63F 13/816: Athletics, e.g. track-and-field sports
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- A63F 13/61: using advertising information
- A63F 13/79: involving player-related data, e.g. identities, accounts, preferences or play histories
- A63F 2300/203: Image generating hardware
- A63F 2300/303: for displaying additional data, e.g. simulating a Head Up Display
- A63F 2300/5553: user representation in the game field, e.g. avatar
- A63F 2300/6009: for importing or creating game content, e.g. authoring tools during game development, adapting content to different platforms, use of a scripting language to create content
- A63F 2300/66: for rendering three dimensional images
- A63F 2300/6653: for altering the visibility of an object, e.g. preventing the occlusion of an object, partially hiding an object
- A63F 2300/69: Involving elements of the real world in the game world, e.g. measurement in live races, real video
- A63F 2300/8005: Athletics
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Embodiments of the present invention provide a system and a method for visualizing synthetic objects within a real-world video clip. The embodiments allow interactive, synthetic objects or characters to appear as if they are “embedded” inside or form part of a video clip. These objects move independently of the environment within the video clip and are controlled by a user. In one embodiment the video frames are set as textures in the furthermost layer in a 3D environment viewer or 3D “engine”. In another embodiment separate layers of video frames are superimposed with the synthetic objects on an output screen and synchronizing the movements and the field of view settings to give the perception that only one camera is used.

## Claims

### Claim 1

A method of visualizing controllable virtual 3D objects within a real-world video clip using a transparent 3D scene generated using the real-world video clip, comprising:
selecting frames from a real-world video clip to render as a background layer of a 3D environment based upon a speed input using a 3D game engine, where the 3D environment further comprises at least one controllable virtual 3D object and a 3D transparent scene including at least one transparent 3D object;
moving the at least one controllable virtual 3D object within the 3D environment independently of the speed input using the 3D game engine; and
rendering the 3D environment based upon the 3D space locations of the at least one controllable virtual 3D object within the 3D environment and the 3D space locations of the at least one transparent 3D object in the 3D transparent scene using the 3D game engine so that portions of at least one controllable virtual 3D object that are located behind a transparent 3D object are occluded by the background layer.

### Claim 2 (depends on 1)

The method of claim 1, wherein moving the at least one controllable virtual 3D object within the 3D environment is partly based on information received from one or more devices located at different networked locations.

### Claim 3 (depends on 1)

The method of claim 1 further comprising obtaining and displaying a browser layer overlaid on the background layer of the video clip, the browser layer used to render least one of a menu control, image, or video file.

### Claim 4 (depends on 1)

The method of claim 1 further comprising rendering a different real-world video within the background layer when a user changes a direction of a path in the real-world video.

### Claim 5 (depends on 1)

The method of claim 1, wherein movement of the at least one controllable virtual 3D object within the 3D environment is guided by a path obtained from the real-world video.

### Claim 6 (depends on 1)

The method of claim 1, wherein the at least one transparent 3D object is either statically positioned or dynamically moving within the 3D environment.

### Claim 7 (depends on 1)

The method of claim 1, wherein the background layer is one of a plurality of layers of the video clip and the plurality of layers of the video clip are synched in the 3D environment.

### Claim 8 (depends on 1)

The method of claim 1 further comprising providing one or more functionalities selected from the group consisting of point systems, characters, and sound effects for rendering within the 3D environment.

### Claim 9 (depends on 1)

The method of claim 1 further comprising overlaying a map that provides a real-world location of the real-world video being rendered within the background layer and providing a location of the at least one controllable virtual 3D object within the map in real-time.

### Claim 10 (depends on 1)

The method of claim 1, wherein the at least one transparent 3D object replicates a key object depicted within the real-world video clip.

### Claim 11 (depends on 1)

The method of claim 1, wherein the at least one controllable virtual 3D object is an avatar of a user, wherein the avatar moves based on activities of the user.

### Claim 12 (depends on 1)

The method of claim 1 further comprising extracting at least one of camera movement coordinates, path coordinates and object coordinates from the real-world video clip that provide depth information used in rendering the 3D environment.

### Claim 13 (depends on 1)

The method of claim 1 wherein the speed input causes the 3D game engine to select frames from the real-world video clip at a rate corresponding to a speed of movement through the scene captured by the real-world video clip.

### Claim 14 (depends on 13)

The method of claim 13 further comprising maintaining a pitch of the sound in the real-world video clip when adjusting the rate at which frames are selected from the real-world video clip in response to a change in the speed input.

### Claim 15 (depends on 1)

The method of claim 1 further comprising moving a controllable virtual 3D object to avoid hitting a transparent 3D object based on 3D space locations.

### Claim 16

A system for of visualizing controllable virtual 3D objects within a real-world video clip using a transparent 3D scene generated using the real-world video clip, comprising:
memory; and
one or more processors configured by processor executable instructions stored in the memory to render the video clip, the one or more processors being further configured by the processor executable instructions to:
select frames from a real-world video clip to render as a background layer of a 3D environment based upon a speed input using a 3D game engine, where the 3D environment further comprises at least one controllable virtual 3D object and a 3D transparent scene including at least one transparent 3D object;
move the at least one controllable virtual 3D object within the 3D environment independently of the speed input using the 3D game engine; and
render the 3D environment based upon the 3D space locations of the at least one controllable virtual 3D object within the 3D environment and the 3D space locations of the at least one transparent 3D object in the 3D transparent scene using the 3D game engine so that portions of at least one controllable virtual 3D object that are located behind a transparent 3D object are occluded by the background layer.

### Claim 17 (depends on 16)

The system of claim 16, wherein moving the at least one controllable virtual 3D object within the 3D environment is partly based on information received from one or more devices located at different networked locations.

### Claim 18 (depends on 16)

The system of claim 16, wherein the instructions when executed further configure the one or more processors to obtain and display an HTML browser layer overlaid on the background layer of the video clip, the HTML browser layer used to render least one of a menu control, image, or video file.

### Claim 19 (depends on 16)

The system of claim 16, wherein the instructions when executed further configure the one or more processors to render a different real-world video clip within the background layer when a user changes a direction of a path in the real-world video.

### Claim 20 (depends on 16)

The system of claim 16, wherein movement of the at least one controllable virtual 3D object within the 3D environment is guided by a path obtained from the real-world video.

### Claim 21 (depends on 16)

The system of claim 16, wherein the at least one transparent 3D object is either statically positioned or dynamically moving within the 3D environment.

### Claim 22 (depends on 16)

The system of claim 16, wherein the background layer is one of a plurality of layers of the video clip and the plurality of layers of the video clip are synched in the 3D environment.

### Claim 23 (depends on 16)

The system of claim 16, wherein the instructions when executed further configure the one or more processors to provide one or more functionalities selected from the group consisting of point systems, characters, and sound effects for rendering within the 3D environment.

### Claim 24 (depends on 16)

The system of claim 16, wherein the instructions when executed further configure the one or more processors to overlay a map that provides a real-world location of the real-world video being rendered within the background layer and provide a location of the at least one controllable virtual 3D object within the map in real-time.

### Claim 25 (depends on 16)

The system of claim 16, wherein the at least one transparent 3D object replicates a key object depicted within the real-world video clip.

### Claim 26 (depends on 16)

The system of claim 16, wherein the at least one controllable virtual 3D object is an avatar of a user, wherein the avatar moves based on activities of the user.

### Claim 27 (depends on 16)

The system of claim 16, wherein the instructions when executed further configure the one or more processors to extract at least one of camera movement coordinates, path coordinates and object coordinates from the real-world video clip that provide depth information used in rendering the 3D environment.

### Claim 28 (depends on 16)

The system of claim 16, wherein the speed input causes the 3D game engine to select frames from the real-world video clip at a rate corresponding to a speed of movement through the scene captured by the real-world video clip.

### Claim 29 (depends on 28)

The system of claim 28, wherein the instructions when executed further configure the one or more processors to maintain a pitch of the sound in the real-world video clip when adjusting the rate at which frames are selected from the real-world video clip in response to a change in the speed input.

### Claim 30 (depends on 16)

The system of claim 16, wherein the instructions when executed further configure the one or more processors to move a controllable virtual 3D object to avoid hitting a transparent 3D object based on 3D space locations.
