# US 9,751,015 B2 · Augmented reality videogame broadcast programming

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 9,751,015 B2
- Document ID: US9751015B2
- Status: Granted
- Kind: B2
- Application number: 14/339,133
- Filed: 2014-07-23
- Published: 2014-11-13
- Granted: 2017-09-05
- Earliest claimed: 2009-11-30
- Assignees: Disney Enterprises, Inc.
- Applicants: Disney Enterprises, Inc.
- Inventors: Michael Gay; Aaron Thiel
- Web page: https://patentgrep.com/patent/US9751015B2

## CPC

- A63F 13/655: by importing photos, e.g. of the player
- A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence)
- A63F 13/525: Changing parameters of virtual cameras
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- H04N 13/221: using the relative movement between cameras and objects
- H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence)
- H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
- A63F 13/812: Ball games, e.g. soccer or baseball
- A63F 13/86: Watching games played by other players
- A63F 2300/1093: using visible light
- A63F 2300/577: for watching a game played by other players
- A63F 2300/66: for rendering three dimensional images
- A63F 2300/69: Involving elements of the real world in the game world, e.g. measurement in live races, real video
- A63F 2300/8082: Virtual reality
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

There is provided a system and method for integrating a virtual rendering system and a video capture system using flexible camera control to provide an augmented reality. There is provided a method comprising receiving input data from a plurality of clients for modifying a virtual environment presented using the virtual rendering system, obtaining, from the virtual rendering system, a virtual camera configuration of a virtual camera in the virtual environment, programming the video capture system using the virtual camera configuration to correspondingly control a robotic camera in a real environment, capturing a video capture feed using the robotic camera, obtaining a virtually rendered feed using the virtual camera showing the modifying of the virtual environment, rendering the composite render by processing the feeds, and outputting the composite render to the display.

## Claims

### Claim 1

A method for integrating a virtual rendering system and a video capture system for outputting a composite render to a display, the method comprising:
receiving input data from a plurality of clients for modifying a virtual environment presented using the virtual rendering system;
obtaining, from the virtual rendering system, a first virtual camera configuration of a first virtual camera in the virtual environment and a second virtual camera configuration of a second virtual camera in the virtual environment, wherein the first virtual camera configuration and the second virtual camera configuration include camera movement data including a first motion path of the first virtual camera in the virtual environment and a second motion path of the second virtual camera in the virtual environment, respectively, and wherein the first motion path tracks a path of a virtual object in motion relative to the first virtual camera and the second motion path tracks the path of the virtual object relative to the second virtual camera;
programming the video capture system using the first virtual camera configuration to correspondingly control a first robotic camera in a real environment and using the second virtual camera configuration to correspondingly control a second robotic camera in the real environment;
capturing, from the video capture system, a first video capture feed of the real environment using the first robotic camera and a second video capture feed of the real environment using the second robotic camera;
obtaining, from the virtual rendering system, a first virtually rendered feed of the virtual environment using the first virtual camera and a second virtually rendered feed of the virtual environment using the second virtual camera, wherein the first virtually rendered feed and the second virtually rendered feed show the modifying of the virtual environment;
rendering the composite render by processing the first video capture feed, the first virtually rendered feed, the second video capture feed and the second virtually rendered feed; and
outputting the composite render to the display by simultaneously displaying the first virtually rendered feed and the second virtually rendered feed, wherein the first virtually rendered feed corresponds to a birdseye view of the virtual environment and the second virtually rendered feed corresponds to a flyby view of the virtual environment.

### Claim 2 (depends on 1)

The method of claim 1, wherein the control of the first robotic camera uses the first motion path in the real environment.

### Claim 3 (depends on 1)

The method of claim 1, wherein the first motion path includes a three-dimensional position with respect to time.

### Claim 4 (depends on 1)

The method of claim 1, wherein the first motion path includes a camera orientation or field of view with respect to time.

### Claim 5 (depends on 1)

The method of claim 1, wherein the first virtual camera configuration includes camera lens parameters.

### Claim 6 (depends on 1)

The method of claim 1, wherein the virtual object comprises a virtual ball of a virtual game.

### Claim 7 (depends on 1)

The method of claim 1, wherein the virtual object comprises a virtual player of a virtual game.

### Claim 8 (depends on 1)

The method of claim 1, wherein the first virtual camera configuration and the second virtual camera configuration include a camera focal length, and a camera focal distance.

### Claim 9

A rendering controller for outputting a composite render to a display, the rendering controller comprising:
a processor configured to:
receive input data from a plurality of clients for modifying a virtual environment presented using a virtual rendering system;
obtain, from the virtual rendering system, a first virtual camera configuration of a first virtual camera in the virtual environment and a second virtual camera configuration of a second virtual camera in the virtual environment, wherein the first virtual camera configuration and the second virtual camera configuration include camera movement data including a first motion path of the first virtual camera in the virtual environment and a second motion path of the second virtual camera in the virtual environment, respectively, and wherein the first motion path tracks a path of a virtual object in motion relative to the first virtual camera and the second motion path tracks the path of the virtual object relative to the second virtual camera;
program a video capture system using the virtual camera configuration to correspondingly control a first robotic camera in a real environment and using the second virtual camera configuration to correspondingly control a second robotic camera in the real environment;
capture, from the video capture system, a first video capture feed of the real environment using the first robotic camera and a second video capture feed of the real environment using the second robotic camera;
obtain, from the virtual rendering system, a first virtually rendered feed of the virtual environment using the first virtual camera and a second virtually rendered feed of the virtual environment using the second virtual camera, wherein the first virtually rendered feed and the second virtually rendered feed show the modifying of the virtual environment;
render the composite render by processing the first video capture feed, the first virtually rendered feed, the second video capture feed and the second virtually rendered feed; and
output the composite render to the display by simultaneously displaying the first virtually rendered feed and the second virtually rendered feed, wherein the first virtually rendered feed corresponds to a birdseye view of the virtual environment and the second virtually rendered feed corresponds to a flyby view of the virtual environment.

### Claim 10 (depends on 9)

The rendering controller of claim 9, wherein the control of the first robotic camera uses the first motion path in the real environment.

### Claim 11 (depends on 10)

The rendering controller of claim 10, wherein the first motion path is based on a predetermined path.

### Claim 12 (depends on 9)

The rendering controller of claim 9, wherein the first motion path includes a three-dimensional position with respect to time.

### Claim 13 (depends on 9)

The rendering controller of claim 9, wherein the first motion path includes a camera orientation or field of view with respect to time.

### Claim 14 (depends on 9)

The rendering controller of claim 9, wherein the first virtual camera configuration includes camera lens parameters.

### Claim 15 (depends on 9)

The rendering controller of claim 9, wherein the virtual object comprises a virtual ball of a virtual game.

### Claim 16 (depends on 9)

The rendering controller of claim 9, wherein the virtual object comprises a virtual player of a virtual game.

### Claim 17 (depends on 9)

The rendering controller of claim 9, wherein the first virtual camera configuration and the second virtual camera configuration include a camera focal length, and a camera focal distance.

### Claim 18

A method for integrating a virtual rendering system and a video capture system for outputting a composite render to a display, the method comprising:
receiving input data from a plurality of clients for modifying a virtual environment presented using the virtual rendering system;
obtaining, from the virtual rendering system, a first virtual camera configuration of a first virtual camera in the virtual environment and a second virtual camera configuration of a second virtual camera in the virtual environment, wherein the first virtual camera configuration and the second virtual camera configuration include camera movement data including a first motion path of the first virtual camera in the virtual environment and a second motion path of the second virtual camera in the virtual environment, respectively, and wherein the first motion path tracks a path of a virtual object in motion relative to the first virtual camera and the second motion path tracks the path of the virtual object relative to the second virtual camera;
programming the video capture system using the first virtual camera configuration to correspondingly control a first robotic camera in a real environment and using the second virtual camera configuration to correspondingly control a second robotic camera in the real environment;
capturing, from the video capture system, a first video capture feed of the real environment using the first robotic camera and a second video capture feed of the real environment using the second robotic camera;
obtaining, from the virtual rendering system, a first virtually rendered feed of the virtual environment using the first virtual camera and a second virtually rendered feed of the virtual environment using the second virtual camera, wherein the first virtually rendered feed and the second virtually rendered feed show the modifying of the virtual environment;
rendering the composite render by processing the first video capture feed, the first virtually rendered feed, the second video capture feed and the second virtually rendered feed; and
outputting the composite render to the display by simultaneously displaying the first virtually rendered feed and the second virtually rendered feed, wherein the first virtually rendered feed corresponds to a birdseye view of the virtual environment and the second virtually rendered feed corresponds to a flyby view of the virtual environment.

### Claim 19 (depends on 18)

The method of claim 18, wherein the first virtual camera configuration and the second virtual camera configuration include a camera focal length, and a camera focal distance.