# US 2019/0378331 A1 · Virtual world generation engine

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0378331 A1
- Document ID: US20190378331A1
- Status: Published application
- Kind: A1
- Application number: 16/449,035
- Filed: 2019-06-21
- Published: 2019-12-12
- Earliest claimed: 2014-09-29
- Assignees: Amazon Technologies, Inc.
- Applicants: Amazon Technologies, Inc.
- Inventors: Collin Charles Davis; Michael Anthony Frazzini; Ashraf Alkarmi; Gerard Joseph Heinz, II; Michael Schleif Pesce
- Web page: https://patentgrep.com/patent/US20190378331A1

## CPC

- G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation}
- G06F 16/58: Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F 16/587: using geographical or spatial information, e.g. location
- G06F 16/739: {in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames}
- G06F 16/78: Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06T 19/003: {Navigation within 3D models or images}
- G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals
- G11B 27/034: on discs (G11B27/036, G11B27/038 take precedence)
- G06T 9/00: Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)

## Abstract

A virtual world generation engine and methods for generating virtual worlds from images collected from various sources, including crowdsourced images. A virtual world generation engine may obtain images (e.g., digital photographs, digital video frames, etc.) related to a particular real-world scene, combine the images using image processing techniques such as image stitching techniques to generate composite images representing a view of the scene, and generate models from the composite images. The models may be used in rendering video content representing virtual worlds generated from the collected images of real-world scenes; the video content may be streamed to client device(s). Obtaining the images, generating models, rendering video content from the models, and streaming the video content may be performed in response to user interactions with video content on the client device(s) to allow interactive exploration of the virtual worlds in real-time or near-real time.

## Claims

### Claim 1

A system, comprising:
one or more computing devices configured to implement a video system comprising:
an image collection module configured to obtain digital images from one or more sources, each image capturing a portion of a real-world scene;
a virtual world generation engine configured to:
identify two or more of the images that include overlapping portions of the scene;
combine the identified images according to one or more digital image processing techniques to generate a composite image representing a view of the scene; and
generate a model of the scene according to the composite image;
one or more video processing modules configured to:
render video of the scene from the model of the scene; and
stream the video of the scene to a client device.

### Claim 2 (depends on 1)

The system as recited in claim 1, wherein the video system is configured to perform said identifying, said combining, said generating, said rendering, and said streaming in response to user interactions exploring content of video streamed to the client device.

### Claim 3 (depends on 1)

The system as recited in claim 1, wherein the one or more sources for images include one or more crowdsourcing techniques for soliciting and obtaining digital images from a plurality of online contributors.

### Claim 4 (depends on 1)

The system as recited in claim 1, wherein the image collection module is configured to:
obtain a plurality of images from a plurality of sources, the images capturing different portions of a plurality of scenes;
obtain or generate image metadata for the images, the metadata for a given image including information describing one or more characteristics of the respective image;
correlate the images into a plurality of image groups based on one or more of image content or image metadata, each image group corresponding to a particular scene;
determine spatial relationships among the images within each image group according to one or more of content of the images or image metadata indicating positioning of the images relative to the scene; and
maintain the images, image metadata, and image data describing the image groups for access by the virtual world generation engine.

### Claim 5 (depends on 1)

The system as recited in claim 1, wherein, to identify two or more of the images that include overlapping portions of the scene, the virtual world generation engine is configured to determine overlapping images according to one or more of content of the images or image metadata indicating positioning of the images relative to the scene.

### Claim 6 (depends on 1)

The system as recited in claim 1, wherein, to combine the identified images according to one or more digital image processing techniques to generate a composite image representing a view of the scene, the virtual world generation engine is configured to apply one or more image stitching techniques to:
align the identified images according to one or more of content of the images or image metadata indicating positioning of the images relative to each other; and
merge non-overlapping and overlapping portions of the aligned images according to one or more image processing techniques to generate composited image content.

### Claim 7 (depends on 1)

The system as recited in claim 1, wherein, to generate a model of the scene according to the composite image, the virtual world generation engine is configured to:
generate two-dimensional (2D) or three-dimensional (3D) graphics data from the composite image; and
generate a 2D or 3D model of the scene according to the graphics data.

### Claim 8 (depends on 1)

The system as recited in claim 1, wherein the one or more computing devices that implement the video system are on a provider network, wherein the client device is configured to access the video system on the provider network via an intermediate network, and wherein the video system is configured to leverage one or more computing resources of the provider network to perform said identifying, said combining, said generating, said rendering, and said streaming in response to user input from the client device.

### Claim 9 (depends on 1)

The system as recited in claim 1, wherein the video system is a real-time video exploration (RVE) system or an online game system.

### Claim 10

A method, comprising:
iteratively performing, by a virtual world generation engine implemented on one or more computing devices:
obtaining two or more images that include overlapping portions of a real-world scene;
combining at least two of the obtained images according to one or more digital image processing techniques to generate a composite image representing a view of the scene;
generating or updating a model of the scene according to the composite image; and
providing the model of the scene to a video processing module for rendering.

### Claim 11 (depends on 10)

The method as recited in claim 10, further comprising:
obtaining, by an image collection module implemented on one or more computing devices, a plurality of images from a plurality of sources, the plurality of images capturing different portions of a plurality of scenes; and
obtaining or generating image data for the images, the image data including information indicating image groups corresponding to particular scenes and information indicating spatial relationships among the images within each image group;
wherein said obtaining two or more images that include overlapping portions of a real-world scene comprises obtaining the two or more images from the image collection module according to image data corresponding to the real-world scene.

### Claim 12 (depends on 11)

The method as recited in claim 11, wherein the plurality of sources include a plurality of online contributors, wherein the method further comprises obtaining at least some of the images from the plurality of online contributors via one or more crowdsourcing techniques.

### Claim 13 (depends on 10)

The method as recited in claim 10, further comprising:
iteratively performing, by the video processing module:
receiving a model of the scene from the virtual world generation engine;
rendering new video content at least in part from the model of the scene; and
sending the new video content to at least one client device.

### Claim 14 (depends on 11)

The method as recited in claim 11, wherein the virtual world generation engine performs said obtaining, said combining, said generating, and said providing in response to input received from the video processing module indicating interactions exploring content of video sent to a client device.

### Claim 15 (depends on 10)

The method as recited in claim 10, wherein said combining the images according to one or more digital image processing techniques to generate a composite image representing a view of the scene comprises applying an image stitching technique to the images, wherein the image stitching technique aligns the identified images according to one or more of content of the images or image metadata indicating positioning of the images relative to each other and merges the aligned images according to one or more image processing techniques to generate the composite image.

### Claim 16 (depends on 10)

The method as recited in claim 10, wherein the video processing module is a component of a real-time video exploration (RVE) system or of an online game system.

### Claim 17

A non-transitory computer-readable storage medium storing program instructions that when executed on one or more computers cause the one or more computers to implement a real-time video exploration (RVE) system configured to:
receive input from a client device indicating user interactions with video content on the client device;
in response to the user interactions:
obtain a plurality of images that include content related to a real-world scene indicated by the user interactions with the video content;
select at least two of the obtained images according to the user interactions with the video content; and
send the selected images to the client device for display to the user.

### Claim 18 (depends on 17)

The non-transitory computer-readable storage medium as recited in claim 17, wherein, to send the selected images to the client device for display to the user, the RVE system is further configured to:
combine the selected images according to one or more digital image processing techniques to generate a composite image representing a view of the scene;
generate a two-dimensional (2D) or three-dimensional (3D) model of the scene according to the composite image;
render new video content at least in part from the model of the scene; and
stream video including the new video content to the client device.

### Claim 19 (depends on 18)

The non-transitory computer-readable storage medium as recited in claim 18, wherein, to combine the selected images according to one or more digital image processing techniques to generate a composite image representing a view of the scene, the RVE system is configured to:
align the identified images according to content or features of the images; and
merge non-overlapping and overlapping portions of the aligned images to generate the composite image.

### Claim 20 (depends on 17)

The non-transitory computer-readable storage medium as recited in claim 17, wherein, to send the selected images to the client device for display to the user, the RVE system is further configured to send the selected images to a client interface on the client device for display as two-dimensional (2D) images related to the scene.

### Claim at position 21

-23. (canceled)