# US 11,110,343 B2 · Methods and systems for generating detailed datasets of an environment via gameplay

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,110,343 B2
- Document ID: US11110343B2
- Status: Granted
- Kind: B2
- Application number: 16/377,095
- Filed: 2019-04-05
- Published: 2019-08-01
- Granted: 2021-09-07
- Earliest claimed: 2017-09-08
- Assignees: Niantic, Inc.
- Applicants: Niantic, Inc.
- Inventors: Ryan Michael Hickman; Soohyun Bae
- Web page: https://patentgrep.com/patent/US11110343B2

## CPC

- A63F 13/46: Computing the game score
- A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence)
- A63F 13/215: comprising means for detecting acoustic signals, e.g. using a microphone
- A63F 13/25: Output arrangements for video game devices
- A63F 13/533: for prompting the player, e.g. by displaying a game menu
- A63F 13/56: Computing the motion of game characters with respect to other game characters, game objects or elements of the game scene, e.g. for simulating the behaviour of a group of virtual soldiers or for path finding
- A63F 13/60: Generating or modifying game content before or while executing the game program, e.g. authoring tools specially adapted for game development or game-integrated level editor
- A63F 13/655: by importing photos, e.g. of the player
- G06F 16/51: Indexing; Data structures therefor; Storage structures
- G06F 16/5838: {using colour}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06T 7/00: Image analysis
- G06T 15/00: Three-dimensional [3D] image rendering
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 20/20: in augmented reality scenes
- G06V 40/176: {Dynamic expression}
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An example method includes obtaining, obtaining, from one or more sensors of a computing device, data relating to a feature in an environment. Following, the method includes analyzing the data to identify one or more details of the feature in the environment and determining, based on a comparison of the data to a stored dataset in a database, that the details includes a detail that the stored dataset lacks. The method includes providing one or more game elements for gameplay on an interface of the computing device based on the details including the detail that the stored dataset lacks.

## Claims

### Claim 1

A computer-implemented method, comprising:
analyzing an image of an environment to detect an object in the environment;
responsive to detecting the object, obtaining, from one or more sensors of a computing device, data of one of a plurality of available data types describing a detail of the object in the environment;
determining, based on a comparison of the data to a stored dataset in a database, that the stored dataset lacks the detail of the object, the stored dataset including data describing other details of the object of at least some of the plurality of available data types; and
responsive to determining the data includes the detail that the stored dataset lacks, providing one or more game elements for use in gameplay on an interface of the computing device.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the environment comprises a house, and wherein the object in the environment is a specific room in the house, wherein providing the one or more game elements comprises:
providing information indicating a request to obtain additional data relating to the specific room in the house.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the one or more sensors of the computing device include a depth camera and wherein obtaining, from the one or more sensors of the computing device, the data describing the detail of the object in the environment comprises:
obtaining, using the depth camera, one or more depth images of the object in the environment.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the one or more sensors of the computing device include a microphone, and wherein obtaining, from the one or more sensors of the computing device, the data describing the detail of the object in the environment comprises:
obtaining, using the microphone, audio from the environment.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the one or more sensors of the computing device include an inertial measurement unit (IMU), and wherein obtaining, from the one or more sensors of the computing device, the data describing the detail of the object in the environment comprises:
obtaining, using the IMU, movement data of the computing device in the environment.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the one or more sensors of the computing device include a motion tracking camera, and wherein obtaining, from the one or more sensors of the computing device, the data describing the detail of the object in the environment comprises:
obtaining, using the motion tracking camera, movement data of the object in the environment.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the one or more sensors of the computing device include a proximity sensor, and wherein obtaining, from the one or more sensors of the computing device, the data describing the detail of the object in the environment comprises:
obtaining, using the proximity sensor, proximity data of the object in the environment within a range of the computing device.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, further comprising:
generating a floorplan of the environment based on the data describing the detail of the object in the environment.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein determining that the stored dataset lacks the detail of the object comprises:
receiving, from a server, an identification of the object based on the data describing the detail of the object;
retrieving the stored dataset from the database, the stored dataset corresponding to the object; and
comparing the data describing the detail of the object to the stored dataset from the database to determine that the data describing the detail of the object is not included in the dataset from the database.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein the one or more game elements for gameplay include at least one of:
one or more points;
one or more achievements;
one or more levels; or
one or more ranks on a leaderboard.

### Claim 11

A non-transitory computer-readable storage medium with encoded instructions that, when executed by a computer system, cause the computer system to perform steps comprising:
analyzing an image of an environment to detect an object in the environment;
responsive to detecting the object, obtaining, from one or more sensors of a computing device, data of one of a plurality of available data types describing a detail of the object in the environment;
determining, based on a comparison of the data to a stored dataset in a database, that the stored dataset lacks the detail of the object, the stored dataset including data describing other details of the object of at least some of the plurality of available data types; and
responsive to determining the data includes the detail that the stored dataset lacks, providing one or more game elements for use in gameplay on an interface of the computing device.

### Claim 12 (depends on 11)

The storage medium of claim 11, wherein the environment comprises a house, and wherein the object in the environment is a specific room in the house, wherein providing the one or more game elements comprises:
providing information indicating a request to obtain additional data relating to the specific room in the house.

### Claim 13 (depends on 11)

The storage medium of claim 11, wherein the one or more sensors of the computing device include a depth camera and wherein obtaining, from the one or more sensors of the computing device, the data describing the detail of the object in the environment comprises:
obtaining, using the depth camera, one or more depth images of the object in the environment.

### Claim 14 (depends on 11)

The storage medium of claim 11, wherein the one or more sensors of the computing device include a microphone, and wherein obtaining, from the one or more sensors of the computing device, the data describing the detail of the object in the environment comprises:
obtaining, using the microphone, audio from the environment.

### Claim 15 (depends on 11)

The storage medium of claim 11, wherein the one or more sensors of the computing device include an inertial measurement unit (IMU), and wherein obtaining, from the one or more sensors of the computing device, the data describing the detail of the object in the environment comprises:
obtaining, using the IMU, movement data of the computing device in the environment.

### Claim 16 (depends on 11)

The storage medium of claim 11, wherein the one or more sensors of the computing device include a motion tracking camera, and wherein obtaining, from the one or more sensors of the computing device, the data describing the detail of the object in the environment comprises:
obtaining, using the motion tracking camera, movement data of the object in the environment.

### Claim 17 (depends on 11)

The storage medium of claim 11, wherein the one or more sensors of the computing device include a proximity sensor, and wherein obtaining, from the one or more sensors of the computing device, the data describing the detail of the object in the environment comprises:
obtaining, using the proximity sensor, proximity data of the object in the environment within a range of the computing device.

### Claim 18 (depends on 11)

The storage medium of claim 11, further comprising:
generating a floorplan of the environment based on the data describing the detail of the object in the environment.

### Claim 19 (depends on 11)

The storage medium of claim 11, wherein determining that the stored dataset lacks the detail of the object comprises:
receiving, from a server, an identification of the object based on the data describing the detail of the object;
retrieving the stored dataset from the database, the stored dataset corresponding to the object; and
comparing the data describing the detail of the object to the stored dataset from the database to determine that the data describing the detail of the object is not included in the dataset lacks from the database.

### Claim 20 (depends on 11)

The storage medium of claim 11, wherein the one or more game elements for gameplay include at least one of:
one or more points;
one or more achievements;
one or more levels; or
one or more ranks on a leaderboard.