# US 11,439,917 B2 · Voice help system using artificial intelligence

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,439,917 B2
- Document ID: US11439917B2
- Status: Granted
- Kind: B2
- Application number: 15/992,092
- Filed: 2018-05-29
- Published: 2019-09-26
- Granted: 2022-09-13
- Earliest claimed: 2018-03-23
- Assignees: Sony Interactive Entertainment Inc.
- Applicants: Sony Interactive Entertainment LLC
- Inventors: Warren Benedetto; Landon Noss; Tom Srisook; Andrew Herman; Chad Rolinski; Parsa Shirazi
- Web page: https://patentgrep.com/patent/US11439917B2

## CPC

- A63F 13/86: Watching games played by other players
- A63F 13/30: Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
- A63F 13/35: Details of game servers
- A63F 13/5375: for graphically or textually suggesting an action, e.g. by displaying an arrow indicating a turn in a driving game
- A63F 13/79: involving player-related data, e.g. identities, accounts, preferences or play histories
- A63F 13/85: Providing additional services to players
- A63F 13/87: Communicating with other players during game play, e.g. by e-mail or chat
- G06F 16/433: {using audio data}
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- A63F 13/422: automatically for the purpose of assisting the player, e.g. automatic braking in a driving game
- A63F 13/497: Partially or entirely replaying previous game actions
- A63F 13/67: adaptively or by learning from player actions, e.g. skill level adjustment or by storing successful combat sequences for re-use
- A63F 2300/303: for displaying additional data, e.g. simulating a Head Up Display
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A method for gaming assistance. Information related to a plurality of game plays of players for a gaming application is received over a network at a back-end server. A query is received from a first player playing the gaming application, wherein the query is related to a first game play of the first player. A current game context of a first game play of a first player is determined from the information. A response to the query and the current game context is generated based on the information related to the plurality of game plays. The response is delivered back to a device of the first player.

## Claims

### Claim 1

A method for providing gaming assistance, comprising:
receiving over a network at a back-end server information related to a plurality of game plays of players for a gaming application;
determining a plurality of game contexts of the plurality of game plays of the players based on the information related to the plurality of game plays of the players;
training a machine learning engine to generate a plurality of modeled responses based on the information related to the plurality of game plays of the players, the plurality of game contexts, and a plurality of queries for the plurality of game plays of the players originating from the players;
receiving a query from a first player playing the gaming application, the query related to a first game play of the first player;
receiving information related to the first game play of the first player;
determining a current game context of the first game play based on the information related to the first game play of the first player and the information related to the plurality of game plays of the players;
matching for the first game play a modeled response from the plurality of modeled responses to the query and the current game context using the machine learning engine based on the information related to the plurality of game plays of the players and the information related to the first game play of the first player used to match one or more game contexts of the plurality of game plays of the players to the current game context of the first player, wherein each of the one or more game contexts of the plurality of game plays of the players that are matched to the current game context have a corresponding matching vector with a corresponding matching value that indicates similarity between a corresponding game context and the current game context, the corresponding matching value exceeding a threshold; and
sending the modeled response to a device of the first player.

### Claim 2 (depends on 1)

The method of claim 1, wherein the matching for the first game play the modeled response includes:
interpreting the query; and
providing as input the query that is interpreted and the current game context for the gaming application into the machine learning engine configured to match the query that is interpreted and the current game context to the modeled response, the modeled response being one of the plurality of modeled responses for the gaming application.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
sending the query and the current game context to a third party artificial intelligence engine configured for interpreting the query and matching the query that is interpreted and current game context to the modeled response.

### Claim 4 (depends on 2)

The method of claim 2, further comprising:
translating the query from a first format to a second format, wherein the first format is audio and the second format is text.

### Claim 5 (depends on 1)

The method of claim 1, wherein the query is presented in a format including at least one of audio, text, and video.

### Claim 6 (depends on 1)

The method of claim 1, wherein the modeled response is presented in a format including at least one of audio, text, and video.

### Claim 7 (depends on 1)

The method of claim 1 wherein the device comprises a companion mobile device, wherein the first game play is presented on a display.

### Claim 8 (depends on 1)

The method of claim 1, wherein the device comprises a display configured for presenting the first game play and the modeled response.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
assigning to the first player an expert for obtaining assistance; and
generating a communication session between the device of the first player and a device of the expert to enable the expert to render the assistance to the first player.

### Claim 10 (depends on 1)

The method of claim 1, wherein the information related to the first game play of the first player includes game context data and global context data.

### Claim 11 (depends on 10)

The method of claim 10, wherein the game context data includes game state data and operating system (OS) level data.

### Claim 12 (depends on 11)

The method of claim 11, wherein the game state data includes at least one of the following:
character;
character race or type;
current quest for a first character in the first game play;
next quest for the first character in the first game play;
location of the first game play in a gaming environment;
level of game playing the gaming environment;
assets;
loadout; and
skill set of the first character.

### Claim 13 (depends on 11)

The method of claim 11, wherein the OS level data includes at least one of the following:
gaming input;
controller input;
timestamps of controller input.

### Claim 14 (depends on 10)

The method of claim 10, wherein the global context data includes at least one of the following:
player profile data;
playing style of the first player;
skill level of the first player;
whether or not the first player is currently playing the gaming application;
whether or not the first player is online;
the title of the gaming application;
co-players the first player is playing with;
frequency of input controller actuation; and
friends of the first player.

### Claim 15

A non-transitory computer-readable medium storing a computer program for providing gaming assistance, the computer-readable medium comprising:
program instructions for receiving over a network at a back-end server information related to a plurality of game plays of players for a gaming application;
program instructions for determining a plurality of game contexts of the plurality of game plays of the players based on the information related to the plurality of game plays of the players;
program instructions for training a machine learning engine to generate a plurality of modeled responses based on the information related to the plurality of game plays of the players, the plurality of game contexts, and a plurality of queries for the plurality of game plays of the players originating from the players;
program instructions for receiving a query from a first player playing the gaming application, the query related to a first game play of the first player;
program instructions for receiving information related to the first game play of the first player;
program instructions for determining a current game context of the first game play based on the information related to the first game play of the first player and the information related to the plurality of game plays of the players;
program instructions for matching for the first game play a modeled response from the plurality of modeled responses to the query and the current game context using the machine learning engine based on the information related to the plurality of game plays of the players and the information related to the first game play of the first player used to match one or more game contexts of the plurality of game plays of the players to the current game context of the first player, wherein each of the one or more game contexts of the plurality of game plays of the players that are matched to the current game context have a corresponding matching vector with a corresponding matching value that indicates similarity between a corresponding game context and the current game context, the corresponding matching value exceeding a threshold; and
program instructions for sending the modeled response to a device of the first player.

### Claim 16 (depends on 15)

The computer-readable medium of claim 15, wherein the program instructions for matching for the first game play the modeled response includes:
program instructions for interpreting the query; and
program instructions for providing as input the query that is interpreted and the current game context for the gaming application into the machine learning engine configured to match the query that is interpreted and the current game context to the modeled response, the modeled response being one of the plurality of modeled responses for the gaming application.

### Claim 17 (depends on 16)

The computer-readable medium of claim 16, further comprising:
program instructions for sending the query and the current game context to a third party artificial intelligence engine configured for interpreting the query and matching the query that is interpreted and current game context to the modeled response.

### Claim 18 (depends on 16)

The computer-readable medium of claim 16, further comprising:
program instructions for translating the query from a first format to a second format, wherein the first format is audio and the second format is text.

### Claim 19

A computer system comprising:
a processor; and
memory coupled to the processor and having stored therein instructions that, if executed by the computer system, cause the computer system to execute a method for providing gaming assistance comprising:
receiving over a network at a back-end server information related to a plurality of game plays of players for a gaming application;
determining a plurality of game contexts of the plurality of game plays of the players based on the information related to the plurality of game plays of the players;
training a machine learning engine to generate a plurality of modeled responses based on the information related to the plurality of game plays of the players, the plurality of game contexts, and a plurality of queries for the plurality of game plays of the players originating from the players;
receiving a query from a first player playing the gaming application, the query related to a first game play of the first player;
receiving information related to the first game play of the first player;
determining a current game context of the first game play based on the information related to the first game play of the first player and the information related to the plurality of game plays of the players;
matching for the first game play a modeled response from the plurality of modeled responses to the query and the current game context using the machine learning engine based on the information related to the plurality of game plays of the players and the information related to the first game play of the first player used to match one or more game contexts of the plurality of game plays of the players to the current game context of the first player, wherein each of the one or more game contexts of the plurality of game plays of the players that are matched to the current game context have a corresponding matching vector with a corresponding matching value that indicates similarity between a corresponding game context and the current game context, the corresponding matching value exceeding a threshold; and
sending the modeled response to a device of the first player.

### Claim 20 (depends on 19)

The computer system of claim 19, wherein the matching for the first game play the modeled response in the method includes:
interpreting the query; and
providing as input the query that is interpreted and the current game context for the gaming application into the deep learning engine configured to match the query that is interpreted and the current game context to the modeled response, the modeled response being one of the plurality of modeled responses for the gaming application.

### Claim 21 (depends on 20)

The computer system of claim 20, the method further comprising:
sending the query and the current game context to a third party artificial intelligence engine configured for interpreting the query and matching the query that is interpreted and current game context to the modeled response.

### Claim 22 (depends on 20)

The computer system of claim 20, the method further comprising:
translating the query from a first format to a second format, wherein the first format is audio and the second format is text.