# US 12,373,907 B2 · Adjustable robot for providing scale of virtual assets and identifying objects in an interactive scene

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,373,907 B2
- Document ID: US12373907B2
- Status: Granted
- Kind: B2
- Application number: 17/224,051
- Filed: 2021-04-06
- Published: 2022-10-06
- Granted: 2025-07-29
- Earliest claimed: 2021-04-06
- Assignees: Sony Interactive Entertainment Inc.
- Applicants: Sony Interactive Entertainment LLC
- Inventors: Derek Crosby; Geoff Wedig
- Web page: https://patentgrep.com/patent/US12373907B2

## CPC

- G06T 1/0014: {Image feed-back for automatic industrial control, e.g. robot with camera (robots B25J19/023)}
- A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence)
- A63F 13/52: involving aspects of the displayed game scene
- A63F 13/56: Computing the motion of game characters with respect to other game characters, game objects or elements of the game scene, e.g. for simulating the behaviour of a group of virtual soldiers or for path finding
- A63F 13/65: automatically by game devices or servers from real world data, e.g. measurement in live racing competition
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06T 7/32: using correlation-based methods
- G06T 7/73: using feature-based methods
- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- G06T 13/80: Two-dimensional [2D] animation, e.g. using sprites
- G06T 15/205: {Image-based rendering}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- A63F 2300/6607: for animating game characters, e.g. skeleton kinematics
- G05B 2219/40132: Haptic joystick with force feedback based on accelerometer included in joystick
- G06T 2207/30196: Human being; Person
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

A method for generating action sequence data for a character to be rendered in a video game is provided. The method includes, during recording of an actor in a performance space, providing a cue to the actor to perform an action that simulates interacting with a virtual object at a location in the performance space. The method also includes instructing a robot to move to the location in the performance space at which the actor is to simulate interaction with the virtual object, and instructing the robot to adjust a physical attribute of the robot when at the location. The physical attribute of the robot, when adjusted, is placed at a three-dimensional location in the performance space to simulate a scale of the virtual object with which the actor is to simulate interaction. The virtual object can be either an animated object or a non-animated object.

## Claims

### Claim 1

A method for generating action sequence data for a character to be rendered in a video game, comprising:
during recording of an actor in a performance space to capture data to be processed to generate the action sequence data for the character to be rendered in the video game, the performance space is a real-world space having a controller for managing communication in the performance space;
providing, via the controller, a cue to the actor to perform an action that simulates making eye contact during a simulated interaction with a virtual object at a location in the performance space, wherein the actor corresponds to the character to be rendered in the video game, wherein the eye contact is defined by a line of sight between the actor and a portion of a robot at a coordinate relative to a base of the robot;
instructing, via the controller, the robot to move to the location in the performance space at which the actor is to simulate interaction with the virtual object;
instructing, via the controller, the robot to adjust a configuration of the robot when at the location, the configuration of the robot is adjusted by adjusting a physical attribute of the robot, the physical attribute when adjusted is placed at a three-dimensional location in the performance space in a configuration that corresponds to a scale of the virtual object in the video game to enable the actor to simulate interaction with the virtual object using the robot as a frame of reference at a scale based at least in part on the line of sight and the coordinate that minimizes a need to edit the action sequence data to correct scaling issues; and
capturing the action by the actor when the actor performs the action that simulates interacting with the virtual object at the location in the performance space, wherein a database stores files of the captured action by the actor along with files of a plurality of other captured actions, and the captured actions are implemented in a later rendering of the video game to generate action sequences by the character when rendered in the video game.

### Claim 2 (depends on 1)

The method of claim 1, wherein the location in the performance space corresponds to a virtual location of the virtual object in a virtual space of the video game, wherein a real distance in the performance space between the actor and the location is correlated to a virtual distance in the virtual space between the character and the virtual object, and wherein changes in the real distance are tracked to cause corresponding changes in the virtual distance.

### Claim 3 (depends on 1)

The method of claim 1, wherein the virtual object is an animated object to be rendered in the video game.

### Claim 4 (depends on 3)

The method of claim 3, wherein the action performed by the actor is one of a handshake with the animated object, a facial expression directed toward the animated object, a motion made by the character in relation to the animated object, a motion of input by the character made to or received from the animated object, or an interactive action made by the character in relation to the animated object.

### Claim 5 (depends on 1)

The method of claim 1, wherein the robot includes an adjustable arm to which a prop representing a hand is attached, and wherein the instructing the robot to adjust the physical attribute of the robot includes instructing the robot to place the prop representing the hand at the three-dimensional location in the performance space that corresponds to an approximate location of an animated object's hand in a virtual space of the video game.

### Claim 6 (depends on 1)

The method of claim 1, wherein the virtual object represents a physical object in a virtual space.

### Claim 7 (depends on 1)

The method of claim 1, wherein the performance space is provided with a camera situated along a periphery of the performance space and a depth sensor disposed within the performance space to enable the capturing of the action by the actor in relation to the scale of the virtual object.

### Claim 8

A method for generating action sequence data for a character to be rendered in a video game, comprising:
during recording of an actor in a performance space to capture data to be processed to generate the action sequence data for the character to be rendered in the video game, the performance space is a real-world space having a controller for managing communication in the performance space;
providing, via the controller, a cue to the actor to perform an action that simulates making eye contact during a simulated interaction with a virtual object at a location in the performance space, wherein the actor corresponds to the character to be rendered in the video game, wherein the eye contact is defined by a line of sight between the actor and a portion of a robot at a coordinate relative to a base of the robot;
instructing, via the controller, the robot to move to the location in the performance space at which the actor is to simulate interaction with the virtual object;
instructing, via the controller, the robot to adjust a configuration of the robot when at the location, the configuration of the robot is adjusted by adjusting a physical attribute of the robot, the physical attribute when adjusted is placed at a three-dimensional location in a configuration that corresponds to a scale of the virtual object in the video game to enable the actor to simulate interaction with the virtual object using the robot as a frame of reference at a scale based at least in part on the line of sight and the coordinate that minimizes a need to edit the action sequence data to correct scaling issues; and
capturing the action by the actor when the actor performs the action that simulates interacting with the virtual object at the location in the performance space, wherein a database stores files of the captured action by the actor along with files of a plurality of other captured actions, and the captured actions are implemented in a later rendering of the video game to generate to action sequences by the character when rendered in the video game.

### Claim 9 (depends on 8)

The method of claim 8, wherein the location in the performance space corresponds to a virtual location of the virtual object in a virtual space of the video game, wherein a real distance in the performance space between the actor and the location is correlated to a virtual distance in the virtual space between the character and the virtual object, and wherein changes in the real distance are tracked to cause corresponding changes in the virtual distance.

### Claim 10 (depends on 8)

The method of claim 8, wherein the virtual object is one of an animated object or a non-animated object.

### Claim 11 (depends on 10)

The method of claim 10, wherein the action performed by the actor is one of going around the animated object or the non-animated object, jumping over the animated object or the non-animated object, climbing over the animated object or the non-animated object, stepping on the animated object or the non-animated object, or an interactive action made by the character in relation to the animated object or the non-animated object.

### Claim 12 (depends on 10)

The method of claim 10, wherein the virtual object is a non-animated object, and the non-animated object is one of a table, a fence, a wall, a rock, a game-generated object, or game-generated topography.

### Claim 13 (depends on 10)

The method of claim 10, wherein the virtual object is an animated object, and the animated object is one of a vehicle, a river, or a game-generated character.

### Claim 14 (depends on 8)

The method of claim 8, wherein the performance space is provided with a camera situated along a periphery of the performance space and a depth sensor disposed within the performance space to enable the capturing of the action by the actor in relation to the scale of the obstacle object.

### Claim 15

A non-transitory computer readable medium containing program instructions for generating action sequence data for a character to be rendered in a video game, wherein execution of the program instructions by one or more processors of a computer system causes the one or more processors to carry out operations comprising:
during recording of an actor in a performance space to capture data to be processed to generate the action sequence data for the character to be rendered in the video game, the performance space is a real-world space having a controller for managing communication in the performance space;
providing, via the controller, a cue to the actor to perform an action that simulates making eye contact during a simulated interaction with a virtual object at a location in the performance space, wherein the actor corresponds to the character to be rendered in the video game, wherein the eye contact is defined by a line of sight between the actor and a portion of a robot at a coordinate relative to a base of the robot;
instructing, via the controller, the robot to move to the location in the performance space at which the actor is to simulate interaction with the virtual object;
instructing, via the controller, the robot to adjust a configuration of the robot when at the location, the configuration of the robot is adjusted by adjusting a physical attribute of the robot, the physical attribute when adjusted is placed at a three-dimensional location in the performance space in a configuration that corresponds to a scale of the virtual object in the video game to enable the actor to simulate interaction with the virtual object using the robot as a frame of reference at a scale based at least in part on the line of sight and the coordinate that minimizes a need to edit the action sequence data to correct scaling issues; and
capturing the action by the actor when the actor performs the action that simulates interacting with the virtual object at the location in the performance space, wherein a database stores files of the captured action by the actor along with files of a plurality of other captured actions, and the captured actions are implemented in a later rendering of the video game to generate action sequences by the character when rendered in the video game.

### Claim 16 (depends on 15)

The computer readable medium of claim 15, wherein the location in the performance space corresponds to a virtual location of the virtual object in a virtual space of the video game, wherein a real distance in the performance space between the actor and the location is correlated to a virtual distance in the virtual space between the character and the virtual object, and wherein changes in the real distance are tracked to cause corresponding changes in the virtual distance.

### Claim 17 (depends on 15)

The computer readable medium of claim 15, wherein the performance space is provided with a camera situated along a periphery of the performance space and a depth sensor disposed within the performance space to enable the capturing of the action by the actor in relation to the scale of the virtual object.