# US 2025/0191239 A1 · Information processing device, information processing method, and program

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0191239 A1
- Document ID: US20250191239A1
- Status: Published application
- Kind: A1
- Application number: 18/843,930
- Filed: 2023-03-14
- Published: 2025-06-12
- Earliest claimed: 2022-03-31
- Assignees: Sony Group Corporation
- Applicants: Sony Group Corporation
- Inventors: Yuki HASEGAWA; Jun YOKONO; Sayaka WATANABE; Yoshiaki IWAI; Jianing WU; Takuya OHASHI
- Web page: https://patentgrep.com/patent/US20250191239A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06T 11/00: Two-dimensional [2D] image generation
- A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence)
- A63F 13/42: by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
- G06F 16/258: {Data format conversion from or to a database}
- G06F 16/288: {Entity relationship models}
- G06V 20/00: Scenes; Scene-specific elements (control of digital cameras H04N23/60)
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- A63F 2300/8011: Ball
- A63F 2300/8082: Virtual reality
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

There is provided an information processing device, an information processing method, and a program capable of converting a real space image into a virtual space image by appropriately reflecting a relationship (interaction) with a person or an object in real space. A real space image including a user and a person or an object around the user is acquired from an imaging device, a relationship (interaction) indicating a motion that the user takes with respect to the person or the object is extracted from the real space image, the user and the person or the object are converted into an avatar and a virtual object in a virtual space on the basis of the relationship, and the real space image is converted and generated into a virtual space image. The present disclosure can be applied to a virtual space display system.

## Claims

### Claim 1

An information processing device comprising
a relationship extraction unit that extracts a relationship between a user in real space and a person or an object around the user from a real space image including the user and the person or the object, and
a virtual space image generation unit that converts the user and the person or the object into an avatar and a virtual object in a virtual space on a basis of the relationship, and converts and generates the real space image into a virtual space image.

### Claim 2 (depends on 1)

The information processing device according to claim 1, wherein
the relationship is a motion of the user with respect to the person or the object.

### Claim 3 (depends on 2)

The information processing device according to claim 2, wherein
the relationship is a direct motion of the user with respect to the person or the object.

### Claim 4 (depends on 2)

The information processing device according to claim 2, wherein
the relationship is an indirect motion of the user with respect to the person or the object.

### Claim 5 (depends on 1)

The information processing device according to claim 1, further comprising
an object shape extraction unit that extracts an object shape that is a shape of the object included in the real space image, wherein
the virtual space image generation unit converts the user and the person or the object into an avatar and a virtual object in the virtual space on a basis of the relationship and the object shape, and converts and generates the real space image into a virtual space image.

### Claim 6 (depends on 5)

The information processing device according to claim 5, further comprising
a virtual object candidate database in which a virtual object candidate to be a candidate of the virtual object into which the object is converted is registered in association with a relationship and an object shape set for each of the virtual object candidates, wherein
the virtual space image generation unit converts the object into the virtual object corresponding to the virtual object candidate having the relationship and the object shape similar to those of the object among the virtual object candidates registered in the virtual object candidate database.

### Claim 7 (depends on 6)

The information processing device according to claim 6, further comprising
a similarity calculation unit that calculates, as a virtual object candidate similarity, a similarity with the relationship and the object shape of the object based on the relationship and the object shape set for each of the virtual object candidates registered in the virtual object candidate database, wherein
the virtual space image generation unit converts the object into the virtual object corresponding to the virtual object candidate having the highest virtual object candidate similarity with the object among the virtual object candidates registered in the virtual object candidate database.

### Claim 8 (depends on 7)

The information processing device according to claim 7, wherein
the similarity calculation unit calculates the virtual object candidate similarity by a weighted product sum of a similarity of the relationship and a similarity of the object shape of each of the virtual object candidates registered in the virtual object candidate database.

### Claim 9 (depends on 7)

The information processing device according to claim 7, wherein
the virtual space image generation unit converts the user and the person or the object into an avatar and a virtual object in the virtual space on a basis of a scene, the relationship, and the object shape, and converts and generates the real space image into a virtual space image.

### Claim 10 (depends on 9)

The information processing device according to claim 9, wherein
the virtual object candidate database includes a plurality of databases set according to the scene, and
the virtual space image generation unit switches the plurality of databases forming the virtual object candidate database according to the scene, and converts the object into the virtual object corresponding to the virtual object candidate having the highest virtual object candidate similarity with the object among the registered virtual object candidates.

### Claim 11 (depends on 6)

The information processing device according to claim 6, wherein
in the virtual object candidate database, a default value of an object effect that is an effect of the virtual object is further registered for each of the virtual object candidates, and
when the virtual space image generation unit converts the user and the person or the object into an avatar and a virtual object in a virtual space on a basis of the relationship and converts and generates the real space image into a virtual space image, the virtual space image generation unit adds the object effect based on the default value registered in association with the virtual object candidate identified as the virtual object to the virtual object.

### Claim 12 (depends on 11)

The information processing device according to claim 11, further comprising
an object state extraction unit that extracts an object state that is a state of the object from the real space image, and
an object effect calculation unit that calculates the object effect from the default value and the object state, wherein
the virtual space image generation unit adds the object effect calculated from a default value registered in association with the virtual object candidate identified as the virtual object and the object state to the virtual object.

### Claim 13 (depends on 12)

The information processing device according to claim 12, wherein
the object state is a degree of management indicating a state of management of the object.

### Claim 14 (depends on 13)

The information processing device according to claim 13, wherein
the degree of management is a value set from 0 to 100%, and
the object effect calculation unit calculates the object effect by multiplying the default value of the object effect by the degree of management.

### Claim 15 (depends on 6)

The information processing device according to claim 6, wherein
an item is further registered in the virtual object candidate database for each of the virtual object candidates, and
when the virtual space image generation unit converts the user and the person or the object into an avatar and a virtual object in a virtual space on a basis of the relationship and converts and generates the real space image into a virtual space image, the virtual space image generation unit adds the item registered in association with the virtual object candidate identified as the virtual object to the avatar.

### Claim 16

An information processing method comprising
extracting a relationship between a user in real space and a person or an object around the user from a real space image including the user and the person or the object, and
converting the user and the person or the object into an avatar and a virtual object in a virtual space on a basis of the relationship, and converting and generating the real space image into a virtual space image.

### Claim 17

A program for causing a computer to function as
a relationship extraction unit that extracts a relationship between a user in real space and a person or an object around the user from a real space image including the user and the person or the object, and
a virtual space image generation unit that converts the user and the person or the object into an avatar and a virtual object in a virtual space on a basis of the relationship, and converts and generates the real space image into a virtual space image.
