# US 11,994,673 B2 · Information processing device and image processing method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,994,673 B2
- Document ID: US11994673B2
- Status: Granted
- Kind: B2
- Application number: 17/310,884
- Filed: 2020-02-07
- Published: 2022-05-12
- Granted: 2024-05-28
- Earliest claimed: 2019-03-12
- Assignees: Sony Group Corporation
- Applicants: Sony Group Corporation
- Inventors: Mari Saito; Kenji Sugihara; Soichiro Inatani
- Web page: https://patentgrep.com/patent/US11994673B2

## CPC

- G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking}
- G02B 27/0075: {with means for altering, e.g. increasing, the depth of field or depth of focus}
- G02B 27/017: {Head mounted}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G09G 5/38: with means for controlling the display position
- H04N 21/44218: {Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV programme (methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45)}
- H04N 21/4728: for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G09G 2340/0464: Positioning
- G09G 2340/12: Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
- H04N 21/816: {involving special video data, e.g 3D video}
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An information processing device includes a control unit that has control so that, when detected a status transition of a user including a rising motion or a moving motion of the user during reproduction of a wide-angle image covering the entire field of view of the user, a real-space image acquired by an outward-facing camera provided on a display device that is worn on the head of the user and has a display unit covering the entire field of view is displayed on the display unit in real-time and a 2D image corresponding to a portion of the wide-angle image is displayed on the real-space image.

## Claims

### Claim 1

An information processing device, comprising:
a control unit communicatively coupled to a display device, wherein the control unit is configured to:
detect a status transition of a user including a rising motion or a moving motion of the user during reproduction of a wide-angle image covering an entire field of view of the user;
receive, based on the detected status transition, a real-space image acquired by an outward-facing camera included in the display device, wherein
the display device is wearable-on a head of the user, and
the display device includes a display unit configured to cover the entire field of view;
generate a 2D image corresponding to a first portion of the wide-angle image based on content of the wide-angle image; and
control the display unit to display the 2D image on the real-space image at a display position based on a detected moving direction of the user.

### Claim 2 (depends on 1)

The information processing device according to claim 1, wherein the control unit is further configured to generate, as the 2D image, an image of a second portion including a gazing area based on line-of-sight information of the user in the wide-angle image before the detection of the status transition.

### Claim 3 (depends on 1)

The information processing device according to claim 1, wherein
the control unit is further configured to generate a plurality of 2D images that transition from an image of a second portion of the wide-angle image that includes a gazing area-and the first portion before the detection of the status transition to an enlarged image of the first portion of the wide-angle image, and
the gazing area of second portion is based on line-of-sight information of the user.

### Claim 4 (depends on 1)

The information processing device according to claim 1, wherein the control unit is further configured to generate a plurality of 2D images that transition from a gazing area based on line-of-sight information of the user in the wide-angle image before the detection of the status transition to the first portion.

### Claim 5 (depends on 1)

The information processing device according to claim 1, wherein the control unit is further configured to control a power supply of the outward-facing camera to turn on based on the detection of-the status transition.

### Claim 6 (depends on 1)

The information processing device according to claim 1, wherein
the control unit is further configured to control the display unit to switch from a display of the 2D image to a display of the wide-angle image covering the entire field of view of the user that is based on line-of-sight information of the user, and
the switch from the display of the 2D image to the display of the wide-angle image is based on detection of a movement stop motion or a sitting position of the user.

### Claim 7

An information processing method, comprising:
detecting a status transition of a user including a rising motion or a moving motion of the user during reproduction of a wide-angle image covering an entire field of view of the user;
receiving, based on the detected status transition, a real-space image acquired by an outward-facing camera included in a display device, wherein
the display device is wearable on a head of the user, and
the display device includes-a display unit that covers the entire field of view;
generating a 2D image corresponding to a portion of the wide-angle image based on content of the wide-angle image; and
displaying, the 2D image on the real-space image at a display position on the display unit, based on a detected moving direction of the user.

### Claim 8

A non-transitory computer-readable medium having stored thereon computer executable instructions, which when executed by one or more processors, cause the one or more processors to execute operations comprising:
detecting a status transition of a user including a rising motion or a moving motion of the user during reproduction of a wide-angle image covering an entire field of view of the user;
receiving, based on the detected status transition, a real-space image acquired by an outward-facing camera included in a display device, wherein
the display device is wearable on a head of the user, and
the display device includes a display unit that covers-the entire field of view;
generating a 2D image corresponding to a portion of the wide-angle image based on content of the wide-angle image; and
displaying, the 2D image on the real-space image at a display position on the display unit, based on a detected moving direction of the user.