# US 11,310,472 B2 · Information processing device and image generation method for projecting a subject image onto a virtual screen > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,310,472 B2 - Document ID: US11310472B2 - Status: Granted - Kind: B2 - Application number: 16/759,655 - Filed: 2017-11-16 - Published: 2020-10-22 - Granted: 2022-04-19 - Earliest claimed: 2017-11-16 - Assignees: Sony Interactive Entertainment Inc. - Applicants: Sony Interactive Entertainment Inc. - Inventors: Takayuki Shinohara; Shinpei Yamaguchi; Tomo Matayoshi; Hidehiko Morisada - Web page: https://patentgrep.com/patent/US11310472B2 ## CPC - H04N 9/75: Chroma key - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/012: {Head tracking input arrangements} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06T 1/00: General purpose image data processing - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics - G09G 3/20: for presentation of an assembly of a number of characters, e.g. a page, by composing the assembly by combination of individual elements arranged in a matrix {no fixed position being assigned to or needed to be assigned to the individual characters or partial characters} - H04N 5/272: Means for inserting a foreground image in a background image, i.e. inlay, outlay - H04N 7/18: Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00) ## Abstract An image acquisition unit 40 acquires a subject image in which a subject is positioned in front of a background with a single color. An eye gaze information determination unit 32 determines an eye direction and a viewpoint position of a user wearing an HMD and provides the eye direction and the viewpoint position for a display control unit 50. The display control unit 50 performs chroma-key composition such that the subject included in the subject image may become a foreground image with respect to a background image and thereby generates a display image according to the viewpoint position of the user. The display control unit 50 determines an orientation of the subject such that the subject may be made to face the viewpoint position of the user, in a case where the viewpoint position has been changed. ## Claims ### Claim 1 An information processing device comprising: a three-dimensional data acquisition unit that acquires a three-dimensional space model produced from shot images of a space; an image acquisition unit that acquires a subject image in which a subject moves in front of a background with a single color; a subject layer image generation unit that projects the subject image onto a virtual screen moving in conformity with a movement of the subject included in the subject image and that extracts a subject portion through application of a chroma-key process; and a display processing unit that generates a display image according to a viewpoint position of a user with a subject layer image placed in the three-dimensional space model, wherein if it is determined in real-time by the information processing device that the subject is shielded by an object in the space, a portion of the subject portion is shielded in the display image generated by the display processing unit. ### Claim 2 (depends on 1) The information processing device according to claim 1, wherein the subject layer image generation unit determines an orientation of the virtual screen on which the subject image is projected, in accordance with a position of the subject included in the subject image. ### Claim 3 (depends on 1) The information processing device according to claim 1, wherein the subject image is an image in which the subject moving in the space imaged for production of the three-dimensional space model is imaged. ### Claim 4 (depends on 1) The information processing device according to claim 1, further comprising: a head-mounted display information acquisition unit that acquires posture information and positional information regarding a head-mounted display worn on a head part of the user; and an eye gaze information determination unit that determines an eye direction and the viewpoint position of the user in accordance with the posture information and the positional information regarding the head-mounted display, wherein the display control unit generates the display image in accordance with the eye direction and the viewpoint position. ### Claim 5 An image generation method comprising: acquiring a three-dimensional space model produced from shot images of a space; acquiring a subject image in which a subject moves in front of a background with a single color; projecting the subject image onto a virtual screen moving in conformity with a movement of the subject included in the subject image; extracting a subject portion through application of a chroma-key process to the virtual screen on which the subject image is projected; generating a display image according to a viewpoint position of a user with the subject placed in the three-dimensional space model; and determining in real-time if the subject is shielded by an object in the space, wherein if it is determined that the subject is shielded by the object in the space, a portion of the subject portion is shielded in the display image. ### Claim 6 A non-transitory computer readable medium having stored thereon a program for a computer, the program comprising: by a three-dimensional data acquisition unit, acquiring a three-dimensional space model produced from shot images of a space; by an image acquisition unit, acquiring a subject image in which a subject moves in front of a background with a single color; by a subject layer image generation unit, projecting the subject image onto a virtual screen moving in conformity with a movement of the subject included in the subject image; by the subject layer image generation unit, extracting a subject portion through application of a chroma-key process to the virtual screen on which the subject image is projected; and by a display processing unit, generating a display image according to a viewpoint position of a user with the subject placed in the three-dimensional space model, wherein a perpendicular drawn from the viewpoint position to the subject image passes through the center of the subject image during change in orientation of the subject image.