# US 2021/0005014 A1 · Non-transitory computer-readable medium, image processing method, and image processing system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0005014 A1
- Document ID: US20210005014A1
- Status: Published application
- Kind: A1
- Application number: 16/917,728
- Filed: 2020-06-30
- Published: 2021-01-07
- Earliest claimed: 2019-07-05
- Assignees: Square Enix Co., Ltd.
- Applicants: Square Enix Co., Ltd.
- Inventors: Masato HAYASHI
- Web page: https://patentgrep.com/patent/US20210005014A1

## CPC

- G06T 15/20: Perspective computation
- G06T 15/205: {Image-based rendering}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2219/2016: Rotation, translation, scaling
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A non-transitory computer-readable medium including a program is provided. The program causes a computer connected to an image capturing apparatus to: specify a virtual space; generate a virtual space image obtained by photographing the virtual space including at least one virtual object; output a synthetic image of the captured image and the virtual space image to a display device with a touch panel; determine a content of a user operation against the touch panel; update a position or a posture of each of the virtual camera and the virtual object when the user operation is a first operation in which a touch position against the touch panel does not move after the touch panel is touched; and rotate the virtual object in a predetermined direction when the user operation is a second operation in which the touch position is slid on the touch panel.

## Claims

### Claim 1

A non-transitory computer-readable medium including a program for causing a computer to perform functions, the computer being connected to an image capturing apparatus, the functions comprising:
a specifying function configured to specify a virtual space corresponding to a captured image, the captured image being taken by the image capturing apparatus;
a generating function configured to generate a virtual space image, the virtual space image being obtained by photographing the virtual space including at least one virtual object by means of a virtual camera corresponding to the image capturing apparatus;
an outputting function configured to output a synthetic image to a display device, the synthetic image being obtained by synthesizing the captured image and the virtual space image, the display device including a touch panel;
a determining function configured to determine a content of a user operation against the touch panel;
an updating function configured to update, in a case where it is determined that the user operation is a first operation, a position or a posture of each of the virtual camera and the virtual object in a state where a relationship between the virtual camera and the virtual object is fixed in accordance with movement or a change in a posture of the image capturing apparatus while the first operation is continuing, the first operation being an operation in which a touch position against the touch panel does not move after the touch panel is touched; and
a rotating function configured to rotate the virtual object in a predetermined direction in a case where it is determined that the user operation is a second operation, the second operation being an operation in which the touch position is slid on the touch panel, the predetermined direction being defined by the second operation.

### Claim 2 (depends on 1)

The non-transitory computer-readable medium according to claim 1,
wherein the updating function includes a function configured to update the position or the posture of the virtual object in the virtual space, the virtual object being displayed at the touch position of the first operation.

### Claim 3 (depends on 1)

The non-transitory computer-readable medium according to claim 1,
wherein the rotating function includes a function configured to rotate the virtual object in a state where the position of the virtual object in the virtual space is fixed.

### Claim 4 (depends on 1)

The non-transitory computer-readable medium according to claim 1,
wherein the updating function includes a function configured to update the posture of the virtual object so as to rotate the virtual object around a photographing direction of the virtual camera in a case where it is determined that the user operation is the first operation and a posture of the computer is rotated by using an imaging direction of the image capturing apparatus as an axis, and
wherein the rotating function includes a function configured to rotate the virtual object around a direction orthogonal to the photographing direction in a case where it is determined that the user operation is the second operation in which the touch position is slid on the touch panel.

### Claim 5 (depends on 1)

The non-transitory computer-readable medium according to claim 1,
wherein the updating function includes a function configured to change from the state where relationship between the virtual camera and the virtual object is fixed to a state where the relationship is not fixed and update the state in a case where a predetermined condition in the virtual space is satisfied while the first operation is continuing.

### Claim 6 (depends on 1)

The non-transitory computer-readable medium according to claim 1,
wherein the updating function includes a function configured to update the position or the posture of only the virtual camera in a case where it is determined that the user operation is an operation to change a posture of the computer without touching the touch panel.

### Claim 7

An image processing method of controlling progress of information processing in response to a user operation, the image processing method comprising:
a specifying process of specifying a virtual space corresponding to a captured image, the captured image being taken by an image capturing apparatus connected to a computer,
a generating process of generating a virtual space image, the virtual space image being obtained by photographing the virtual space including at least one virtual object by means of a virtual camera corresponding to the image capturing apparatus;
an outputting process of outputting a synthetic image to a display device, the synthetic image being obtained by synthesizing the captured image and the virtual space image, the display device including a touch panel;
a determining process of determining a content of a user operation against the touch panel;
an updating process of updating, in a case where it is determined that the user operation is a first operation, a position or a posture of each of the virtual camera and the virtual object in a state where a relationship between the virtual camera and the virtual object is fixed in accordance with movement or a change in a posture of the image capturing apparatus while the first operation is continuing, the first operation being an operation in which a touch position against the touch panel does not move after the touch panel is touched; and
a rotating process of rotating the virtual object in a predetermined direction in a case where it is determined that the user operation is a second operation, the second operation being an operation in which the touch position is slid on the touch panel, the predetermined direction being defined by the second operation.

### Claim 8

An image processing system for controlling progress of information processing in response to a user operation, the image processing system comprising a communication network, a server, and a user terminal to which an image capturing apparatus is connected, the image processing system further comprising:
a specifying unit configured to specify a virtual space corresponding to a captured image, the captured image being taken by the image capturing apparatus;
a generating unit configured to generate a virtual space image, the virtual space image being obtained by photographing the virtual space including at least one virtual object by means of a virtual camera corresponding to the image capturing apparatus;
an outputting unit configured to output a synthetic image to a display device, the synthetic image being obtained by synthesizing the captured image and the virtual space image, the display device including a touch panel;
a determining unit configured to determine a content of a user operation against the touch panel;
an updating unit configured to update, in a case where it is determined that the user operation is a first operation, a position or a posture of each of the virtual camera and the virtual object in a state where a relationship between the virtual camera and the virtual object is fixed in accordance with movement or a change in a posture of the image capturing apparatus while the first operation is continuing, the first operation being an operation in which a touch position against the touch panel does not move after the touch panel is touched; and
a rotating unit configured to rotate the virtual object in a predetermined direction in a case where it is determined that the user operation is a second operation, the second operation being an operation in which the touch position is slid on the touch panel, the predetermined direction being defined by the second operation.
