# US 11,308,670 B2 · Image processing apparatus and method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,308,670 B2 - Document ID: US11308670B2 - Status: Granted - Kind: B2 - Application number: 16/494,063 - Filed: 2018-03-09 - Published: 2021-04-29 - Granted: 2022-04-19 - Earliest claimed: 2017-03-22 - Assignees: Sony Corporation - Applicants: Sony Corporation - Inventors: Kenji Tanaka; Satoshi Ihara; Cody Poultney - Web page: https://patentgrep.com/patent/US11308670B2 ## CPC - G02B 27/017: {Head mounted} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/014: {Hand-worn input/output arrangements, e.g. data gloves} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/033: Pointing devices displaced or positioned by the user {, e.g. mice, trackballs, pens or joysticks}; Accessories therefor (digitisers characterised by the transducing means G06F3/041) - G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors - G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object - G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour - G06T 11/60: Creating or editing images; Combining images with text - G06T 15/205: {Image-based rendering} - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G02B 2027/0138: {comprising image capture systems, e.g. camera} - G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye} - G06F 3/03547: {Touch pads, in which fingers can move on a surface} - G06F 3/0482: Interaction with lists of selectable items, e.g. menus - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract The present technology relates to an image processing apparatus and method, and a program that are capable of more facilitating editing. The image processing apparatus includes: an operation detection unit configured to detect an operation input by an operation unit; and a display control unit configured to output, to an immersive presentation device, part or entirety of a spherical image on which an image of the operation unit is superimposed, as a presentation image, to cause the immersive presentation device to display the presentation image, the display control unit being configured to control, in a case where the operation input has been detected, the display of the presentation image such that the operation input is reflected. The present technology is applicable to an editing system. ## Claims ### Claim 1 An image processing apparatus, comprising: circuitry configured to: control an immersive presentation device to display a spherical image; superimpose an image of a control device on the spherical image to generate a presentation image representing the image of the control device in the immersive presentation device, wherein the presentation image includes an auxiliary screen that comprises a plurality of buttons, and each button of the plurality of buttons is associated with a corresponding edit operation of the spherical image; control the immersive presentation device to display the presentation image; detect an operation input from the control device, wherein the operation input corresponds to selection of at least one button of the plurality of buttons in the auxiliary screen such that the operation input is reflected on the presentation image; and edit, based on the detected operation input associated with the edit, the spherical image in the presentation image displayed on the immersive presentation device. ### Claim 2 (depends on 1) The image processing apparatus according to claim 1, wherein the circuitry is further configured to: generate the image of the control device based on a captured image of the control device, wherein the control device is captured as a subject in the captured image. ### Claim 3 (depends on 2) The image processing apparatus according to claim 2, wherein the circuitry is further configured to: control display of the image of the control device at a specific position on the spherical image, wherein the specific position of the image of the control device is based on a position of the immersive presentation device and an orientation of the immersive presentation device. ### Claim 4 (depends on 1) The image processing apparatus according to claim 1, wherein the circuitry is further configured to: generate an image of a model of the control device as the image of the control device. ### Claim 5 (depends on 4) The image processing apparatus according to claim 4, wherein the circuitry is further configured to: control display of the image of the control device at a specific position on the spherical image, and wherein the specific position is based on a position of the control device and an orientation of the control device. ### Claim 6 (depends on 1) The image processing apparatus according to claim 1, wherein the auxiliary screen is at a specific position on the presentation image. ### Claim 7 (depends on 1) The image processing apparatus according to claim 1, wherein the auxiliary screen is at a specific position on the presentation image, and the specific position is based on a position of the immersive presentation device and an orientation of the immersive presentation device. ### Claim 8 (depends on 1) The image processing apparatus according to claim 1, wherein the circuitry is further configured to: output the presentation image to a plurality of immersive presentation devices. ### Claim 9 (depends on 1) The image processing apparatus according to claim 1, the circuitry is further configured to detect the operation input based on a position of the immersive presentation device and an orientation of the immersive presentation device. ### Claim 10 An image processing method, comprising: in an image processing apparatus: controlling an immersive presentation device to display a spherical image; superimposing an image of a control device on the spherical image to generate a presentation image representing the image of the control device in the immersive presentation device, wherein the presentation image includes an auxiliary screen that comprises a plurality of buttons, and each button of the plurality of buttons is associated with a corresponding edit operation of the spherical image; controlling the immersive presentation device to display the presentation image; detecting an operation input from the control device, wherein the operation input corresponds to selection of at least one button of the plurality of buttons in the auxiliary screen such that the operation input is reflected on the presentation image; and editing, based on the detected operation input associated with the edit, the spherical image in the presentation image displayed in the immersive presentation device. ### Claim 11 A non-transitory computer-readable medium having stored thereon computer-executable instructions which, when executed by a computer, cause the computer to execute operations, the operations comprising: controlling an immersive presentation device to display a spherical image; superimposing an image of a control device on the spherical image to generate a presentation image representing the image of the control device in the immersive presentation device, wherein the presentation image includes an auxiliary screen that comprises a plurality of buttons, and each button of the plurality of buttons is associated with a corresponding edit operation of the spherical image; controlling the immersive presentation device to display the presentation image; detecting an operation input from the control device, wherein the operation input corresponds to selection of at least one button of the plurality of buttons in the auxiliary screen such that the operation input is reflected on the presentation image; and editing, based on the detected operation input associated with the edit, the spherical image in the presentation image displayed on the immersive presentation device.