# US 12,114,071 B2 · Image processing apparatus, system, image processing method, and image processing program > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,114,071 B2 - Document ID: US12114071B2 - Status: Granted - Kind: B2 - Application number: 17/783,080 - Filed: 2020-12-03 - Published: 2023-01-26 - Granted: 2024-10-08 - Earliest claimed: 2019-12-11 - Assignees: Sony Interactive Entertainment Inc. - Applicants: Sony Interactive Entertainment Inc. - Inventors: Naoyuki Miyada - Web page: https://patentgrep.com/patent/US12114071B2 ## CPC - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - H04N 23/683: {performed by a processor, e.g. controlling the readout of an image memory} - H04N 23/6811: {based on the image signal} - H04N 23/6815: {by distinguishing pan or tilt from motion} - G06T 2207/30244: Camera pose - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract An image processing apparatus includes a reception unit that receives an image signal acquired by an image sensor, an acceptance unit that accepts operation input made by a user for driving a device carrying the image sensor, a calculation unit that calculates a movement amount of the image sensor in an entire angle of view according to the image signal and the operation input, and a correction processing unit that performs a correction process for an image constructed from the image signal, according to the movement amount. ## Claims ### Claim 1 An image processing apparatus comprising: a reception unit that receives an image signal acquired by an image sensor; an acceptance unit that accepts operation input made by a user for driving a device carrying the image sensor; a calculation unit that calculates a movement amount of the image sensor in an entire angle of view, according to the image signal and the operation input; and a correction processing unit that performs a correction process for an image constructed from the image signal, according to the movement amount, wherein the calculation unit calculates a first movement amount according to the image signal, calculates a second movement amount according to the operation input, and calculates the movement amount in the entire angle of view by subtracting the second movement amount from the first movement amount. ### Claim 2 (depends on 1) The image processing apparatus according to claim 1, wherein the image sensor includes an event-driven vision sensor that asynchronously generates the image signal when an intensity change of light entering for each pixel is detected. ### Claim 3 (depends on 2) The image processing apparatus according to claim 2, wherein the image sensor further includes a camera that generates the image signal by synchronously scanning all pixels at a predetermined timing, the reception unit receives the image signals from both the vision sensor and the camera, the calculation unit calculates the movement amount in the entire angle of view according to at least the image signal generated by the vision sensor and the operation input, and the correction processing unit performs the correction process for the image constructed from the image signal generated by the camera, according to the movement amount. ### Claim 4 (depends on 2) The image processing apparatus according to claim 2, wherein the correction processing unit constructs the image by performing an interpolation process for the image signal generated by the vision sensor, and performs the correction process for the constructed image. ### Claim 5 (depends on 1) The image processing apparatus according to claim 1, wherein the operation input is operation input made by the user for driving a portion that is included in the device and that is capable of simultaneously displacing and rotating the image sensor integrated into one body. ### Claim 6 (depends on 1) The image processing apparatus according to claim 1, wherein the device is a device attachable to a head of the user, and the acquisition unit acquires information indicating a movement of the head of the user as the information indicating the movement of the user. ### Claim 7 (depends on 6) The image processing apparatus according to claim 6, wherein the acquisition unit further acquires information associated with a visual line of the user as the information indicating the movement of the user. ### Claim 8 An image processing apparatus comprising: a reception unit that receives an image signal acquired by an image sensor; an acquisition unit that acquires information indicating a movement of a user that causes a shift of a device carrying the image sensor and attachable to the body of the user; a calculation unit that calculates a movement amount of the image sensor in an entire angle of view, according to the image signal and the information indicating the movement of the user; and a correction processing unit that performs a correction process for an image constructed from the image signal, according to the movement amount, wherein the calculation unit calculates a first movement amount according to the image signal, calculates a second movement amount according to the information indicating the movement of the user, and calculates the movement amount in the entire angle of view by subtracting the second movement amount from the first movement amount. ### Claim 9 (depends on 8) The image processing apparatus according to claim 8, wherein the image sensor includes an event-driven vision sensor that asynchronously generates the image signal when an intensity change of light entering for each pixel is detected. ### Claim 10 (depends on 9) The image processing apparatus according to claim 9, wherein the image sensor further includes a camera that generates the image signal by synchronously scanning all pixels at a predetermined timing, the reception unit receives the image signals from both the vision sensor and the camera, the calculation unit calculates the movement amount in the entire angle of view according to at least the image signal generated by the vision sensor and the information indicating the movement of the user, and the correction processing unit performs the correction process for an image constructed from the image signal generated by the camera, according to the movement amount. ### Claim 11 (depends on 9) The image processing apparatus according to claim 9, wherein the correction processing unit constructs the image by performing an interpolation process for the image signal generated by the vision sensor, and performs the correction process for the constructed image. ### Claim 12 A system comprising: an image sensor that acquires an image signal; an acceptance unit that accepts operation input made by a user for driving a device carrying the image sensor; a drive unit that drives the device carrying the image sensor, according to the operation input; and a terminal apparatus, the terminal apparatus including a reception unit that receives the image signal, a calculation unit that calculates a movement amount of the image sensor in an entire angle of view, according to the image signal and the operation input, and a correction processing unit that performs a correction process for an image constructed from the image signal, according to the movement amount, wherein the calculation unit calculates a first movement amount according to the image signal, calculates a second movement amount according to the operation input, and calculates the movement amount in the entire angle of view by subtracting the second movement amount from the first movement amount. ### Claim 13 A system comprising: an image sensor that acquires an image signal; an acquisition unit that acquires information indicating a movement of a user that causes a shift of a device carrying the image sensor and attachable to the body of the user; and a terminal apparatus, the terminal apparatus including a reception unit that receives the image signal, a calculation unit that calculates a movement amount of the image sensor in an entire angle of view, according to the image signal and the information indicating the movement of the user, and a correction unit that performs a correction process for an image constructed from the image signal, according to the movement amount, wherein the calculation unit calculates a first movement amount according to the image signal, calculates a second movement amount according to the information indicating the movement of the user, and calculates the movement amount in the entire angle of view by subtracting the second movement amount from the first movement amount. ### Claim 14 An image processing method comprising: receiving an image signal acquired by an image sensor; accepting operation input made by a user for driving a device carrying the image sensor; calculating a movement amount of the image sensor in an entire angle of view according to the image signal and the operation input; and performing a correction process for an image constructed from the image signal, according to the movement amount, wherein the calculating includes calculating a first movement amount according to the image signal, calculating a second movement amount according to the operation input, and calculating the movement amount in the entire angle of view by subtracting the second movement amount from the first movement amount. ### Claim 15 An image processing method comprising: receiving an image signal acquired by an image sensor; acquiring information indicating a movement of a user that causes a shift of a device carrying the image sensor and attachable to the body of the user; calculating a movement amount of the image sensor in an entire angle of view according to the image signal and the information indicating the movement of the user; and performing a correction process for an image constructed from the image signal, according to the movement amount, wherein the calculation unit calculates a first movement amount according to the image signal, calculates a second movement amount according to the information indicating the movement of the user, and calculates the movement amount in the entire angle of view by subtracting the second movement amount from the first movement amount. ### Claim 16 A non-transitory, computer readable storage medium containing an image processing program, which when executed by a computer, causes the computer to perform an image processing method by carrying out actions, comprising: receiving an image signal acquired by an image sensor; accepting operation input made by a user for driving a device carrying the image sensor; calculating a movement amount of the image sensor in an entire angle of view according to the image signal and the operation input; and performing a correction process for an image constructed from the image signal, according to the movement amount, wherein the calculating includes calculating a first movement amount according to the image signal, calculating a second movement amount according to the operation input, and calculating the movement amount in the entire angle of view by subtracting the second movement amount from the first movement amount. ### Claim 17 A non-transitory, computer readable storage medium containing an image processing program, which when executed by a computer, causes the computer to perform an image processing method by carrying out actions, comprising: receiving an image signal acquired by an image sensor; acquiring information indicating a movement of a user that causes a shift of a device carrying the image sensor and attachable to the body of the user; calculating a movement amount of the image sensor in an entire angle of view according to the image signal and the information indicating the movement of the user; and performing a correction process for an image constructed from the image signal, according to the movement amount, wherein the calculation unit calculates a first movement amount according to the image signal, calculates a second movement amount according to the information indicating the movement of the user, and calculates the movement amount in the entire angle of view by subtracting the second movement amount from the first movement amount.