# US 11,115,631 B2 · Image processing apparatus, image processing method, and storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,115,631 B2 - Document ID: US11115631B2 - Status: Granted - Kind: B2 - Application number: 16/659,732 - Filed: 2019-10-22 - Published: 2020-04-30 - Granted: 2021-09-07 - Earliest claimed: 2018-10-26 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Yuichi Nakada - Web page: https://patentgrep.com/patent/US11115631B2 ## CPC - H04N 9/3141: {Constructional details thereof (details not peculiar to the presence of an electronic spatial light modulator G03B21/14)} - H04N 9/3147: {Multi-projection systems (video walls G06F3/1446)} - G03B 21/14: Details - H04N 5/74: Projection arrangements for image reproduction, e.g. using eidophor - H04N 9/3185: {Geometric adjustment, e.g. keystone or convergence (optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14)} - H04N 13/363: using image projection screens (volumetric display H04N13/388) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G03B: APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR (optical parts of such apparatus G02B; photosensitive materials or processes for photographic purposes G03C; apparatus for processing exposed photographic materials G03D) ## Abstract There is provided with an image processing apparatus. A setting unit sets a display magnification on the display surface for the captured image data. A generating unit generates the display image data from the captured image data, based on a relationship between a first direction from an observer viewpoint toward a position on the display surface at which an object is displayed when display is provided based on the captured image data in accordance with the display magnification and a second direction from an capturing viewpoint of the captured image data toward the object, such that an image of the object in the second direction from the capturing viewpoint of the captured image data is displayed at a position on the display surface in the first direction from the observer viewpoint. ## Claims ### Claim 1 An image processing apparatus for converting captured image data into display image data for display on a display surface, comprising one or more processors and one or more memories storing one or more programs, wherein the one or more programs cause the one or more processors to perform: setting a display magnification on the display surface for the captured image data of an image of an object; calculating, based on the display magnification and a first direction from an observer viewpoint toward a position on the display surface, a second direction from a capturing viewpoint of the captured image data toward the object; and generating, based on the captured image data, the display image data such that the image of the object in the second direction from the capturing viewpoint of the captured image data is displayed at a position on the display surface in the first direction from the observer viewpoint, wherein an elevation angle of the first direction is an elevation angle of the second direction multiplied by the display magnification, and an azimuth angle of the first direction is an azimuth angle of the second direction multiplied by the display magnification. ### Claim 2 (depends on 1) The image processing apparatus according to claim 1, wherein the calculating comprises: identifying the first direction for a position of interest on the display surface; and identifying the second direction based on the display magnification and the first direction, and wherein the generating comprises: identifying a pixel position on the captured image data corresponding to the second direction; and determining image information of the display image data at the position of interest on the display surface based on image information of the captured image data for the identified pixel position. ### Claim 3 (depends on 1) The image processing apparatus according to claim 1, wherein a ratio between an elevation angle and an azimuth angle of the first direction is equal to a ratio between an elevation angle and an azimuth angle of the second direction. ### Claim 4 (depends on 1) The image processing apparatus according to claim 1, wherein an elevation angle in the first direction is an elevation angle in the second direction multiplied by the display magnification and an azimuth angle in the first direction is an azimuth angle in the second direction multiplied by the display magnification. ### Claim 5 (depends on 1) The image processing apparatus according to claim 1, wherein the setting includes setting the display magnification based on a field angle from the capturing viewpoint corresponding to at least a partial region of the captured image data and a viewing angle of the display surface from the observer viewpoint. ### Claim 6 (depends on 1) The image processing apparatus according to claim 1, wherein the setting includes setting the display magnification such that a viewing angle from the observer viewpoint with respect to the display image data displayed on the display surface is equal to a viewing angle of the display surface from the observer viewpoint in a case where a field angle from the capturing viewpoint corresponding to the captured image data is smaller than the viewing angle of the display surface. ### Claim 7 (depends on 1) The image processing apparatus according to claim 1, wherein the generating includes generating, based on the captured image data, the display image data such that, at a position on the display surface in the first direction from the observer viewpoint of the display surface, the image of the object in the second direction from the capturing viewpoint of the captured image data is displayed, and wherein a ratio between an elevation angle and an azimuth angle of the first direction is equal to a ratio between an elevation angle and an azimuth angle of the second direction. ### Claim 8 (depends on 1) The image processing apparatus according to claim 1, wherein the one or more programs further cause the one or more processors to perform acquiring display environment information identifying a direction from the observer viewpoint of the display surface toward each position on the display surface. ### Claim 9 (depends on 8) The image processing apparatus according to claim 8, wherein the display environment information comprises position information of the observer viewpoint and arrangement information of the display surface. ### Claim 10 (depends on 1) The image processing apparatus according to claim 1, wherein the one or more programs further cause the one or more processors to perform acquiring, for each pixel of the captured image data, capturing environment information identifying a direction from a capturing viewpoint of an image capturing apparatus having acquired the captured image data toward an object projected on the pixel. ### Claim 11 (depends on 10) The image processing apparatus according to claim 10, wherein the capturing environment information includes field angle information of the image capturing apparatus. ### Claim 12 (depends on 1) The image processing apparatus according to claim 1, wherein the one or more programs further cause the one or more processors to perform acquiring a table indicating correspondence between pixel positions in the display image data corresponding to the first direction and pixel positions in the captured image data corresponding to the second direction. ### Claim 13 (depends on 12) The image processing apparatus according to claim 12, wherein the acquiring includes acquiring the table in response to a change in at least one of display environment information identifying a direction from the observer viewpoint of the display surface toward each position on the display surface, capturing environment information identifying, for each pixel in the captured image data, a direction from the capturing viewpoint of an image capturing apparatus having acquired the captured image data toward the object projected on the pixel, or the display magnification on the display surface for the captured image data. ### Claim 14 (depends on 1) The image processing apparatus according to claim 1, wherein the generating includes generating the display image such that aberrations caused by an image capturing apparatus having acquired the captured image data is compensated. ### Claim 15 (depends on 1) The image processing apparatus according to claim 1, wherein the generating includes correcting the display image data such that corrected image data has a predetermined image size with a center of the display image data maintained. ### Claim 16 An image processing method for converting captured image data into display image data for display on a display surface, comprising: setting a display magnification on the display surface for the captured image data of an image of an object; calculating, based on the display magnification and a first direction from an observer viewpoint toward a position on the display surface, a second direction from a capturing viewpoint of the captured image data toward the object; and generating, based on the captured image data, the display image data such that the image of the object in the second direction from the capturing viewpoint of the captured image data is displayed at a position on the display surface in the first direction from the observer viewpoint, wherein an elevation angle of the first direction is an elevation angle of the second direction multiplied by the display magnification, and an azimuth angle of the first direction is an azimuth angle of the second direction multiplied by the display magnification. ### Claim 17 (depends on 16) The method according to claim 16, wherein the calculating comprises: identifying the first direction for a position of interest on the display surface; and identifying the second direction based on the display magnification and the first direction, and wherein the generating comprises: identifying a pixel position on the captured image data corresponding to the second direction; and determining image information of the display image data at the position of interest on the display surface based on image information of the captured image data for the identified pixel position. ### Claim 18 A non-transitory computer-readable medium storing a program which, when executed by a computer comprising a processor and a memory, causes the computer to perform: setting a display magnification on a display surface for captured image data of an image of an object; calculating, based on the display magnification and a first direction from an observer viewpoint toward a position on the display surface, a second direction from a capturing viewpoint of the captured image data toward the object; and generating, based on the captured image data, display image data such that the image of the object in the second direction from the capturing viewpoint of the captured image data is displayed at a position on the display surface in the first direction from the observer viewpoint, wherein an elevation angle of the first direction is an elevation angle of the second direction multiplied by the display magnification, and an azimuth angle of the first direction is an azimuth angle of the second direction multiplied by the display magnification. ### Claim 19 (depends on 18) The medium according to claim 18, wherein the calculating comprises: identifying the first direction for a position of interest on the display surface; and identifying the second direction based on the display magnification and the first direction, and wherein the generating comprises: identifying a pixel position on the captured image data corresponding to the second direction; and determining image information of the display image data at the position of interest on the display surface based on image information of the captured image data for the identified pixel position.