# US 12,771,816 B2 · Image processing apparatus, image processing method, and program > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,771,816 B2 - Document ID: US12771816B2 - Status: Granted - Kind: B2 - Application number: 18/049,623 - Filed: 2022-10-25 - Published: 2023-03-02 - Granted: 2026-10-20 - Earliest claimed: 2020-04-27 - Assignees: FUJIFILM Corporation - Applicants: FUJIFILM Corporation - Inventors: Yasunori Murakami; Masahiko Miyata; Takashi Aoki; Fuminori Irie; Kazunori Tamura - Web page: https://patentgrep.com/patent/US12771816B2 ## CPC - H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems - G06F 3/14: Digital output to display device {; Cooperation and interconnection of the display device with other functional units} - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G06V 10/761: {Proximity, similarity or dissimilarity measures} - H04N 13/383: for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract An image processing apparatus generates and outputs a virtual viewpoint image with reference to a position and an orientation of a target object included in an imaging region on the basis of a plurality of images obtained by imaging the imaging region with a plurality of imaging devices of which at least either of imaging positions or imaging directions are different, and controls a display aspect of the virtual viewpoint image according to an amount of temporal changes in at least one of the position or the orientation. ## Claims ### Claim 1 An image processing apparatus comprising: a processor; and a memory built in or connected to the processor, wherein the processor generates and outputs a virtual viewpoint image with reference to a position and an orientation of a target object, wherein the target object is included in an imaging region and a plurality of images on the basis of the plurality of images obtained by imaging the imaging region with a plurality of imaging devices of which at least either of imaging positions or imaging directions are different, generates an adjustment position and an adjustment orientation based on the position and the orientation of the target object by smoothing an amount of temporal changes in at least one of the position or the orientation of the target object, controls a display aspect of the virtual viewpoint image by generating and outputting the virtual viewpoint image with reference to the adjustment position and the adjustment orientation, and controls a display region of the virtual viewpoint image by controlling a resolution for each region included in the display region of the virtual viewpoint image, wherein the display region of the virtual viewpoint image is divided into a facing region facing the orientation and a peripheral region surrounding the facing region, and the processor sets a resolution of the peripheral region to be lower than a resolution of the facing region. ### Claim 2 (depends on 1) The image processing apparatus according to claim 1, wherein the processor controls the display aspect according to the amount of temporal changes smaller than an actual amount of temporal changes in the position and the orientation of the target object. ### Claim 3 (depends on 1) The image processing apparatus according to claim 1, wherein the processor smooths the amount of temporal changes by obtaining a moving average of an amount of time-series changes in the position and the orientation. ### Claim 4 (depends on 1) The image processing apparatus according to claim 1, wherein the processor controls the display aspect of the virtual viewpoint image according to the amount of temporal changes in a case where the amount of temporal changes is within a predetermined range. ### Claim 5 (depends on 1) The image processing apparatus according to claim 1, wherein the processor changes a time interval for generating the virtual viewpoint image according to the amount of temporal changes. ### Claim 6 (depends on 5) The image processing apparatus according to claim 5, wherein, in a case where the amount of temporal changes is equal to or more than a first predetermined value, the processor sets the time interval to be shorter than a first reference time interval. ### Claim 7 (depends on 6) The image processing apparatus according to claim 6, wherein, in a case where the amount of temporal changes is less than the first predetermined value and the time interval is different from a second reference time interval, the processor sets the time interval to the second reference time interval. ### Claim 8 (depends on 5) The image processing apparatus according to claim 5, wherein, in a case where the amount of temporal changes is equal to or less than a first predetermined value, the processor sets the time interval to be longer than a second reference time interval. ### Claim 9 (depends on 8) The image processing apparatus according to claim 8, wherein, in a case where the amount of temporal changes exceeds the first predetermined value and the time interval is different from the second reference time interval, the processor sets the time interval to the second reference time interval. ### Claim 10 (depends on 5) The image processing apparatus according to claim 5, wherein the processor further changes the time interval for generating the virtual viewpoint image according to an instruction received by a reception device. ### Claim 11 (depends on 10) The image processing apparatus according to claim 10, wherein the instruction is an instruction related to a display speed of the virtual viewpoint image. ### Claim 12 (depends on 11) The image processing apparatus according to claim 11, wherein, in a case where the instruction is an instruction for setting the display speed to be lower than a first reference display speed, the processor sets the time interval to be shorter than a third reference time interval. ### Claim 13 (depends on 11) The image processing apparatus according to claim 11, wherein, in a case where the instruction is an instruction for setting the display speed to be higher than a second reference display speed, the processor sets the time interval to be longer than a fourth reference time interval. ### Claim 14 (depends on 1) The image processing apparatus according to claim 1, wherein the processor reduces the resolution of the peripheral region as a distance from the facing region increases. ### Claim 15 (depends on 1) The image processing apparatus according to claim 1, wherein the processor generates and outputs information indicating a positional relationship between a display image and the virtual viewpoint image, the display image being different from the virtual viewpoint image and showing at least a part of the imaging region, on the basis of a deviation between an imaging direction for obtaining the display image and the orientation. ### Claim 16 (depends on 15) The image processing apparatus according to claim 15, wherein the information indicating the positional relationship is information that is visually recognized by a viewer of the virtual viewpoint image. ### Claim 17 (depends on 16) The image processing apparatus according to claim 16, wherein the information indicating the positional relationship is an arrow indicating a direction from a position of the display image to a position of the virtual viewpoint image. ### Claim 18 (depends on 17) The image processing apparatus according to claim 17, wherein the processor expands and contracts a length of the arrow according to a distance between the position of the display image and the position of the virtual viewpoint image. ### Claim 19 (depends on 15) The image processing apparatus according to claim 15, wherein the information indicating the positional relationship is information including at least one of information that is tactilely recognized by a viewer of the virtual viewpoint image or information that is audibly recognized by the viewer. ### Claim 20 (depends on 15) The image processing apparatus according to claim 15, wherein the processor performs control of switching an image to be displayed on a display from the display image to the virtual viewpoint image on condition that an instruction for switching from the display image to the virtual viewpoint image is given in a state in which the display image is displayed on the display. ### Claim 21 (depends on 1) The image processing apparatus according to claim 1, wherein the processor generates and outputs a display screen in which the virtual viewpoint images are arranged in a time series. ### Claim 22 (depends on 1) The image processing apparatus according to claim 1, wherein the target object is a specific person, the position is a viewpoint position of the person, and the orientation is a line-of-sight direction of the person. ### Claim 23 An image processing method comprising: generating and outputting a virtual viewpoint image with reference to a position and an orientation of a target object, wherein the target object is included in an imaging region and a plurality of images on the basis of the plurality of images obtained by imaging the imaging region with a plurality of imaging devices of which at least either of imaging positions or imaging directions are different; generating an adjustment position and an adjustment orientation based on the position and the orientation of the target object by smoothing an amount of temporal changes in at least one of the position or the orientation of the target object; controlling a display aspect of the virtual viewpoint image by generating and outputting the virtual viewpoint image with reference to the adjustment position and the adjustment orientation; and controlling a display region of the virtual viewpoint image by controlling a resolution for each region included in the display region of the virtual viewpoint image, wherein the display region of the virtual viewpoint image is divided into a facing region facing the orientation and a peripheral region surrounding the facing region, and a resolution of the peripheral region is set to be lower than a resolution of the facing region. ### Claim 24 A non-transitory computer-readable storage medium storing a program executable by a computer to perform a process comprising: generating and outputting a virtual viewpoint image with reference to a position and an orientation of a target object, wherein the target object is included in an imaging region and a plurality of images on the basis of the plurality of images obtained by imaging the imaging region with a plurality of imaging devices of which at least either of imaging positions or imaging directions are different; generating an adjustment position and an adjustment orientation based on the position and the orientation of the target object by smoothing an amount of temporal changes in at least one of the position or the orientation of the target object; controlling a display aspect of the virtual viewpoint image by generating and outputting the virtual viewpoint image with reference to the adjustment position and the adjustment orientation; and controlling a display region of the virtual viewpoint image by controlling a resolution for each region included in the display region of the virtual viewpoint image, wherein the display region of the virtual viewpoint image is divided into a facing region facing the orientation and a peripheral region surrounding the facing region, and a resolution of the peripheral region is set to be lower than a resolution of the facing region.