# US 2021/0241425 A1 · Image processing apparatus, image processing system, image processing method, and medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0241425 A1 - Document ID: US20210241425A1 - Status: Published application - Kind: A1 - Application number: 17/159,481 - Filed: 2021-01-27 - Published: 2021-08-05 - Earliest claimed: 2020-01-31 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Hikaru Uchidate; Hiroichi Yamaguchi - Web page: https://patentgrep.com/patent/US20210241425A1 ## CPC - G02B 27/0172: {characterised by optical features} - G06T 5/50: using two or more images, e.g. averaging or subtraction - G02B 2027/0138: {comprising image capture systems, e.g. camera} - G02B 2027/014: {comprising information/image processing systems} - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06T 2207/10016: Video; Image sequence - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract There is provided with an image processing apparatus. An acquisition unit acquires, frame by frame, a video to be displayed in a display unit that is obtained by performing rendering processing on each frame of a captured video. An estimation unit estimates an ID of a frame to be acquired by the acquisition unit. A determination unit determines a drop in the frame rate of the video acquired by the acquisition unit, based on a comparison between an ID of a frame acquired by the acquisition unit and the ID estimated by the estimation unit. ## Claims ### Claim 1 An image processing apparatus comprising: an acquisition unit configured to acquire, frame by frame, a video to be displayed in a display unit that is obtained by performing rendering processing on each frame of a captured video; an estimation unit configured to estimate an ID of a frame to be acquired by the acquisition unit; and a determination unit configured to determine a drop in the frame rate of the video acquired by the acquisition unit, based on a comparison between an ID of a frame acquired by the acquisition unit and the ID estimated by the estimation unit. ### Claim 2 (depends on 1) The image processing apparatus according to claim 1, wherein the estimation unit is further configured to update an ID to be estimated at a period corresponding to a frame rate of the captured video. ### Claim 3 (depends on 1) The image processing apparatus according to claim 1, wherein the estimation unit is further configured to update an ID to be estimated at an update timing of a frame of the captured video acquired by an image capturing unit included in the image processing apparatus. ### Claim 4 (depends on 1) The image processing apparatus according to claim 1, wherein the estimation unit is further configured to update an ID to be estimated at a period corresponding to a frame rate of a video in the display unit. ### Claim 5 (depends on 1) The image processing apparatus according to claim 1, wherein the estimation unit is further configured to update an ID to be estimated at an update timing of a frame of a video in the display unit. ### Claim 6 (depends on 1) The image processing apparatus according to claim 1, wherein the estimation unit is further configured to update an ID to be estimated based on an issuance order of the ID to a frame to be acquired by the acquisition unit, and correct the ID to be estimated if a prescribed condition is satisfied. ### Claim 7 (depends on 6) The image processing apparatus according to claim 6, wherein the frame rate of the captured video is faster than the frame rate of a video in the display unit, and the estimation unit is further configured to, when the ID of the frame acquired by the acquisition unit advances relative to the ID estimated by the estimation unit, correct the estimated ID so as to match the ID of the frame acquired by the acquisition unit. ### Claim 8 (depends on 1) The image processing apparatus according to claim 1, wherein the determination unit is further configured to determine, if the ID of the frame acquired by the acquisition unit differs from the ID estimated by the estimation unit, that the frame rate of the video acquired by the acquisition unit has dropped. ### Claim 9 (depends on 1) The image processing apparatus according to claim 1, wherein the determination unit is further configured to determine a delay in the rendering processing regarding the frame acquired by the acquisition unit based on the ID of the frame acquired by the acquisition unit and the ID estimated by the estimation unit. ### Claim 10 (depends on 1) The image processing apparatus according to claim 1, further comprising a correction unit configured to perform, if the determination unit has determined that the frame rate of the video acquired by the acquisition unit has dropped, image correction on the frame acquired by the acquisition unit. ### Claim 11 (depends on 10) The image processing apparatus according to claim 10, wherein the correction unit is further configured to perform the image correction based on a delay in the rendering processing regarding the frame acquired by the acquisition unit. ### Claim 12 (depends on 11) The image processing apparatus according to claim 11, wherein the correction unit is further configured to perform the image correction based on a change in orientation of the display unit. ### Claim 13 (depends on 11) The image processing apparatus according to claim 11, wherein the image correction is shifting processing on an image or cutout processing from an image. ### Claim 14 (depends on 1) The image processing apparatus according to claim 1, wherein the image processing apparatus is a head mounted display. ### Claim 15 (depends on 1) The image processing apparatus according to claim 1, wherein the captured video is captured by an image capturing unit included in the image processing apparatus. ### Claim 16 An image processing apparatus comprising: an image capturing unit configured to capture a video including a plurality of frames; an acquisition unit configured to acquire, frame by frame, a video obtained by performing rendering processing on each frame of the captured video; a correction unit configured to perform image correction on a frame acquired by the acquisition unit; and a display unit configured to display a video output from the correction unit, wherein a period in which the display unit displays a video is divided into a plurality of subperiods, a corresponding input period is prescribed to each of the plurality of subperiods, the display unit is further configured to update the frame to be displayed at periodical update timings, and the correction unit is further configured to determine whether or not the frame to be displayed in the display unit at a specific update timing has been acquired by the image capturing unit at the input period corresponding to the subperiod that includes the update timing, and perform image correction on a frame to be displayed in the display unit at the specific update timing in accordance with a result of the determination. ### Claim 17 An image processing system comprising: an image capturing unit configured to capture a video including a plurality of frames; an assignment unit configured to assign an ID to each frame of the acquired video; an image processing unit configured to output, frame by frame, a video to be displayed in a display unit by performing rendering processing on each frame of the video; an estimation unit configured to estimate an ID of a frame to be output from the image processing unit; and a determination unit configured to determine a drop in frame rate of the video output from the image processing unit, based on comparison between the ID of a frame output from the image processing unit and the ID estimated by the estimation unit. ### Claim 18 (depends on 17) The image processing system according to claim 17, wherein the image processing system includes a head mounted display and an image processing apparatus, the head mounted display includes: the image capturing unit; a first transmission unit configured to transmit the video captured by the image capturing unit to the image processing unit; and the display unit, and the image processing apparatus includes: the image processing unit; and a second transmission unit configured to transmit the video output from the image processing unit to the head mounted display. ### Claim 19 An image processing method comprising: acquiring, frame by frame, a video to be displayed in a display unit that is obtained by performing rendering processing on each frame of a captured video; estimating an ID of a frame to be acquired; and determining a drop in the frame rate of the acquired video, based on a comparison between an ID of an acquired frame and the estimated ID. ### Claim 20 A non-transitory computer-readable medium storing a program which, when executed by a computer comprising a processor and a memory, causes the computer to perform a method comprising: acquiring, frame by frame, a video to be displayed in a display unit that is obtained by performing rendering processing on each frame of a captured video; estimating an ID of a frame to be acquired; and determining a drop in the frame rate of the acquired video, based on a comparison between an ID of an acquired frame and the estimated ID.