# US 12,113,947 B2 · Image processing apparatus, image processing method, and program for viewpoint transition control > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,113,947 B2 - Document ID: US12113947B2 - Status: Granted - Kind: B2 - Application number: 17/794,413 - Filed: 2021-01-26 - Published: 2023-03-02 - Granted: 2024-10-08 - Earliest claimed: 2020-02-10 - Assignees: Sony Group Corporation - Applicants: Sony Group Corporation - Inventors: Sho Ogura; Yuya Yamashita - Web page: https://patentgrep.com/patent/US12113947B2 ## CPC - H04N 13/117: the virtual viewpoint locations being selected by the viewers or determined by viewer tracking - H04N 21/21805: {enabling multiple viewpoints, e.g. using a plurality of cameras} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/012: {Head tracking input arrangements} - G06F 3/14: Digital output to display device {; Cooperation and interconnection of the display device with other functional units} - G06T 19/003: {Navigation within 3D models or images} - G09G 5/36: characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory - H04N 21/23614: {Multiplexing of additional data and video streams (arrangements for simultaneous broadcast of plural pieces of information H04H20/28)} - H04N 21/42202: {environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes} - H04N 21/4318: {by altering the content in the rendering process, e.g. blanking, blurring or masking an image region (image enhancement or restoration in general G06T5/00)} - H04N 21/4348: {Demultiplexing of additional data and video streams} - H04N 21/440245: {the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment} - H04N 21/4728: for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region - H04N 21/816: {involving special video data, e.g 3D video} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00) ## Abstract A display apparatus (10) that is an example of an image processing apparatus includes: an output unit that outputs part of an image including recommended viewpoint information as a display image to a display unit; and a transition control unit that causes the display range of the image to shift, on the basis of the positional relationship between the viewpoint position corresponding to the display image output to the display unit and the viewpoint position corresponding to the recommended viewpoint information. The output unit outputs part of the image to the display unit, on the basis of the display range that has been made to shift. ## Claims ### Claim 1 An image processing apparatus comprising: an output unit configured to output part of an image including recommended viewpoint information as a display image to a display unit; and a transition control unit configured to cause a display range of the image to shift, based on a positional relationship between a viewpoint position corresponding to the display image output to the display unit and a viewpoint position corresponding to the recommended viewpoint information, wherein the transition control unit causes the display range to shift at a speed determined based on a difference in an angle of a line-of-sight direction from the viewpoint position corresponding to the recommended viewpoint information to the viewpoint position corresponding to the display image, wherein the output unit outputs the part of the image to the display unit based on the display range that has been made to shift, wherein the transition control unit includes: an operation movement amount calculation unit configured to calculate an operation movement amount for moving the viewpoint position corresponding to the display image in accordance with a viewpoint operation based on viewpoint operation information; a return movement amount calculation unit configured to calculate a return movement amount for returning the viewpoint position corresponding to the display image to the viewpoint position corresponding to the recommended viewpoint information; and a viewpoint position calculation unit configured to calculate a viewpoint position of a next frame to be output after a frame of the display image output to the display unit, based on the operation movement amount and the return movement amount, wherein the output unit outputs the part of the image to the display unit, based on the display range determined from the viewpoint position of the next frame, wherein the return movement amount calculation unit calculates the return movement amount, based on the difference in the angle of the line-of-sight direction from the viewpoint position corresponding to the recommended viewpoint information to the viewpoint position corresponding to the display image, and wherein the output unit and the transition control unit are each implemented via at least one processor. ### Claim 2 (depends on 1) The image processing apparatus according to claim 1, wherein the return movement amount calculation unit sets a higher speed for returning to the viewpoint position corresponding to the recommended viewpoint information as the difference in the angle of the line-of-sight direction becomes larger. ### Claim 3 (depends on 2) The image processing apparatus according to claim 2, wherein the return movement amount calculation unit linearly increases the speed for returning to the viewpoint position corresponding to the recommended viewpoint information as the difference in the angle of the line-of-sight direction becomes larger. ### Claim 4 (depends on 2) The image processing apparatus according to claim 2, wherein the return movement amount calculation unit monotonically increases the speed for returning to the viewpoint position corresponding to the recommended viewpoint information, and lowers a rate of the monotonic increase, as the difference in the angle of the line-of-sight direction becomes larger. ### Claim 5 (depends on 1) The image processing apparatus according to claim 1, wherein the viewpoint position calculation unit calculates the viewpoint position of the next frame based on the return movement amount when the operation movement amount is equal to or smaller than a predetermined threshold, and calculates the viewpoint position of the next frame based on the operation movement amount and the return movement amount when the operation movement amount is greater than the predetermined threshold. ### Claim 6 (depends on 5) The image processing apparatus according to claim 5, wherein a value of the threshold becomes greater as a difference in an angle of a line-of-sight direction from the viewpoint position corresponding to the recommended viewpoint information to the viewpoint position corresponding to the display image becomes smaller. ### Claim 7 (depends on 1) The image processing apparatus according to claim 1, wherein the transition control unit includes: an operation movement amount calculation unit configured to calculate an operation movement amount for moving the viewpoint position corresponding to the display image in accordance with a viewpoint operation based on viewpoint operation information; a correction unit configured to correct the operation movement amount, based on a difference in an angle of a line-of-sight direction from the viewpoint position corresponding to the recommended viewpoint information to the viewpoint position corresponding to the display image; and a viewpoint position calculation unit configured to calculate a viewpoint position of a next frame to be output after a frame of the display image output to the display unit, based on the post-correction operation movement amount, and wherein the output unit outputs the part of the image to the display unit, based on the display range determined from the viewpoint position of the next frame. ### Claim 8 (depends on 7) The image processing apparatus according to claim 7, wherein the correction unit makes a rate of a monotonic increase in the post-correction operation movement amount lower than a rate of a monotonic increase in the pre-correction operation movement amount, as the difference in the angle of the line-of-sight direction becomes larger. ### Claim 9 (depends on 7) The image processing apparatus according to claim 7, wherein the correction unit linearly increases the pre-correction operation movement amount and the post-correction operation movement amount at the same gradient until the difference in the angle of the line-of-sight direction increases to a predetermined angle, and, when the difference in the angle of the line-of-sight direction becomes equal to or larger than the predetermined angle, the correction unit linearly increases the post-correction operation movement amount at a gentler gradient than the gradient at which the pre-correction operation movement amount linearly increases. ### Claim 10 (depends on 1) The image processing apparatus according to claim 1, further comprising: a score calculation unit configured to calculate a score for each viewpoint position included in the recommended viewpoint information; and a recommended viewpoint selection unit configured to select one of viewpoint positions included in the recommended viewpoint information, based on the score, wherein the return movement amount calculation unit calculates the return movement amount for returning the viewpoint position corresponding to the display image to the viewpoint position selected from among the viewpoint positions included in the recommended viewpoint information. ### Claim 11 (depends on 10) The image processing apparatus according to claim 10, wherein the score calculation unit calculates the score, based on a distance between the viewpoint position included in the recommended viewpoint information and the viewpoint position corresponding to the display image. ### Claim 12 (depends on 11) The image processing apparatus according to claim 11, wherein the score calculation unit decreases a value of the score as the distance becomes longer. ### Claim 13 (depends on 11) The image processing apparatus according to claim 11, wherein the score calculation unit calculates the score by allocating a predetermined weight to the distance. ### Claim 14 (depends on 13) The image processing apparatus according to claim 13, wherein the score calculation unit increases the value of the score as the weight becomes larger, or decreases the value of the score as the weight becomes smaller. ### Claim 15 (depends on 14) The image processing apparatus according to claim 14, wherein a larger value is assigned to the weight at a recommended viewpoint having a larger number of viewers. ### Claim 16 (depends on 1) The image processing apparatus according to claim 1, wherein the return movement amount calculation unit calculates the return movement amount when the operation movement amount is equal to or smaller than a predetermined threshold. ### Claim 17 An image processing method, implemented by a computer, the method comprising: outputting part of an image including recommended viewpoint information as a display image to a display unit; and causing a display range of the image to shift, based on a positional relationship between a viewpoint position corresponding to the display image output to the display unit and a viewpoint position corresponding to the recommended viewpoint information, wherein the display range is caused to shift at a speed determined based on a difference in an angle of a line-of-sight direction from the viewpoint position corresponding to the recommended viewpoint information to the viewpoint position corresponding to the display image, wherein the part of the image is output to the display unit based on the display range that has been made to shift, wherein causing the display range of the image to shift includes: calculating an operation movement amount for moving the viewpoint position corresponding to the display image in accordance with a viewpoint operation based on viewpoint operation information; calculating a return movement amount for returning the viewpoint position corresponding to the display image to the viewpoint position corresponding to the recommended viewpoint information; and calculating a viewpoint position of a next frame to be output after a frame of the display image output to the display unit, based on the operation movement amount and the return movement amount, wherein the part of the image is output to the display unit, based on the display range determined from the viewpoint position of the next frame, and wherein the return movement amount is calculated based on the difference in the angle of the line-of-sight direction from the viewpoint position corresponding to the recommended viewpoint information to the viewpoint position corresponding to the display image. ### Claim 18 A non-transitory computer-readable storage medium having embodied thereon a program, which when executed by a computer causes the computer to perform a method, the method comprising: outputting part of an image including recommended viewpoint information as a display image to a display unit; and causing a display range of the image to shift, based on a positional relationship between a viewpoint position corresponding to the display image output to the display unit and a viewpoint position corresponding to the recommended viewpoint information, wherein the display range is caused to shift at a speed determined based on a difference in an angle of a line-of-sight direction from the viewpoint position corresponding to the recommended viewpoint information to the viewpoint position corresponding to the display image, wherein the part of the image is output to the display unit based on the display range that has been made to shift, wherein causing the display range of the image to shift includes: calculating an operation movement amount for moving the viewpoint position corresponding to the display image in accordance with a viewpoint operation based on viewpoint operation information; calculating a return movement amount for returning the viewpoint position corresponding to the display image to the viewpoint position corresponding to the recommended viewpoint information; and calculating a viewpoint position of a next frame to be output after a frame of the display image output to the display unit, based on the operation movement amount and the return movement amount, wherein the part of the image is output to the display unit, based on the display range determined from the viewpoint position of the next frame, and wherein the return movement amount is calculated based on the difference in the angle of the line-of-sight direction from the viewpoint position corresponding to the recommended viewpoint information to the viewpoint position corresponding to the display image.