# US 12,183,085 B2 · Apparatus, image display method, image display system, and storage medium storing program > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,183,085 B2 - Document ID: US12183085B2 - Status: Granted - Kind: B2 - Application number: 17/501,830 - Filed: 2021-10-14 - Published: 2022-04-21 - Granted: 2024-12-31 - Earliest claimed: 2020-10-21 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Tatsuya Yamamoto; David Johnston; Nicholas Fulton; Julie Kowald - Web page: https://patentgrep.com/patent/US12183085B2 ## CPC - B60R 1/25: to the sides of the vehicle - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - B60R 2300/105: using multiple cameras - B60R 2300/30: characterised by the type of image processing - B60R 2300/306: using a re-scaling of images - B60R 2300/70: characterised by an event-triggered choice to display a specific image among a selection of captured images - B60Y 2200/22: Agricultural vehicles - G06V 20/68: Food, e.g. fruit or vegetables - B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - B60Y: INDEXING SCHEME RELATING TO ASPECTS CROSS-CUTTING VEHICLE TECHNOLOGY ## Abstract To appropriately determine the display order of a plurality of images captured by at least one image capturing apparatus installed on a movable body, based on an imaging direction of the image capturing apparatus, an apparatus identifies the imaging direction of the image capturing apparatus in capturing each of the plurality of images, and determines the display order of the plurality of images based on a result of the identification. ## Claims ### Claim 1 An apparatus comprising: one or more processors, wherein the one or more processors function as: an acquisition unit configured to acquire a plurality of images captured by at least one image capturing apparatus installed on a movable body; an identification unit configured to identify an imaging direction of the at least one image capturing apparatus in capturing each of the plurality of images, with respect to a traveling direction of the movable body; a determination unit configured to determine a display order of the plurality of images based on a result of the identification by the identification unit; and a display control unit configured to cause a display unit to display the plurality of images based on the determined display order, wherein each of the plurality of images is a captured image of any one of a plurality of imaging targets including a first imaging target and a second imaging target, wherein the determination unit determines the display order of the plurality of images so that the images of the same imaging target are successively displayed, and wherein, in a case where the display control unit changes the displayed image from the captured image of the first imaging target to the captured image of the second imaging target based on the determined display order, the display control unit causes the display unit to display information indicating change of the imaging target. ### Claim 2 (depends on 1) The apparatus according to claim 1, wherein the determination unit determines the display order according to a different standard depending on whether the imaging direction is identified as right or left with respect to the traveling direction of the movable body by the identification unit. ### Claim 3 (depends on 1) The apparatus according to claim 1, wherein the determination unit determines the display order by sorting, in reverse chronological order of image capturing, the images for which the imaging direction is identified as right with respect to the traveling direction of the movable body by the identification unit, and by sorting, in chronological order of image capturing, the images for which the imaging direction is identified as left with respect to the traveling direction of the movable body by the identification unit. ### Claim 4 (depends on 1) The apparatus according to claim 1, wherein each of the plurality of imaging targets is a hedge of fruit trees planted in a row. ### Claim 5 (depends on 4) The apparatus according to claim 4, wherein the fruit trees are grape trees. ### Claim 6 (depends on 1) The apparatus according to claim 1, wherein the one or more processors further function as a detection unit configured to detect an instruction that indicates an image change direction, and wherein, in a case where the instruction is detected by the detection unit while a first image among the plurality of images is displayed, the display control unit changes the displayed image to the image immediately preceding or following the first image in the determined display order, based on the direction indicated by the instruction. ### Claim 7 (depends on 6) The apparatus according to claim 6, wherein the detection unit detects an input indicating a rightward direction or an input indicating a leftward direction, and wherein, in a case where the input indicating the rightward direction is detected by the detection unit while the first image is displayed, the display control unit changes the displayed image to the image immediately following the first image in the determined display order, and wherein, in a case where the input indicating the leftward direction is detected by the detection unit while the first image is displayed, the display control unit changes the displayed image to the image immediately preceding the first image in the determined display order. ### Claim 8 (depends on 7) The apparatus according to claim 7, wherein the at least one image capturing apparatus comprises an image capturing apparatus having captured the first image, and wherein the determination unit determines the display order so that, in a case where the input indicating the rightward direction is detected by the detection unit while the first image is displayed, the displayed image is changed to an image, among the plurality of images, that contains a subject existing to a right of a subject of the first image with respect to the imaging direction of the image capturing apparatus having captured the first image, and in a case where the input indicating the leftward direction is detected by the detection unit while the first image is displayed, the displayed image is changed to an image, among the plurality of images, that contains a subject existing to a left of the subject of the first image with respect to the imaging direction of the image capturing apparatus having captured the first image. ### Claim 9 (depends on 1) The apparatus according to claim 1, wherein, in a case where the display control unit changes the displayed image based on the determined display order, the display control unit causes the display unit to display information indicating that a distance between an imaging position of the image displayed before the change and an imaging position of the image displayed after the change is large. ### Claim 10 An image display system including a movable body, at least one image capturing apparatus installed on the movable body, and an image display apparatus configured to display a plurality of images captured by the at least one image capturing apparatus, the image display system comprising: one or more processors, wherein the one or more processors performs: capturing the plurality of images using the at least one image capturing apparatus simultaneously with movement of the movable body; identifying an imaging direction of the at least one image capturing apparatus in capturing each of the plurality of images, with respect to a traveling direction of the movable body; determining a display order of the plurality of images based on a result of the identifying; and causing a display unit to display the plurality of images based on the determined display order, wherein each of the plurality of images is a captured image of any one of a plurality of imaging targets including a first imaging target and a second imaging target, wherein the determined display order of the plurality of images of the same imaging target are successively displayed, and wherein, in a case where the displayed image from the captured image of the first imaging target is changed to the captured image of the second imaging target based on the determined display order, further causing the display unit to display information indicating change of the imaging target. ### Claim 11 An image display method comprising: acquiring a plurality of images captured by at least one image capturing apparatus installed on a movable body; identifying an imaging direction of the at least one image capturing apparatus in capturing each of the plurality of images, with respect to a traveling direction of the movable body; determining a display order of the plurality of images based on a result of the identifying; and causing a display unit to display the plurality of images based on the determined display order, wherein each of the plurality of images is a captured image of any one of a plurality of imaging targets including a first imaging target and a second imaging target, wherein the determined display order of the plurality of images of the same imaging target are successively displayed, and wherein, in a case where the displayed image from the captured image of the first imaging target is changed to the captured image of the second imaging target based on the determined display order, further causing the display unit to display information indicating change of the imaging target. ### Claim 12 A non-transitory computer-readable storage medium storing a program for causing a computer to execute an image display method comprising: acquiring a plurality of images captured by at least one image capturing apparatus installed on a movable body; identifying an imaging direction of the at least one image capturing apparatus in capturing each of the plurality of images, with respect to a traveling direction of the movable body; determining a display order of the plurality of images based on a result of the identifying; and causing a display unit to display the plurality of images based on the determined display order, wherein each of the plurality of images is a captured image of any one of a plurality of imaging targets including a first imaging target and a second imaging target, wherein the determined display order of the plurality of images of the same imaging target are successively displayed, and wherein, in a case where the displayed image from the captured image of the first imaging target is changed to the captured image of the second imaging target based on the determined display order, further causing the display unit to display information indicating change of the imaging target. ### Claim 13 An apparatus comprising: one or more processors, wherein the one or more processors function as: an acquisition unit configured to acquire a plurality of images captured by at least one image capturing apparatus installed on a movable body; an identification unit configured to identify an imaging direction of the at least one image capturing apparatus in capturing each of the plurality of images, with respect to a traveling direction of the movable body; a determination unit configured to determine a display order of the plurality of images based on a result of the identification by the identification unit; and a display control unit configured to cause a display unit to display the plurality of images based on the determined display order, wherein the one or more processors further function as a detection unit configured to detect an instruction that indicates an image change direction, and wherein, in a case where the instruction is detected by the detection unit while a first image among the plurality of images is displayed, the display control unit changes the displayed image to the image immediately preceding or following the first image in the determined display order, based on the direction indicated by the instruction. ### Claim 14 (depends on 13) The apparatus according to claim 13, wherein the determination unit determines the display order according to a different standard depending on whether the imaging direction is identified as right or left with respect to the traveling direction of the movable body by the identification unit. ### Claim 15 (depends on 13) The apparatus according to claim 13, wherein the determination unit determines the display order by sorting, in reverse chronological order of image capturing, the images for which the imaging direction is identified as right with respect to the traveling direction of the movable body by the identification unit, and by sorting, in chronological order of image capturing, the images for which the imaging direction is identified as left with respect to the traveling direction of the movable body by the identification unit. ### Claim 16 (depends on 13) The apparatus according to claim 13, wherein each of the plurality of images is a captured image of any one of a plurality of imaging targets including a first imaging target and a second imaging target, and wherein the determination unit determines the display order of the plurality of images so that the images of the same imaging target are successively displayed. ### Claim 17 (depends on 16) The apparatus according to claim 16, wherein each of the plurality of imaging targets is a hedge of fruit trees planted in a row. ### Claim 18 (depends on 17) The apparatus according to claim 17, wherein the fruit trees are grape trees. ### Claim 19 (depends on 13) The apparatus according to claim 13, wherein the detection unit detects an input indicating a rightward direction or an input indicating a leftward direction, and wherein, in a case where the input indicating the rightward direction is detected by the detection unit while the first image is displayed, the display control unit changes the displayed image to the image immediately following the first image in the determined display order, and wherein, in a case where the input indicating the leftward direction is detected by the detection unit while the first image is displayed, the display control unit changes the displayed image to the image immediately preceding the first image in the determined display order. ### Claim 20 (depends on 19) The apparatus according to claim 19, wherein the at least one image capturing apparatus comprises an image capturing apparatus having captured the first image, and wherein the determination unit determines the display order so that, in a case where the input indicating the rightward direction is detected by the detection unit while the first image is displayed, the displayed image is changed to an image, among the plurality of images, that contains a subject existing to a right of a subject of the first image with respect to the imaging direction of the image capturing apparatus having captured the first image, and in a case where the input indicating the leftward direction is detected by the detection unit while the first image is displayed, the displayed image is changed to an image, among the plurality of images, that contains a subject existing to a left of the subject of the first image with respect to the imaging direction of the image capturing apparatus having captured the first image. ### Claim 21 (depends on 13) The apparatus according to claim 13, wherein, in a case where the display control unit changes the displayed image based on the determined display order, the display control unit causes the display unit to display information indicating that a distance between an imaging position of the image displayed before the change and an imaging position of the image displayed after the change is large. ### Claim 22 An image display system including a movable body, at least one image capturing apparatus installed on the movable body, and an image display apparatus configured to display a plurality of images captured by the at least one image capturing apparatus, the image display system comprising: one or more processors, wherein the one or more processors performs: capturing the plurality of images using the at least one image capturing apparatus simultaneously with movement of the movable body; identifying an imaging direction of the at least one image capturing apparatus in capturing each of the plurality of images, with respect to a traveling direction of the movable body; determining a display order of the plurality of images based on a result of the identifying; causing a display unit to display the plurality of images based on the determined display order; and detecting an instruction that indicates an image change direction, wherein, in a case where the instruction is detected while a first image among the plurality of images is displayed, the displayed image is changed to the image immediately preceding or following the first image in the determined display order, based on the direction indicated by the instruction. ### Claim 23 An image display method comprising: acquiring a plurality of images captured by at least one image capturing apparatus installed on a movable body; identifying an imaging direction of the at least one image capturing apparatus in capturing each of the plurality of images, with respect to a traveling direction of the movable body; determining a display order of the plurality of images based on a result of the identifying; causing a display unit to display the plurality of images based on the determined display order; and detecting an instruction that indicates an image change direction, wherein, in a case where the instruction is detected while a first image among the plurality of images is displayed, the displayed image is changed to the image immediately preceding or following the first image in the determined display order, based on the direction indicated by the instruction. ### Claim 24 A non-transitory computer-readable storage medium storing a program for causing a computer to execute an image display method comprising: acquiring a plurality of images captured by at least one image capturing apparatus installed on a movable body; identifying an imaging direction of the at least one image capturing apparatus in capturing each of the plurality of images, with respect to a traveling direction of the movable body; determining a display order of the plurality of images based on a result of the identifying; causing a display unit to display the plurality of images based on the determined display order; and detecting an instruction that indicates an image change direction, wherein, in a case where the instruction is detected while a first image among the plurality of images is displayed, the displayed image is changed to the image immediately preceding or following the first image in the determined display order, based on the direction indicated by the instruction. ### Claim 25 An apparatus comprising: one or more processors, wherein the one or more processors function as: an acquisition unit configured to acquire a plurality of images captured by at least one image capturing apparatus installed on a movable body; an identification unit configured to identify an imaging direction of the at least one image capturing apparatus in capturing each of the plurality of images, with respect to a traveling direction of the movable body; a determination unit configured to determine a display order of the plurality of images based on a result of the identification by the identification unit; and a display control unit configured to cause a display unit to display the plurality of images based on the determined display order, wherein, in a case where the display control unit changes the displayed image based on the determined display order, the display control unit causes the display unit to display information indicating that a distance between an imaging position of the image displayed before the change and an imaging position of the image displayed after the change is large. ### Claim 26 (depends on 25) The apparatus according to claim 25, wherein the determination unit determines the display order according to a different standard depending on whether the imaging direction is identified as right or left with respect to the traveling direction of the movable body by the identification unit. ### Claim 27 (depends on 25) The apparatus according to claim 25, wherein the determination unit determines the display order by sorting, in reverse chronological order of image capturing, the images for which the imaging direction is identified as right with respect to the traveling direction of the movable body by the identification unit, and by sorting, in chronological order of image capturing, the images for which the imaging direction is identified as left with respect to the traveling direction of the movable body by the identification unit. ### Claim 28 (depends on 25) The apparatus according to claim 25, wherein each of the plurality of images is a captured image of any one of a plurality of imaging targets including a first imaging target and a second imaging target, and wherein the determination unit determines the display order of the plurality of images so that the images of the same imaging target are successively displayed. ### Claim 29 (depends on 28) The apparatus according to claim 28, wherein each of the plurality of imaging targets is a hedge of fruit trees planted in a row. ### Claim 30 (depends on 29) The apparatus according to claim 29, wherein the fruit trees are grape trees. ### Claim 31 (depends on 25) The apparatus according to claim 25, wherein the one or more processors further function as a detection unit configured to detect an instruction that indicates an image change direction, and wherein, in a case where the instruction is detected by the detection unit while a first image among the plurality of images is displayed, the display control unit changes the displayed image to the image immediately preceding or following the first image in the determined display order, based on the direction indicated by the instruction, wherein the detection unit detects an input indicating a rightward direction or an input indicating a leftward direction, and wherein, in a case where the input indicating the rightward direction is detected by the detection unit while the first image is displayed, the display control unit changes the displayed image to the image immediately following the first image in the determined display order, and wherein, in a case where the input indicating the leftward direction is detected by the detection unit while the first image is displayed, the display control unit changes the displayed image to the image immediately preceding the first image in the determined display order. ### Claim 32 (depends on 31) The apparatus according to claim 31, wherein the at least one image capturing apparatus comprises an image capturing apparatus having captured the first image, and wherein the determination unit determines the display order so that, in a case where the input indicating the rightward direction is detected by the detection unit while the first image is displayed, the displayed image is changed to an image, among the plurality of images, that contains a subject existing to a right of a subject of the first image with respect to the imaging direction of the image capturing apparatus having captured the first image, and in a case where the input indicating the leftward direction is detected by the detection unit while the first image is displayed, the displayed image is changed to an image, among the plurality of images, that contains a subject existing to a left of the subject of the first image with respect to the imaging direction of the image capturing apparatus having captured the first image. ### Claim 33 An image display system including a movable body, at least one image capturing apparatus installed on the movable body, and an image display apparatus configured to display a plurality of images captured by the at least one image capturing apparatus, the image display system comprising: one or more processors, wherein the one or more processors performs: capturing the plurality of images using the at least one image capturing apparatus simultaneously with movement of the movable body; identifying an imaging direction of the at least one image capturing apparatus in capturing each of the plurality of images, with respect to a traveling direction of the movable body; determining a display order of the plurality of images based on a result of the identifying; and causing a display unit to display the plurality of images based on the determined display order, wherein, in a case where the displayed image is changed based on the determined display order, further causing the display unit to display information indicating that a distance between an imaging position of the image displayed before the change and an imaging position of the image displayed after the change is large. ### Claim 34 An image display method comprising: acquiring a plurality of images captured by at least one image capturing apparatus installed on a movable body; identifying an imaging direction of the at least one image capturing apparatus in capturing each of the plurality of images, with respect to a traveling direction of the movable body; determining a display order of the plurality of images based on a result of the identifying; and causing a display unit to display the plurality of images based on the determined display order, wherein, in a case where the displayed image is changed based on the determined display order, further causing the display unit to display information indicating that a distance between an imaging position of the image displayed before the change and an imaging position of the image displayed after the change is large. ### Claim 35 A non-transitory computer-readable storage medium storing a program for causing a computer to execute an image display method comprising: acquiring a plurality of images captured by at least one image capturing apparatus installed on a movable body; identifying an imaging direction of the at least one image capturing apparatus in capturing each of the plurality of images, with respect to a traveling direction of the movable body; determining a display order of the plurality of images based on a result of the identifying; and causing a display unit to display the plurality of images based on the determined display order, wherein, in a case where the displayed image is changed based on the determined display order, further causing the display unit to display information indicating that a distance between an imaging position of the image displayed before the change and an imaging position of the image displayed after the change is large.