# US 10,121,219 B2 · Information processing apparatus, method and computer-readable medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,121,219 B2 - Document ID: US10121219B2 - Status: Granted - Kind: B2 - Application number: 15/603,027 - Filed: 2017-05-23 - Published: 2017-11-09 - Granted: 2018-11-06 - Earliest claimed: 2009-11-30 - Assignees: Sony Corporation - Applicants: Sony Corporation - Inventors: Masashi Kimoto; Shigeatsu Yoshioka - Web page: https://patentgrep.com/patent/US10121219B2 ## CPC - G06T 1/00: General purpose image data processing - G06F 16/2457: with adaptation to user needs - G06F 16/24573: {using data annotations, e.g. user-defined metadata} - G06F 16/33: Querying - G06F 16/338: Presentation of query results - G06F 16/58: Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually - G06F 16/583: using metadata automatically derived from the content - G06T 11/00: Two-dimensional [2D] image generation - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract In one example embodiment, an information processing apparatus causes a display device to display a first image from images associated with an observation target object. The images include the first image and a second image which corresponds to an annotation mark. In this embodiment, the information processing apparatus also causes the display device to display the annotation mark corresponding to the second image. In this embodiment, the displayed annotation mark overlaps the first image. ## Claims ### Claim 1 An information processing apparatus comprising: a processor; and a memory device operatively coupled to the processor, the memory device storing instructions that cause the processor, in cooperation with the memory device, to: obtain first position information of a first annotation mark at a first position on a first image associated with an observation target object; determine a second position of a second annotation mark based on the first position information; and cause a display device to alternatively display: (a) the first annotation mark on the first image, wherein the observation target object is associated with a plurality of images including the first image and a second image; and (b) the second annotation mark on the second image. ### Claim 2 (depends on 1) The information processing apparatus of claim 1, wherein the images are obtained by a microscope. ### Claim 3 (depends on 1) The information processing apparatus of claim 1, wherein the observation target object includes a section of biological tissue. ### Claim 4 (depends on 1) The information processing apparatus of claim 1, wherein the first image and the second image have a first resolution. ### Claim 5 (depends on 4) The information processing apparatus of claim 4, wherein the instructions, when executed by the processor, cause the processor to enable a user to change the first resolution of the first image and the second image to a second, different resolution. ### Claim 6 (depends on 1) The information processing apparatus of claim 1, wherein the images correspond to a plurality of different positions in an optical axis direction. ### Claim 7 (depends on 6) The information processing apparatus of claim 6, wherein the plurality of different positions in the optical axis direction are determined based on focusing positions. ### Claim 8 (depends on 1) The information processing apparatus of claim 1, wherein the first annotation mark includes a predetermined shaped frame. ### Claim 9 (depends on 1) The information processing apparatus of claim 1, wherein the instructions, when executed by the processor, cause the processor to display an annotation comment overlaying the displayed first image, the annotation comment being different from the displayed first annotation mark. ### Claim 10 (depends on 1) The information processing apparatus of claim 1, wherein: (a) an input device is operatively coupled to the processor; and (b) the instructions, when executed by the processor, cause the processor to operate with the input device to enable a user to input annotation information. ### Claim 11 (depends on 1) The information processing apparatus of claim 1, wherein: (a) the first annotation mark corresponds to a position on the second image; and (b) the instructions, when executed by the processor, cause the processor to display the first annotation mark at a position on the first image which corresponds to the position of the second image. ### Claim 12 (depends on 1) The information processing apparatus of claim 1, wherein the instructions, when executed by the processor, cause the processor to display a second annotation mark corresponding to the first image. ### Claim 13 (depends on 12) The information processing apparatus of claim 12, wherein: (a) the first image corresponds to a first layer which corresponds to a first position in an optical axis direction; and (b) the second image corresponds to a second layer which corresponds to a second position in the optical axis direction. ### Claim 14 (depends on 13) The information processing apparatus of claim 13, wherein: (a) the first annotation mark and the second annotation mark have at least one different color, brightness, chroma, alpha blend values, widths of lines, and styles of lines; and (b) the instructions, when executed by the processor, cause the processor to change at least one of the color, the chroma, the alpha blend values, the widths of lines, and the styles of lines based on said layers which correspond to said positions in the optical axis direction. ### Claim 15 (depends on 12) The information processing apparatus of claim 12, wherein the instructions, when executed by the processor, cause the processor to: (a) enable a user to select the first annotation mark; and (b) in response to the first annotation mark being selected, display the second image, wherein the first annotation mark overlaps the displayed second image. ### Claim 16 (depends on 12) The information processing apparatus of claim 12, wherein the instructions, when executed by the processor, cause the processor to: (a) enable a user to select one of the first annotation mark and the second annotation mark; (b) in response to the first annotation mark being selected, cancel the display of the second annotation mark; and (c) in response to the second annotation mark being selected, cancel the display of the first annotation mark. ### Claim 17 (depends on 15) The information processing apparatus of claim 15, wherein the instructions, when executed by the processor, cause the processor to, in response to the first annotation mark being selected, arrange and display the first annotation mark on a center portion of the displayed second image. ### Claim 18 (depends on 1) The information processing apparatus of claim 1, wherein the instructions, when executed by the processor, cause the processor to: (a) determine whether a user has requested a change from displaying the first image to displaying the second image; (b) in response to the change being requested, determine whether a mode is set; and (c) in response to the mode being set, display: (i) the second image; and (ii) the first annotation mark overlapping the second image. ### Claim 19 (depends on 18) The information processing apparatus of claim 18, wherein the instructions, when executed by the processor, cause the processor to, in response to the mode being set, display a third annotation mark overlapping the second image, the third annotation mark corresponding to a third image. ### Claim 20 (depends on 18) The information processing apparatus of claim 18, wherein the instructions, when executed by the processor, cause the processor to, in response to the mode not being set, display: (a) the second image; and (b) the first annotation mark overlapping the second image. ### Claim 21 (depends on 1) The information processing apparatus of claim 1, wherein the instructions, when executed by the processor, cause the processor to, for each displayed image associated with the observation target object, enable a user to register annotation information in a control table. ### Claim 22 (depends on 1) The information processing apparatus of claim 1, wherein the first image and the second image are images of the observation target object obtained at a different position in a thickness direction of the observation target object. ### Claim 23 (depends on 1) The information processing apparatus of claim 1, wherein a plurality of annotation marks are respectively positioned on the plurality of images, and at least three of the plurality of annotation marks are displayed simultaneously on at least one of the plurality of images. ### Claim 24 (depends on 1) The information processing apparatus of claim 1, wherein the first annotation mark is displayed by visually compounding at least one of the position information and a content information with the first image. ### Claim 25 (depends on 1) The information processing apparatus of claim 1, wherein the plurality of images is stored in a client computer or a server. ### Claim 26 (depends on 1) The information processing apparatus of claim 1, wherein the plurality of images includes at least one of medical images and pathological images. ### Claim 27 A method of operating an information processing apparatus including instructions, the method comprising: obtaining first position information of a first annotation mark at a first position on a first image associated with an observation target object; determining a second position of a second annotation mark based on the first position information; and causing a display device to alternatively display: (a) the first annotation mark on the first image, wherein the observation target object is associated with a plurality of images including the first image and a second image; and (b) the second annotation mark on the second image. ### Claim 28 (depends on 27) The method of claim 27, wherein the images are obtained by a microscope. ### Claim 29 (depends on 27) The method of claim 27, wherein the observation target object includes a section of biological tissue. ### Claim 30 (depends on 27) The method of claim 27, wherein the first image and the second image have a first resolution. ### Claim 31 (depends on 30) The method of claim 30, further comprising enabling a user to change the first resolution of the first image and the second image to a second, different resolution. ### Claim 32 (depends on 27) The method of claim 27, wherein the images correspond to a plurality of different positions in an optical axis direction. ### Claim 33 (depends on 32) The method of claim 32, wherein the plurality of different positions in the optical axis direction are determined based on focusing positions. ### Claim 34 (depends on 27) The method of claim 27, wherein the first annotation mark includes a predetermined shaped frame. ### Claim 35 (depends on 27) The method of claim 27, further comprising causing the processor to display an annotation comment overlaying the displayed first image, the annotation comment being different from the displayed first annotation mark. ### Claim 36 (depends on 27) The method of claim 27, wherein: (a) an input device is operatively coupled to the processor; and (b) further comprising causing the processor to operate with the input device to enable a user to input annotation information. ### Claim 37 (depends on 27) The method of claim 27, wherein: (a) the first annotation mark corresponds to a position on the second image; and (b) further comprising causing the processor to display the first annotation mark at a position on the first image which corresponds to the position of the second image. ### Claim 38 (depends on 27) The method of claim 27, further comprising causing the processor to display a second annotation mark corresponding to the first image. ### Claim 39 (depends on 38) The method of claim 38, wherein: (a) the first image corresponds to a first layer which corresponds to a first position in an optical axis direction; and (b) the second image corresponds to a second layer which corresponds to a second position in the optical axis direction. ### Claim 40 (depends on 39) The method of claim 39, wherein: (a) the first annotation mark and the second annotation mark have at least one different color, brightness, chroma, alpha blend values, widths of lines, and styles of lines; and (b) further comprising changing at least one of the color, the chroma, the alpha blend values, the widths of lines, and the styles of lines based on said layers which correspond to said positions in the optical axis direction. ### Claim 41 (depends on 38) The method of claim 38, further comprising: (a) enabling a user to select the first annotation mark; and (b) in response to the first annotation mark being selected, displaying the second image, wherein the first annotation mark overlaps the displayed second image. ### Claim 42 (depends on 38) The method of claim 38, further comprising: (a) enabling a user to select one of the first annotation mark and the second annotation mark; (b) in response to the first annotation mark being selected, canceling the display of the second annotation mark; and (c) in response to the second annotation mark being selected, canceling the display of the first annotation mark. ### Claim 43 (depends on 41) The method of claim 41, further comprising, in response to the first annotation mark being selected, arranging and displaying the first annotation mark on a center portion of the displayed second image. ### Claim 44 (depends on 27) The method of claim 27, further comprising: (a) determining whether a user has requested a change from displaying the first image to displaying the second image; (b) in response to the change being requested, determining whether a mode is set; and (c) in response to the mode being set, displaying: (i) the second image; and (ii) the first annotation mark overlapping the second image. ### Claim 45 (depends on 39) The method of claim 39, further comprising, in response to the mode being set, displaying a third annotation mark overlapping the second image, the third annotation mark corresponding to a third image. ### Claim 46 (depends on 39) The method of claim 39, further comprising, in response to the mode not being set, displaying: (a) the second image; and (b) the first annotation mark overlapping the second image. ### Claim 47 (depends on 27) The method of claim 27, further comprising, for each displayed image associated with the observation target object, enabling a user to register annotation information in a control table. ### Claim 48 (depends on 27) The method of claim 27, wherein the first image and the second image are images of the observation target object obtained at a different position in a thickness direction of the observation target object. ### Claim 49 (depends on 27) The method of claim 27, wherein a plurality of annotation marks are respectively positioned on the plurality of images, and at least three of the plurality of annotation marks are displayed simultaneously on at least one of the plurality of images. ### Claim 50 (depends on 27) The method of claim 27, wherein the first annotation mark is displayed by visually compounding at least one of the position information and a content information with the first image. ### Claim 51 (depends on 27) The method of claim 27, wherein the plurality of images is stored in a client computer or a server. ### Claim 52 (depends on 27) The method of claim 27, wherein the plurality of images includes at least one of medical images and pathological images. ### Claim 53 A non-transitory computer-readable medium storing instructions structured to cause an information processing apparatus to: obtain first position information of a first annotation mark at a first position on a first image associated with an observation target object; determine a second position of a second annotation mark based on the first position information; and cause a display device to alternatively display: (a) the first annotation mark on the first image, wherein the observation target object is associated with a plurality of images including the first image and a second image; and (b) the second annotation mark on the second image.