# US 12,183,448 B2 · Medical image processing apparatus, trocar, medical observation system, image processing method, and computer readable recording medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,183,448 B2 - Document ID: US12183448B2 - Status: Granted - Kind: B2 - Application number: 17/183,373 - Filed: 2021-02-24 - Published: 2021-09-23 - Granted: 2024-12-31 - Earliest claimed: 2020-03-18 - Assignees: Sony Olympus Medical Solutions Inc. - Applicants: Sony Olympus Medical Solutions Inc. - Inventors: Kiyoshi Ichikawa - Web page: https://patentgrep.com/patent/US12183448B2 ## CPC - G16H 30/40: for processing medical images, e.g. editing - A61B 1/00006: {of control signals} - A61B 1/00045: {Display arrangement} - A61B 1/3132: {for laparoscopy} - A61B 17/34: Trocars; Puncturing needles - A61B 17/3421: {Cannulas} - A61B 90/37: {Surgical systems with images on a monitor during operation} - A61B 90/98: using electromagnetic means, e.g. transponders - G16H 40/63: for local operation - A61B 90/361: {Image-producing devices, e.g. surgical cameras} - A61B 2017/00119: {alarm; indicating an abnormal situation} - A61B 2017/347: {Locking means, e.g. for locking instrument in cannula} - A61B 2090/062: {penetration depth} - A61B 2090/064: {for measuring force, pressure or mechanical tension} - A61B 2090/067: {for measuring angles} - A61B 2090/0811: {for the position of a particular part of an instrument with respect to the rest of the instrument, e.g. position of the anvil of a stapling instrument} - A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image} - A61B 2562/0247: Pressure sensors - G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) ## Abstract A medical image processing apparatus includes a control unit configured to cause, based on a detection result of at least one trocar into which a medical instrument is inserted, the at least one trocar being configured to detect a first state and a second state, the first state and the second state indicating an insertion state of the medical instrument into a subject, a display device to display at least one of a first image generated by capturing an image of an observation object inside the subject and a second image of the observation object inside the subject, the second image being previously acquired. ## Claims ### Claim 1 A medical image processing apparatus comprising a circuit configured to output, based on a detection result of at least one trocar into which a medical instrument is inserted, the at least one trocar including a first detector configured to detect a first state and a second detector configured to detect a second state, the first state and the second state indicating an insertion state of the medical instrument into the at least one trocar and a subject, to a display, at least one of a first image generated by capturing a live image of an observation object inside the subject and a second image that is a simulation image of the observation object inside the subject, the second image being previously acquired, wherein, in response to the first state being detected, output at least the second image, and in response to the second state being detected, only output the first image of the first and second images. ### Claim 2 (depends on 1) The medical image processing apparatus according to claim 1, wherein the first state is a state where the medical instrument is located at a position inside the at least one trocar, and the second state is a state where at least a part of the medical instrument extends through the at least one trocar and exposed into the subject through the at least one trocar. ### Claim 3 (depends on 1) The medical image processing apparatus according to claim 1, wherein the circuit is configured to cause the display to display a medical instrument image corresponding to the medical instrument in a superimposed manner on the second image based on medical instrument information about the medical instrument detected by the at least one trocar. ### Claim 4 (depends on 3) The medical image processing apparatus according to claim 3, wherein the circuit is configured to control a display mode of the medical instrument image and the second image based on details of an operation of an operator on the medical instrument. ### Claim 5 (depends on 4) The medical image processing apparatus according to claim 4, wherein the circuit is configured to output to the display the second image with the observation object rotated in at least one of a horizontal direction and a vertical direction based on at least one of a rotation amount of the medical instrument configured to rotate about an insertion portion of the at least one trocar and an insertion amount of the medical instrument into the insertion portion of the at least one trocar, the rotation amount and the insertion amount being detected by the at least one trocar. ### Claim 6 (depends on 4) The medical image processing apparatus according to claim 4, wherein, in response to the at least one trocar switching from the second state to the first state, the circuit is configured to estimate a position of an organ included in the first image and a position of the organ included in the second image, and cause the display to display the second image with the position of the organ included in the second image aligned with the position of the organ included in the first image based on the estimate. ### Claim 7 (depends on 1) The medical image processing apparatus according to claim 1, wherein the circuit is configured to, when the first state is detected, cause the display to display the first image and the second image in parallel or to display the first image reduced in size and superimposed on the second image. ### Claim 8 (depends on 1) The medical image processing apparatus according to claim 1, wherein in response to the at least one trocar switching from the second state to the first state, the circuit is configured to cause the display to output a warning. ### Claim 9 (depends on 1) The medical image processing apparatus according to claim 1, wherein the at least one trocar comprises two trocars, and the circuit is configured to control the display to display the first image in a case where each of the two trocars detects the second state, and display at least the second image in a case where each of the two trocars detects the first state. ### Claim 10 (depends on 1) The medical image processing apparatus according to claim 1, wherein the circuit is configured to output to the display, based on a distance between the observation object and a distal end of a medical imaging device configured to capture an image of the observation object, distance information including an optimum distance between the observation object and the distal end in observation of fluorescence of the observation object. ### Claim 11 A medical observation system comprising: an image sensor configured to generate a first image by capturing a live image of an observation object of a subject; a memory configured to store a second image based on three-dimensional image data or a two-dimensional image data of the observation object of the subject, the second image being previously acquired; a display; at least one trocar into which a medical instrument is inserted, the at least one trocar including a first detector configured to detect a first state and a second detector configured to detect a second state, the first state and the second state indicating an insertion state of the medical instrument into the at least one trocar and the subject; and a control circuit configured to output at least one of the first image and the second image to the display based on a detection result of the at least one trocar, wherein, in response to the first state being detected, output at least the second image, and in response to the second state being detected, output only the first image of the first and second images to the display. ### Claim 12 An image processing method comprising determining a state of at least one trocar into which a medical instrument is inserted, the at least one trocar including a first detector configured to detect a first state and a second detector configured to detect a second state, the first state and the second state indicating an insertion state of the medical instrument into the at least one trocar and a subject, and controlling output to a display of at least one of a first image generated by capturing a live image of an observation object inside the subject and a second image of the observation object inside the subject, the second image being previously acquired, wherein in response to the first state being detected, outputting at least the second image to the display, and in response to the second state being detected, outputting only the first image of the first and second images to the display. ### Claim 13 A non-transitory computer-readable recording medium on which an executable program is recorded, the program causing a processor to execute; determining a state of at least one trocar into which a medical instrument is inserted, the at least one trocar including a first detector configured to detect a first state and a second detector configured to detect a second state, the first state and the second state indicating an insertion state of the medical instrument into the at least one trocar and a subject, and controlling output to a display of at least one of a first image generated by capturing a live image of an observation object inside the subject and a second image of the observation object inside the subject, the second image being previously acquired, wherein in response to the first state being detected, outputting at least the second image, and in response to the second state being detected, outputting only the first image of the first and second images to the display. ### Claim 14 (depends on 13) The non-transitory computer-readable recording medium according to claim 13, wherein the program causes the processor to control the display to display a medical instrument image corresponding to the medical instrument in a superimposed manner on the second image based on medical instrument information about the medical instrument detected by the at least one trocar. ### Claim 15 (depends on 14) The non-transitory computer-readable recording medium according to claim 14, wherein the program causes the processor to control a display mode of the medical instrument image and the second image based on details of an operation of an operator on the medical instrument. ### Claim 16 (depends on 15) The non-transitory computer-readable recording medium according to claim 15, wherein the program causes the processor to, in response to the at least one trocar switching from the second state to the first state, estimate a position of an organ included in the first image and a position of the organ included in the second image, and control the display to display the second image with the position of the organ included in the second image aligned with the position of the organ included in the first image based on the estimate. ### Claim 17 (depends on 15) The non-transitory computer-readable recording medium according to claim 15, wherein the program causes the processor to output to the display the second image with the observation object rotated in at least one of a horizontal direction and a vertical direction based on at least one of a rotation amount of the medical instrument configured to rotate about an insertion portion of the at least one trocar and an insertion amount of the medical instrument into the insertion portion of the at least one trocar, the rotation amount and the insertion amount being detected by the at least one trocar. ### Claim 18 (depends on 13) The non-transitory computer-readable recording medium according to claim 13, wherein the program causes the processor to control, when the first state is detected, the display to display the first image and the second image in parallel or to display the first image reduced in size and superimposed on the second image. ### Claim 19 (depends on 13) The non-transitory computer-readable recording medium according to claim 13, wherein the program causes the processor to, in response to the at least one trocar switching from the second state to the first state, cause the display to output a warning. ### Claim 20 (depends on 13) The non-transitory computer-readable recording medium according to claim 13, wherein the at least one trocar includes two trocars, the program causes the processor to, control the display to display the first image in a case where each of the two trocars detects the second state, and display at least the second image in a case where each of the two trocars detects the first state.