# US 11,908,188 B2 · Image analysis method, microscope video stream processing method, and related apparatus > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,908,188 B2 - Document ID: US11908188B2 - Status: Granted - Kind: B2 - Application number: 17/225,966 - Filed: 2021-04-08 - Published: 2021-07-22 - Granted: 2024-02-20 - Earliest claimed: 2019-03-08 - Assignees: Tencent Technology (Shenzhen) Company Limited - Applicants: Tencent Technology (Shenzhen) Company Limited - Inventors: Weijia Lu; Jianhua Yao; Xiao Han; Niyun Zhou - Web page: https://patentgrep.com/patent/US11908188B2 ## CPC - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G02B 21/365: {Control or image processing arrangements for digital or video microscopes (G02B21/361, G02B21/362 take precedence)} - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06T 7/0012: {Biomedical image inspection} - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - G06V 20/69: Microscopic objects, e.g. biological cells or cellular parts - G06T 2207/10016: Video; Image sequence - G06T 2207/10056: Microscopic image - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30096: Tumor; Lesion - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Embodiments of this application disclose methods, systems, and devices for medical image analysis and medical video stream processing. In one aspect, a method comprises extracting video frames from a medical image video stream that includes at least two pathological-section-based video frames. The method also comprises identifying single-frame image features in the video frames, mapping the single-frame image features into single-frame diagnostic classification results, and performing a classification mapping based on a video stream feature sequence that comprises the single-frame image features. The classification mapping comprises performing a convolution operation on the video stream feature sequence through a preset convolutional layer, obtaining a convolution result in accordance with the convolution operation, and performing fully connected mapping on the convolution result through a preset fully connected layer. In accordance with the classification mapping, a target diagnostic classification result corresponding to the medical image video stream is determined. ## Claims ### Claim 1 An image analysis method for medical diagnosis using a microscope, performed by a computer device that includes one or more processors and memory storing one or more programs for execution by the one or more processors, the method comprising: extracting video frames from a medical image video stream captured by the microscope that includes at least two pathological-section-based video frames; identifying single-frame image features in the video frames; mapping the single-frame image features into single-frame diagnostic classification results; performing classification mapping based on a video stream feature sequence that comprises the single-frame image features and the single-frame diagnostic classification results, the classification mapping comprising: performing a convolution operation on the video stream feature sequence through a preset convolutional layer; obtaining a convolution result in accordance with the convolution operation; and performing fully connected mapping on the convolution result through a preset fully connected layer; obtaining operation information of the microscope corresponding to the capturing of the medical image video stream; performing the classification mapping based on the operation information of the microscope and the video stream feature sequence; and determining a target diagnostic classification result corresponding to the medical image video stream and in accordance with the classification mapping. ### Claim 2 (depends on 1) The method according to claim 1, wherein the target diagnostic classification result comprises a classification result corresponding to the medical image video stream and key-frame information of the medical image video stream. ### Claim 3 (depends on 2) The method according to claim 2, wherein performing the classification mapping based on the video stream feature sequence further comprises: performing the classification mapping based on the single-frame image features and the single-frame diagnostic classification results to determine the classification result; and determining the key-frame information based on weight information corresponding to the video frames in the medical image video stream in the classification mapping. ### Claim 4 (depends on 1) The method according to claim 1, wherein extracting the video frames in the medical image video stream comprises: extracting pending video frames of the medical image video stream; and sharpening the pending video frames to obtain the video frames. ### Claim 5 (depends on 1) The method according to claim 1, wherein obtaining the operation information of the microscope corresponding to the capturing of the medical image video stream comprises: determining the operation information corresponding to the capturing of the medical image video stream according to an inter-frame difference between video frames of the medical image video stream. ### Claim 6 (depends on 1) The method according to claim 1, further comprising: after the classification mapping, displaying on the computing device the medical image video stream, position information of a currently played video frame corresponding to the medical image video stream, and the key-frame information at preset positions. ### Claim 7 (depends on 6) The method according to claim 6, wherein: the single-frame classification results comprise a classification result and heat maps corresponding to the video frames; the target classification result further comprises a heat map of a key frame corresponding to the medical image video stream; and the displaying further comprises: correspondingly displaying the heat map of the key frame when the key frame is displayed in the process of displaying the image video stream. ### Claim 8 (depends on 7) The method according to claim 7, wherein the correspondingly displaying the heat map of the key frame when the key frame is displayed comprises: smoothing the heat map of the key frame based on the operation information to obtain a target heat map of the key frame; and correspondingly displaying the target heat map of the key frame when the key frame is displayed in the process of displaying the image video stream. ### Claim 9 (depends on 1) The method according to claim 1, wherein performing the classification mapping further comprises: determining a patient sequence according to patient information corresponding to the medical image video stream; and performing the classification mapping based on the video stream feature sequence and the patient sequence to obtain the target classification result. ### Claim 10 (depends on 9) The method according to claim 9, wherein performing the classification mapping based on the video stream feature sequence and the patient sequence to obtain the target classification result further comprises: performing a convolution operation on the video stream feature sequence through the preset convolutional layer to obtain the convolution result; and performing fully connected mapping on the convolution result and the patient sequence through the preset fully connected layer to obtain the target classification result. ### Claim 11 A computing device for performing medical diagnosis using a microscope, comprising: one or more processors; and memory storing one or more programs for execution by the one or more processors, the one or more programs comprising instructions for: extracting video frames from a medical image video stream captured by the microscope that includes at least two pathological-section-based video frames; identifying single-frame image features in the video frames; mapping the single-frame image features into single-frame diagnostic classification results; performing classification mapping based on a video stream feature sequence that comprises the single-frame image features and the single-frame diagnostic classification results, the classification mapping comprising: performing a convolution operation on the video stream feature sequence through a preset convolutional layer; obtaining a convolution result in accordance with the convolution operation; and performing fully connected mapping on the convolution result through a preset fully connected layer; obtaining operation information of the microscope corresponding to the capturing of the medical image video stream; performing the classification mapping based on the operation information of the microscope and the video stream feature sequence; and determining a target diagnostic classification result corresponding to the medical image video stream and in accordance with the classification mapping. ### Claim 12 (depends on 11) The computing device according to claim 11, wherein the target diagnostic classification result comprises a classification result corresponding to the medical image video stream and key-frame information of the medical image video stream. ### Claim 13 (depends on 12) The computing device according to claim 12, wherein the instructions for performing the classification mapping based on the video stream feature sequence further comprises instructions for: performing the classification mapping based on the single-frame image features and the single-frame diagnostic classification results to determine the classification result; and determining the key-frame information based on weight information corresponding to the video frames in the medical image video stream in the classification mapping. ### Claim 14 (depends on 11) The computing device according to claim 11, wherein the instructions for extracting the video frames in the medical image video stream comprises instructions for: extracting pending video frames of the medical image video stream; and sharpening the pending video frames to obtain the video frames. ### Claim 15 A non-transitory computer-readable storage medium storing computer instructions that, when executed by one or more processors of a computing device for performing medical diagnosis using a microscope, cause the one or more processors to perform operations comprising: extracting video frames from a medical image video stream captured by the microscope that includes at least two pathological-section-based video frames; identifying single-frame image features in the video frames; mapping the single-frame image features into single-frame diagnostic classification results; performing classification mapping based on a video stream feature sequence that comprises the single-frame image features and the single-frame diagnostic classification results, the classification mapping comprising: performing a convolution operation on the video stream feature sequence through a preset convolutional layer; obtaining a convolution result in accordance with the convolution operation; and performing fully connected mapping on the convolution result through a preset fully connected layer; obtaining operation information of the microscope corresponding to the capturing of the medical image video stream; performing the classification mapping based on the operation information of the microscope and the video stream feature sequence; and determining a target diagnostic classification result corresponding to the medical image video stream and in accordance with the classification mapping. ### Claim 16 (depends on 15) The non-transitory computer-readable storage medium according to claim 15, wherein obtaining the operation information of the microscope corresponding to the capturing of the medical image video stream comprises: determining the operation information corresponding to the capturing of the medical image video stream according to an inter-frame difference between video frames of the medical image video stream. ### Claim 17 (depends on 15) The non-transitory computer-readable storage medium according to claim 15, further comprising: after the classification mapping, displaying on the computing device the medical image video stream, position information of a currently played video frame corresponding to the medical image video stream, and the key-frame information at preset positions.