# US 2021/0174518 A1 · Tracking device, endoscope system, and tracking method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0174518 A1 - Document ID: US20210174518A1 - Status: Published application - Kind: A1 - Application number: 17/179,919 - Filed: 2021-02-19 - Published: 2021-06-10 - Earliest claimed: 2019-03-28 - Assignees: Olympus Corporation - Applicants: Olympus Corporation - Inventors: Makoto Ishikake - Web page: https://patentgrep.com/patent/US20210174518A1 ## CPC - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G06T 2207/10016: Video; Image sequence - G06T 2207/10068: Endoscopic image - G06T 2207/20081: Training; Learning - G06T 2207/20212: Image combination - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A tracking device includes a processor including hardware. The processor is configured to: set a start frame and an end frame in a video including multiple frames; perform forward tracking in frames subsequent to the start frame based on a region of a tracking target in the start frame; perform backward tracking in frames previous to the end frame based on a region of the tracking target in the end frame; and generate a combined mask image by combining a forward mask image based on the forward tracking with a backward mask image based on the backward tracking. ## Claims ### Claim 1 A tracking device comprising: a processor including hardware, the processor being configured to: set a start frame and an end frame in a video including multiple frames; perform forward tracking of tracking a tracking target in a forward direction in frames subsequent to the start frame based on a region of the tracking target in the start frame; perform backward tracking of tracking the tracking target in a backward direction in frames previous to the end frame based on a region of the tracking target in the end frame; and generate a combined mask image indicating a region of the tracking target in each of the frames in which the forward tracking and the backward tracking have been performed by combining a forward mask image based on the forward tracking with a backward mask image based on the backward tracking. ### Claim 2 (depends on 1) The tracking device as defined in claim 1, wherein the region of the tracking target in the start frame and the region of the tracking target in the end frame are set based on tagging data added to the tracking target. ### Claim 3 (depends on 1) The tracking device as defined in claim 1, wherein the processor sets weight information that causes the forward mask image to be weighted more than the backward mask image in a frame closer to the start frame, sets the weight information that causes the backward mask image to be weighted more than the forward mask image in a frame closer to the end frame, and generates the combined mask image by combining the forward mask image with the backward mask image based on the weight information. ### Claim 4 (depends on 1) The tracking device as defined in claim 1, wherein the processor applies the forward mask image as the combined mask image in a predetermined number of frames subsequent to the start frame, and applies the backward mask image as the combined mask image in a predetermined number of frames previous to the end frame. ### Claim 5 (depends on 1) The tracking device as defined in claim 1, wherein the processor performs a display process of superimposing the combined mask image on the video such that a non-overlapping portion where the forward mask image and the backward mask image do not overlap each other in the combined mask image has a higher transparency than an overlapping portion where the forward mask image and the backward mask image overlap each other in the combined mask image. ### Claim 6 (depends on 1) The tracking device as defined in claim 1, wherein the processor extracts start tracking points from a contour of the tracking target in the start frame and performs the forward tracking based on the extracted start tracking points, and extracts end tracking points from a contour of the tracking target in the end frame and performs the backward tracking based on the extracted end tracking points. ### Claim 7 (depends on 6) The tracking device as defined in claim 6, wherein the forward mask image is an image identifying a region inside a closed curve that connects a plurality of points as a result of tracking the start tracking points, and the backward mask image is an image identifying a region inside a closed curve that connects a plurality of points as a result of tracking the end tracking points. ### Claim 8 (depends on 1) The tracking device as defined in claim 1, wherein the processor combines the forward mask image with the backward mask image based on a weighting coefficient that is dependent on a frame position in the video and a pixel position in an image. ### Claim 9 (depends on 8) The tracking device as defined in claim 8, wherein the processor generates the combined mask image using a forward weighting coefficient wf and a backward weighting coefficient wb and based on the following expression: I(t,x)=wf(t,x)×If(t,x)+wb(t,x)×Ib(t,x) where x is the pixel position, t is the frame position, I is the combined mask image, If is the forward mask image, and Ib is the backward mask image. ### Claim 10 An endoscope system comprising: a memory storing a trained model; an endoscopic scope configured to capture a detection image; and a processor configured to receive the detection image as input and detect a position of a given subject in the detection image using the trained model, wherein the trained model is trained by machine learning based on training data in which a combined mask image is associated with an image of each of multiple frames included in a video, the combined mask image being generated by: setting a start frame and an end frame in the video including the multiple frames; performing forward tracking of tracking a tracking target in a forward direction in frames subsequent to the start frame based on a region of the tracking target in the start frame; performing backward tracking of tracking the tracking target in a backward direction in frames previous to the end frame based on a region of the tracking target in the end frame; and generating the combined mask image indicating a region of the tracking target in each of the frames in which the forward tracking and the backward tracking have been performed by combining a forward mask image based on the forward tracking with a backward mask image based on the backward tracking. ### Claim 11 A tracking method comprising: setting a start frame and an end frame in a video including multiple frames; performing forward tracking of tracking a tracking target in a forward direction in frames subsequent to the start frame based on a region of the tracking target in the start frame; performing backward tracking of tracking the tracking target in a backward direction in frames previous to the end frame based on a region of the tracking target in the end frame; and generating a combined mask image indicating a region of the tracking target in each of the frames in which the forward tracking and the backward tracking have been performed by combining a forward mask image based on the forward tracking with a backward mask image based on the backward tracking.