# US 10,121,254 B2 · Methods and systems of detecting object boundaries > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,121,254 B2 - Document ID: US10121254B2 - Status: Granted - Kind: B2 - Application number: 14/014,077 - Filed: 2013-08-29 - Published: 2015-03-05 - Granted: 2018-11-06 - Earliest claimed: 2013-08-29 - Assignees: Disney Enterprises, Inc. - Applicants: Disney Enterprises, Inc. - Inventors: Oliver Wang; Aljosa Smolic - Web page: https://patentgrep.com/patent/US10121254B2 ## CPC - G06T 7/12: Edge-based segmentation - G06T 7/174: involving the use of two or more images - G06T 7/215: Motion-based segmentation - G06T 2207/10016: Video; Image sequence - G06T 2207/20021: Dividing image into blocks, subimages or windows - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Methods and systems described herein detect object boundaries of videos. A window around the pixel may be followed in adjacent image frames of the image frame to determine object boundaries. Inconsistencies in image patches over a temporal window are detected, and each pixel of the image frame of a video is assigned an object boundary probability. The pixel may belong to a texture edge if the window content does not change throughout the adjacent image frames, or the pixel may belong to an object boundary if the window content changes. A probability value indicating the likelihood of the pixel belonging to an object boundary is determined based on the window content change and is assigned to the corresponding pixel. ## Claims ### Claim 1 A computer-implemented method of detecting object boundaries in a video, comprising: identifying a pixel of an image frame of a video; defining a patch for the pixel; identifying a set of neighboring image frames for the image frame, the set of neighboring image frames comprising an image frame that comes before and an image frame that comes after the image frame in the video; detecting a set of patch matches to the patch from the set of neighboring image frames by searching a plurality of patch transformations of the patch in each image frame of the set of neighboring image frames, wherein each patch match of the set of patch matches is detected by comparing pixels of the patch with pixels of a patch that is being matched, wherein the set of patch matches comprises a patch match for each image frame of the set of neighboring image frames; and calculating an object boundary probability value for the patch corresponding to the pixel by applying a difference metric to the patch and the set of patch matches; wherein the difference metric distinguishes patches corresponding to texture edges from patches corresponding to object boundaries, wherein the difference metric is a function of the set of patch matches, and wherein the difference metric is based on color and gradient information of the patch and the set of patch matches. ### Claim 2 (depends on 1) The method of claim 1, further comprising: calculating a set of object boundary probability values for a set of patches, each patch of the set of patches corresponding to a pixel of all the pixels of the image frame; and determining a probability map for the image frame based on the set of object boundary probability values. ### Claim 3 (depends on 2) The method of claim 2, further comprising refining the probability map by using a confidence map to generate a refined probability map for the image frame. ### Claim 4 (depends on 1) The method of claim 1, wherein the patch is a region of the image frame. ### Claim 5 (depends on 1) The method of claim 1, wherein the patch is a rotated and scaled square. ### Claim 6 (depends on 1) The method of claim 1, wherein the set of neighboring image frames are identified based on a temporal window width. ### Claim 7 (depends on 1) The method of claim 1, wherein the plurality of patch transformations comprise a shift, a relative scale, and a rotation. ### Claim 8 (depends on 1) The method of claim 1, wherein a difference between each patch of the set of patch matches and the patch is less than a predetermined value. ### Claim 9 (depends on 1) The method of claim 1, further comprising warping and resampling a patch of the set of patch matches in the image frame corresponding to the patch such that a direct pixel-to-pixel comparison is performed. ### Claim 10 (depends on 1) The method of claim 1, wherein the patch is configured to have a fixed scale and a fixed rotation. ### Claim 11 (depends on 1) The method of claim 1, wherein the set of neighboring image frames comprise a plurality of image frames that come before and a plurality of image frames that come after the image frame in the video. ### Claim 12 (depends on 1) The method of claim 1, wherein calculating the object boundary probability value for the patch corresponding to the pixel comprises: calculating a pixel-wise absolute color difference between the patch and each patch match of the set of patch matches; and storing each difference in a two-dimensional error patch. ### Claim 13 (depends on 1) The method of claim 1, wherein detecting a set of patch matches comprises defining a vector that gives position, scale, and rotation offsets of the patch relative to a patch match. ### Claim 14 A system of detecting object boundaries in a video, comprising: a non-transitory memory storing a set of instructions; and a processor coupled to the non-transitory memory, wherein the set of instructions are configured to cause the processor to perform: identifying a pixel of an image frame of a video; defining a patch for the pixel; identifying a set of neighboring image frames for the image frame, the set of neighboring image frames comprising an image frame that comes before and an image frame that comes after the image frame in the video; detecting a set of patch matches to the patch from the set of neighboring image frames by searching a plurality of patch transformations of the patch in each image frame of the set of neighboring image frames, wherein each patch match of the set of patch matches is detected by comparing pixels of the patch with pixels of a patch that is being matched, wherein the set of patch matches comprises a patch match for each image frame of the set of neighboring image frames; and calculating an object boundary probability value for the patch corresponding to the pixel by applying a difference metric to the patch and the set of patch matches; wherein the difference metric distinguishes patches corresponding to texture edges from patches corresponding to object boundaries, wherein the difference metric is a function of the set of patch matches, and wherein the difference metric is based on color and gradient information of the patch and the set of patch matches. ### Claim 15 (depends on 14) The system of claim 14, wherein the set of instructions are further configured to cause the processor to perform: calculating a set of object boundary probability values for a set of patches, each patch of the set of patches corresponding to a pixel of all the pixels of the image frame; and determining a probability map for the image frame based on the set of object boundary probability values. ### Claim 16 (depends on 15) The system of claim 15, wherein the set of instructions are further configured to cause the processor to perform using a confidence map to generate a refined probability map for the image frame. ### Claim 17 (depends on 14) The system of claim 14, wherein the set of neighboring image frames are identified based on a temporal window width. ### Claim 18 (depends on 14) The system of claim 14, wherein the plurality of patch transformations comprise a shift, a relative scale, and a rotation. ### Claim 19 (depends on 14) The system of claim 14, wherein the set of instructions are further configured to cause the processor to perform warping and resampling a patch of the set of patch matches in the image frame corresponding to the patch such that a direct pixel-to-pixel comparison is performed. ### Claim 20 (depends on 14) The system of claim 14, wherein the set of neighboring image frames comprise a plurality of image frames that come before and a plurality of image frames that come after the image frame in the video.