# US 10,873,697 B1 · Identifying regions of interest in captured video data objects by detecting movement within higher resolution frames of the regions > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,873,697 B1 - Document ID: US10873697B1 - Status: Granted - Kind: B1 - Application number: 16/564,447 - Filed: 2019-09-09 - Granted: 2020-12-22 - Earliest claimed: 2019-03-29 - Assignees: Facebook, Inc. - Applicants: Facebook, Inc. - Inventors: Anshul Kumar Jain; Abhinav Garlapati; Eric W. Hwang - Web page: https://patentgrep.com/patent/US10873697B1 ## CPC - G06T 7/248: {involving reference images or patches} - G06F 18/254: {of classification results, e.g. of results related to same input data} - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G06V 10/809: {of classification results, e.g. where the classifiers operate on the same input data} - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - H04N 21/234318: {by decomposing into objects, e.g. MPEG-4 objects} - H04N 21/44008: {involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - H04N 23/60: Control of cameras or camera modules - H04N 23/61: based on recognised objects - G06T 2207/10016: Video; Image sequence - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Multiple users communicate over a network via client devices that include one or more cameras and a display to enable video messaging. At least one of the client devices modifies regions of video data captured by the client device's camera to more prominently identify the people within the video data. To identify a person, the client device disambiguates between actual people and static objects that may appear like people. The client device uses pose models to identify bounding boxes and applies a motion model to determine if a bounding box may include a person based on an amount of movement within the bounding box. If a threshold amount of movement is detected in a bounding box, the client device obtains a higher resolution portion of the scene including the bounding box and classifies whether the bounding box contains a person based on movement within the higher resolution video. ## Claims ### Claim 1 A method comprising: capturing video data of a local area within a field of view of an image capture device included in a client device, the captured video data comprising a plurality of frames; identifying a background portion and one or more foreground portions of frames of the captured video data, the foreground portions including at least a threshold amount of movement in consecutive frames of the captured video data; identifying one or more regions within the captured video data including an object; identifying a candidate region from the identified one or more regions based on amounts of each of the identified one or more regions identified as foreground portions of frames of the captured video data; capturing higher resolution frames of the candidate region via the image capture device included in the client device, while maintaining capture of the video data of the local area; determining an amount of movement within the candidate region between consecutive frames of the higher resolution frames of the candidate region; and identifying the candidate region includes a moving object in response to the amount of movement within the candidate region between consecutive frames of the higher resolution frames of the candidate region equaling or exceeding a different threshold amount of movement. ### Claim 2 (depends on 1) The method of claim 1, wherein identifying the candidate region from the identified one or more regions based on amounts of each of the identified one or more regions identified as foreground portions of frames of the captured video data comprises: identifying the candidate region as an identified region having at least a threshold ratio of a number of pixels in the identified region as foreground portions of the captured video data to a total number of pixels in the identified region. ### Claim 3 (depends on 1) The method of claim 1, wherein identifying the candidate region from the identified one or more regions based on amounts of each of the identified one or more regions identified as foreground portions of frames of the captured video data comprises: determining a score for each of the identified one or more regions, a score for an identified region based on a ratio of a number of pixels in an identified region as foreground portions of the captured video data to a total number of pixels in the identified region; and identifying the candidate region as an identified region having a score greater than a threshold value and less than a maximum value. ### Claim 4 (depends on 3) The method of claim 3, wherein determining the score for each of the identified one or more regions comprises: adjusting the score for the identified region based on an amount of time for a model applied by the client device to classify the identified region as the background portion of the captured video data, so the score for the identified region is increased in response to a smaller amount of time for the model applied by the client device to classify the identified region as the background portion of the captured video data. ### Claim 5 (depends on 3) The method of claim 3, wherein determining the score for each of the identified one or more regions comprises: increasing the score for the identified region in response to identifying a pose or a face in the identified region. ### Claim 6 (depends on 1) The method of claim 1, wherein identifying the candidate region from the identified one or more regions based on amounts of each of the identified one or more regions identified as foreground portions of frames of the captured video data comprises: determining a score for each of the identified one or more regions, a score for an identified region based on a ratio of a number of pixels in an identified region as foreground portions of the captured video data to a total number of pixels in the identified region; ranking the identified regions based on their scores; and identifying the candidate region as an identified region having at least a threshold position in the ranking. ### Claim 7 (depends on 1) The method of claim 1, wherein capturing higher resolution frames of the candidate region via the image capture device included in the client device comprises: identifying coordinates within the frames of the captured video of the candidate region; and capturing higher resolution frames of the identified coordinates of the candidate region through a separate channel from the image capture device than a channel of the image capture device capturing the video data of the local area. ### Claim 8 (depends on 1) The method of claim 1, wherein determining the amount of movement within the candidate region between consecutive frames of the higher resolution frames of the candidate region comprises: determining a number of pixels within the higher resolution frames of the candidate region that change during a specific period of time. ### Claim 9 (depends on 8) The method of claim 8, wherein identifying the candidate region includes the moving object in response to the amount of movement within the candidate region between consecutive frames of the higher resolution frames of the candidate region equaling or exceeding the different threshold amount of movement comprises: identifying the candidate region includes the moving object in response to the number of pixels within the higher resolution frames of the candidate region that change during the specific period of time equals or exceed a threshold value. ### Claim 10 (depends on 1) The method of claim 1, wherein identifying the candidate region includes the moving object in response to the amount of movement within the candidate region between consecutive frames of the higher resolution frames of the candidate region equaling or exceeding the different threshold amount of movement comprises storing a label indicating the candidate region includes the moving object in association with coordinates specifying a bounding box corresponding to the candidate region. ### Claim 11 (depends on 1) The method of claim 1, further comprising: identifying the candidate region does not include the moving object in response to the amount of movement within the candidate region between consecutive frames of the higher resolution frames of the candidate region being less than the different threshold amount of movement. ### Claim 12 A computer program product comprising a non-transitory computer-readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to: capture video data of a local area within a field of view of an image capture device included in a client device, the captured video data comprising a plurality of frames; identify a background portion and one or more foreground portions of frames of the captured video data; identify one or more regions within the captured video data including at least a threshold amount of movement in consecutive frames of the captured video data; identify a candidate region from the identified one or more regions based on amounts of each of the identified one or more regions identified as foreground portions of frames of the captured video data; capture higher resolution frames of the candidate region via the image capture device included in the client device, while maintaining capture of the video data of the local area; determine an amount of movement within the candidate region between consecutive frames of the higher resolution frames of the candidate region; and identify the candidate region includes a moving object in response to the amount of movement within the candidate region between consecutive frames of the higher resolution frames of the candidate region equaling or exceeding a different threshold amount of movement. ### Claim 13 (depends on 12) The computer program product of claim 12, wherein identify the candidate region from the identified one or more regions based on amounts of each of the identified one or more regions identified as foreground portions of frames of the captured video data comprises: identify the candidate region as an identified region having at least a threshold ratio of a number of pixels in the identified region as foreground portions of the captured video data to a total number of pixels in the identified region. ### Claim 14 (depends on 12) The computer program product of claim 12, wherein identify the candidate region from the identified one or more regions based on amounts of each of the identified one or more regions identified as foreground portions of frames of the captured video data comprises: determine a score for each of the identified one or more regions, a score for an identified region based on a ratio of a number of pixels in an identified region as foreground portions of the captured video data to a total number of pixels in the identified region; and identify the candidate region as an identified region having a score greater than a threshold value and less than a maximum value. ### Claim 15 (depends on 14) The computer program product of claim 14, wherein determining the score for each of the identified one or more regions comprises: adjust the score for the identified region based on an amount of time for a model applied by the client device to classify the identified region as the background portion of the captured video data, so the score for the identified region is increased in response to a smaller amount of time for the model applied by the client device to classify the identified region as the background portion of the captured video data. ### Claim 16 (depends on 14) The computer program product of claim 14, wherein determine the score for each of the identified one or more regions comprises: increase the score for the identified region in response to identifying a pose or a face in the identified region. ### Claim 17 (depends on 12) The computer program product of claim 12, wherein identify the candidate region from the identified one or more regions based on amounts of each of the identified one or more regions identified as foreground portions of frames of the captured video data comprises: determine a score for each of the identified one or more regions, a score for an identified region based on a ratio of a number of pixels in an identified region as foreground portions of the captured video data to a total number of pixels in the identified region; rank the identified regions based on their scores; and identify the candidate region as an identified region having at least a threshold position in the ranking. ### Claim 18 (depends on 12) The computer program product of claim 12, wherein capture higher resolution frames of the candidate region via the image capture device included in the client device comprises: identify coordinates within the frames of the captured video of the candidate region; and capture higher resolution frames of the identified coordinates of the candidate region through a separate channel from the image capture device than a channel of the image capture device capturing the video data of the local area. ### Claim 19 (depends on 12) The computer program product of claim 12, wherein determine the amount of movement within the candidate region between consecutive frames of the higher resolution frames of the candidate region comprises: determine a number of pixels within the higher resolution frames of the candidate region that change during a specific period of time. ### Claim 20 (depends on 12) The computer program product of claim 12, wherein identify the candidate region includes the moving object in response to the amount of movement within the candidate region between consecutive frames of the higher resolution frames of the candidate region equaling or exceeding the different threshold amount of movement comprises: identify the candidate region includes the moving object in response to the number of pixels within the higher resolution frames of the candidate region that change during the specific period of time equals or exceed a threshold value. ### Claim 21 (depends on 12) The computer program product of claim 12, wherein identify the candidate region includes the moving object in response to the amount of movement within the candidate region between consecutive frames of the higher resolution frames of the candidate region equaling or exceeding the different threshold amount of movement comprises store a label indicating the candidate region includes the moving object in association with coordinates specifying a bounding box corresponding to the candidate region.