# US 2021/0241049 A1 · Object tracking assisted with hand or eye tracking > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0241049 A1 - Document ID: US20210241049A1 - Status: Published application - Kind: A1 - Application number: 17/234,522 - Filed: 2021-04-19 - Published: 2021-08-05 - Earliest claimed: 2018-04-10 - Assignees: Facebook Technologies, LLC - Applicants: Facebook Technologies, LLC - Inventors: Michael Hall; Byron Taylor - Web page: https://patentgrep.com/patent/US20210241049A1 ## CPC - G06F 3/012: {Head tracking input arrangements} - G06V 30/2504: {Coarse or fine approaches, e.g. resolution of ambiguities or multiscale approaches} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06T 7/73: using feature-based methods - G06V 20/20: in augmented reality scenes - G06V 40/113: {Recognition of static hand signs} - G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships} - G06V 40/193: {Preprocessing; Feature extraction} - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - G06F 3/0425: {using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected (tracking a projected light spot to determine a position on a display surface G06F3/0386)} - G06F 3/04883: for inputting data by handwriting, e.g. gesture or text - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Embodiments relate to tracking and determining a location of an object in an environment surrounding a user. A system includes one or more imaging devices and an object tracking unit. The system identifies an object in a search region, determines a tracking region that is smaller than the search region corresponding to the object, and scans the tracking region to determine a location associated with the object. The system may generate a ranking of objects, determine locations associated with the objects, and generate a model of the search region based on the locations associated with the objects. ## Claims ### Claim 1 A system comprising: one or more imaging devices; and a processor configured to: receive search region image data of a search region captured by the one or more imaging devices; identify objects in the search region based on the search region image data; generate a ranking of the objects; determine a subset of the objects based on the ranking; determine tracking regions corresponding to the subset of the objects, each of the tracking regions being smaller than the search region; and receive a set of tracking region image data of the tracking regions corresponding to the subset of the objects captured by the one or more imaging devices; determine locations associated with the subset of the objects using the set of tracking region image data; and generate a model of the search region based on the locations associated with the subset of the objects. ### Claim 2 (depends on 1) The system of claim 1, wherein the ranking of the objects is generated based on amounts of user interaction with the objects. ### Claim 3 (depends on 2) The system of claim 2, wherein the processor is configured to determine the amounts of user interaction with the objects based on based on user eye tracking. ### Claim 4 (depends on 2) The system of claim 2, wherein the processor is configured to determine the amounts of user interaction with the objects based on based on user hand tracking. ### Claim 5 (depends on 1) The system of claim 1, wherein the one or more imaging devices include multiple imaging devices, each imaging device configured to capture tracking region image data for one object of the subset of objects. ### Claim 6 (depends on 1) The system of claim 1, wherein the processor is further configured to combine objects and models of multiple search regions into an aggregated model of an environment. ### Claim 7 (depends on 1) The system of claim 1, wherein the one or more imaging devices include: a first imaging device configured to generate the search region image data of the search region; and a second imaging device configured to generate tracking region image data of a first tracking region of the tracking regions. ### Claim 8 (depends on 7) The system of claim 7, wherein the first imaging device is a depth camera and the second imaging device is a time of flight sensor. ### Claim 9 (depends on 7) The system of claim 7, wherein: the second imaging device is configured to generate a first portion of the tracking region image data; and the one or more imaging devices include a third imaging device configured to generate a second portion of the tracking region image data of the tracking region. ### Claim 10 (depends on 9) The system of claim 9, wherein the second imaging device and the third imaging device capture different sensor input types. ### Claim 11 (depends on 1) The system of claim 1, wherein the one or more imaging devices includes a single imaging device configured to scan the search region to generate the search region image data and scan the tracking regions to generate the set of tracking region image data, wherein the single imaging device includes one or more lenses to generate the search region image data at a first zoom and the set of tracking region image data at a second zoom greater than the first zoom. ### Claim 12 (depends on 1) The system of claim 1, wherein the subset of objects in the search region image data is at a first resolution and the subset of objects in the set of tracking region image data is at a second resolution higher than the first resolution. ### Claim 13 (depends on 12) The system of claim 12, wherein the one or more imaging devices include a plurality of low resolution cameras to scan the search region to generate the search region image data, and a high-resolution imaging device to scan a tracking region to generate tracking region image data, the high-resolution imaging device including at least one of an acoustic-optic scanner, a line scanner, and a narrow structured light sensor. ### Claim 14 A method, comprising: receiving search region image data of a search region captured by one or more imaging devices; identifying objects in the search region based on the search region image data; generating a ranking of the objects; determining a subset of the objects based on the ranking; determining tracking regions corresponding to the subset of the objects, each of the tracking regions being smaller than the search region; and receiving a set of tracking region image data of the tracking regions corresponding to the subset of the objects captured by the one or more imaging devices; determining locations associated with the subset of the objects using the set of tracking region image data; and generating a model of the search region based on the locations associated with the subset of the objects. ### Claim 15 (depends on 14) The method of claim 14, wherein the ranking of the objects is generated based on amounts of user interaction with the objects ### Claim 16 (depends on 15) The method of claim 15, further comprising determining the amounts of user interaction with the objects based on based on user eye tracking. ### Claim 17 (depends on 15) The method of claim 15, further comprising determining the amounts of user interaction with the objects based on based on user hand tracking. ### Claim 18 (depends on 14) The method of claim 14, further comprising combining objects and models of multiple search regions into an aggregated model of an environment. ### Claim 19 (depends on 14) The method of claim 14, further comprising: generating, by a depth camera of the one or more imaging devices, the search region image data; and generating, by a time of flight sensor of the one or more imaging devices, tracking region image data of a first tracking region of the tracking regions, wherein the ### Claim 20 A non-transitory computer readable medium comprising stored instructions that, when executed by a processor, configures the processor to: receive search region image data of a search region captured by one or more imaging devices; identify objects in the search region based on the search region image data; generate a ranking of the objects; determine a subset of the objects based on the ranking; determine tracking regions corresponding to the subset of the objects, each of the tracking regions being smaller than the search region; and receive a set of tracking region image data of the tracking regions corresponding to the subset of the objects captured by the one or more imaging devices; determine locations associated with the subset of the objects using the set of tracking region image data; and generate a model of the search region based on the locations associated with the subset of the objects.