# US 12,182,320 B2 · Systems and methods for streaming-based ojbect recognition > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,182,320 B2 - Document ID: US12182320B2 - Status: Granted - Kind: B2 - Application number: 17/718,958 - Filed: 2022-04-12 - Published: 2023-10-12 - Granted: 2024-12-31 - Earliest claimed: 2022-04-12 - Assignees: Meta Platforms Technologies, LLC - Applicants: Meta Platforms Technologies, LLC - Inventors: Kyungho Kim; Yee Sin Chan; Kyunghun Jung - Web page: https://patentgrep.com/patent/US12182320B2 ## CPC - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 16/70: of video data - G01S 17/06: Systems determining position data of a target - G06T 7/248: {involving reference images or patches} - H04N 13/344: with head-mounted left-right displays - H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - H04N 21/816: {involving special video data, e.g 3D video} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Systems and methods for streaming-based object recognition may include a device which transmits, to a server, video data and first coordinates of a first viewport for the device at a first time instance. The video data may include one or more objects in the first viewport. The device may transmit second coordinates of a second viewport for the device at a second time instance. The device may receive, from the server, data corresponding to the one or more objects within the second viewpoint. The data may be received in a sequence according to coordinates of the one or more objects relative to the second viewport. The device may render the data relative to the one or more objects within a third viewport at a third time instance. ## Claims ### Claim 1 A method comprising: transmitting, by a device to a server, video data and first coordinates of a first viewport for the device at a first time instance, the video data including one or more objects in the first viewport; transmitting, by the device, second coordinates of a second viewport for the device at a second time instance; receiving, by the device from the server, data corresponding to the one or more objects within the second viewport, the data received in a sequence according to coordinates of the one or more objects relative to the second viewport; and rendering, by the device, the data relative to the one or more objects within a third viewport at a third time instance. ### Claim 2 (depends on 1) The method of claim 1, comprising: receiving, by the device, the data from the server in the sequence according to the coordinates of the one or more objects in relation to a reference point in the second viewport. ### Claim 3 (depends on 2) The method of claim 2, wherein the one or more objects comprise a first object located at a first distance from the reference point and a second object located at a second distance from the reference point, the method comprising: receiving, by the device, the data for the first object from the server prior to the data for the second object, when the second distance is greater than the first distance. ### Claim 4 (depends on 1) The method of claim 1, comprising: determining, by the device, the first coordinates for the first viewport based on an orientation of the device using sensor data of the device. ### Claim 5 (depends on 1) The method of claim 1, further comprising: establishing, by the device with the server, one or more parameters for at least one of processing or transmitting the video data; and providing, by the device, the video data according to the established one or more parameters. ### Claim 6 (depends on 1) The method of claim 1, comprising: determining, by the device, whether available computational resources of the device satisfy a threshold criteria. ### Claim 7 (depends on 6) The method of claim 6, comprising, at a fourth time instance, responsive to the available computational resources satisfying the threshold criteria: detecting, by the device, an object from second video data; determining, by the device, one or more features for the object; and transmitting, by the device, the object's coordinates and the one or more features to the server. ### Claim 8 (depends on 1) The method of claim 1, wherein the one or more objects comprise at least one object located within the first viewport and outside the second viewport, the method comprising: determining, by the device, that the at least one object is within the third viewport; and rendering, by the device, the data relative to the at least one object within the third viewport. ### Claim 9 (depends on 1) The method of claim 1, wherein the one or more objects comprise at least one object located within the second viewport, the method comprising: determining, by the device, that the at least one object is outside the third viewport; and forego rendering, by the device within the third viewport, the data corresponding to the at least one object. ### Claim 10 A device comprising: a transceiver configured to: transmit, to a server, video data and first coordinates of a first viewport for the device at a first time instance, the video data including one or more objects in the first viewport; transmit, to the server, second coordinates of a second viewport for the device at a second time instance; receive, from the server, data corresponding to the one or more objects within the second viewport, the data received in a sequence according to coordinates of the one or more objects relative to the second viewport; and at least one processor configured to render the data relative to the one or more objects within a third viewport at a third time instance. ### Claim 11 (depends on 10) The device of claim 10, wherein the transceiver is configured to: receive the data from the server in the sequence according to the coordinates of the one or more objects in relation to a reference point in the second viewport. ### Claim 12 (depends on 11) The device of claim 11, wherein the one or more objects comprise a first object located at a first distance from the reference point and a second object located at a second distance from the reference point, wherein the transceiver is configured to: receive the data for the first object from the server prior to the data for the second object, when the second distance is greater than the first distance. ### Claim 13 (depends on 10) The device of claim 10, wherein the at least one processor is configured to: determine the first coordinates for the first viewport based on an orientation of the device using sensor data of the device. ### Claim 14 (depends on 10) The device of claim 10, wherein the at least one processor is configured to: establish, with the server, one or more parameters for at least one of processing or transmitting the video data; and provide the video data according to the established one or more parameters. ### Claim 15 (depends on 10) The device of claim 10, wherein the at least one processor is configured to: determine, at a fourth time instance, whether available computational resources of the device satisfy a threshold criteria. ### Claim 16 (depends on 15) The device of claim 15, wherein the at least one processor is configured to: responsive to the available computational resources satisfying the threshold criteria, detect an object from second video data; and determine one or more features for the object; wherein the transceiver is configured to transmit the object's coordinates and the one or more features to the server. ### Claim 17 (depends on 10) The device of claim 10, wherein the one or more objects comprise at least one object located within the first viewport and outside the second viewport, wherein the at least one processor is configured to: determine that the at least one object is within the third viewport; and render the data relative to the at least one object within the third viewport. ### Claim 18 (depends on 10) The device of claim 10, wherein the one or more objects comprise at least one object located within the second viewport, wherein the at least one processor is configured to: determine that the at least one object is outside the third viewport; and forego rendering of the data corresponding to the at least one object within the third viewport. ### Claim 19 A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to: transmit, via a transceiver to a server, video data and first coordinates of a first viewport for the device at a first time instance, the video data including one or more objects in the first viewport; transmit, via the transceiver to the server, second coordinates of a second viewport for the device at a second time instance; receive, via the transceiver from the server, data corresponding to the one or more objects within the second viewport, the data received in a sequence according to coordinates of the one or more objects relative to the second viewport; and render the data relative to the one or more objects within a third viewport at a third time instance. ### Claim 20 (depends on 19) The non-transitory computer-readable medium of claim 19, wherein the instructions further cause the one or more processors to: receive the data from the server in the sequence according to the coordinates of the one or more objects in relation to a reference point in the second viewport. ### Claim 21 (depends on 20) The non-transitory computer-readable medium of claim 20, wherein the one or more objects comprise a first object located at a first distance from the reference point and a second object located at a second distance from the reference point, wherein the instructions further cause the one or more processors to: receive the data for the first object from the server prior to the data for the second object, when the second distance is greater than the first distance.