# US 2021/0279465 A1 · Streaming object detection within sensor data > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0279465 A1 - Document ID: US20210279465A1 - Status: Published application - Kind: A1 - Application number: 16/812,154 - Filed: 2020-03-06 - Published: 2021-09-09 - Earliest claimed: 2020-03-06 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Jonathon Shlens; Vijay Vasudevan; Jiquan Ngiam; Wei Han; Zhifeng Chen; Brandon Chauloon Yang; Benjamin James Caine; Zhengdong Zhang; Christoph Sprunk; Ouais Alsharif; Junhua Mao; Chen Wu - Web page: https://patentgrep.com/patent/US20210279465A1 ## CPC - G01S 17/89: for mapping or imaging - G01S 17/931: of land vehicles - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/048: Activation functions - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06V 10/764: using classification, e.g. of video objects - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for processing data generated by a sensing system that rotationally senses an environment. In one aspect, a method comprises partitioning a predetermined period of time into a plurality of sub-periods, wherein the predetermined period of time is a period of time for which data generated by the sensing system constitutes a complete rotational sensing of the environment; for each sub-period: receiving current data generated by the sensing system during the sub-period and characterizing a respective partial scene of the environment; processing the current data using an object detection neural network to generate a current object detection output that is specific to the respective partial scene of the environment. ## Claims ### Claim 1 A method of processing data, comprising: partitioning a predetermined period of time into a plurality of sub-periods, wherein the predetermined period of time is a period of time for which data generated by a sensing system that rotationally senses an environment constitutes a complete rotational sensing of the environment; for each sub-period: receiving current data generated by the sensing system during the sub-period and characterizing a respective partial scene of the environment; processing the current data to generate a current object detection output that is specific to the respective partial scene of the environment, wherein the processing comprises: processing (i) a current sequential processing subnetwork input and (ii) a preceding sequential processing subnetwork output that is generated by a sequential processing subnetwork from a preceding sub-period, to generate a current sequential processing subnetwork output; and processing, for the current data received, (i) the current sequential processing subnetwork output and (ii) a preceding object detection output that is generated by an object detection subnetwork from the preceding sub period, to generate the current object detection output. ### Claim 2 (depends on 1) The method of claim 1, further comprising: processing the current data generated by the sensing system for the sub-period to generate a current intermediate representation; and processing, by the object detection subnetwork, the current intermediate representation when generating the current object detection output. ### Claim 3 (depends on 1) The method of claim 1, wherein the current sequential processing subnetwork input is the current intermediate representation. ### Claim 4 (depends on 1) The method of claim 1, wherein the environment is an environment in a vicinity of a vehicle, and wherein the sensing system is on-board the vehicle. ### Claim 5 (depends on 1) The method of claim 1, wherein the sensing system comprises a spinning LIDAR sensor, and wherein the data is three-dimensional point cloud data. ### Claim 6 (depends on 1) The method of claim 1, wherein the object detection output comprises: (i) data defining one or more bounding boxes in the current data generated by the spinning sensor in the sub period, and (ii) for each of the one or more bounding boxes, a respective confidence score that represents a likelihood that an object belonging to an object category from a set of one or more object categories is present in the region of the environment shown in the bounding box. ### Claim 7 (depends on 1) The method of claim 1, wherein partitioning the predetermined period of time into the plurality of sub-periods comprises: determining a plurality of candidate sub-periods that are each of different time lengths from each other; computing, for each candidate sub-period, a respective ratio of (i) a time length of the candidate sub-period to (ii) a time length required for processing the current data that is generated by the sensing system during the candidate sub-period using the object detection neural network; determining, based at least on the respective ratios, a determined time length of the sub-period; and partitioning the predetermined period of time into the plurality of sub-periods to each have the determined time length. ### Claim 8 (depends on 7) The method of claim 7, wherein determining the determined time length of the sub-period further comprises, for each candidate sub-period: determining that a measure of accuracy of the current object detection output satisfies a predetermined detection accuracy threshold. ### Claim 9 A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement: a partitoner that partitions a predetermined period of time into a plurality of sub-periods, wherein the predetermined period of time is a period of time for which data generated by a sensing system that rotationally senses an environment constitutes a complete rotational sensing of the environment, and generates, for each sub-period, current data generated by the sensing system during the sub-period and characterizing a respective partial scene of the environment; and an object detection neural network that processes, for each sub-period, the current data using an object detection neural network to generate a current object detection output that is specific to the respective partial scene of the environment, wherein the object detection neural network comprises: a sequential processing subnetwork that processes (i) a current sequential processing subnetwork input and (ii) a preceding sequential processing subnetwork output that is generated by the sequential processing subnetwork from a preceding sub-period, to generate a current sequential processing subnetwork output; and an object detection subnetwork that processes, for the current data received, (i) the current sequential processing subnetwork output and (ii) a preceding object detection output that is generated by the object detection subnetwork from the preceding sub period, to generate the current object detection output. ### Claim 10 (depends on 9) The system of claim 9, wherein: the object detection neural network further comprises a featurization subnetwork that process the current data generated by the sensing system for the sub-period to generate a current intermediate representation; and the object detection subnetwork is also configured to process the current intermediate representation when generating the current object detection output. ### Claim 11 (depends on 9) The system of claim 9, wherein the current sequential processing subnetwork input is the current intermediate representation. ### Claim 12 (depends on 9) The system of claim 9, wherein the environment is an environment in a vicinity of a vehicle, and wherein the sensing system is on-board the vehicle. ### Claim 13 (depends on 9) The system of claim 9, wherein the sensing system comprises a spinning LIDAR sensor, and wherein the data is three-dimensional point cloud data. ### Claim 14 (depends on 9) The system of claim 9, wherein the partitioner partitions the predetermined period of time into the plurality of sub-periods by performing operations comprising: determining a plurality of candidate sub-periods that are each of different time lengths from each other; computing, for each candidate sub-period, a respective ratio of (i) a time length of the candidate sub-period to (ii) a time length required for processing the current data that is generated by the sensing system during the candidate sub-period using the object detection neural network; determining, based at least on the respective ratios, a determined time length of the sub-period; and partitioning the predetermined period of time into the plurality of sub-periods to each have the determined time length. ### Claim 15 (depends on 9) The system of claim 9, wherein determining the determined time length of the sub-period further comprises, for each candidate sub-period: determining that a measure of accuracy of the current object detection output satisfies a predetermined detection accuracy threshold. ### Claim 16 One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising: partitioning a predetermined period of time into a plurality of sub-periods, wherein the predetermined period of time is a period of time for which data generated by a sensing system that rotationally senses an environment constitutes a complete rotational sensing of the environment; for each sub-period: receiving current data generated by the sensing system during the sub-period and characterizing a respective partial scene of the environment; processing the current data using an object detection neural network to generate a current object detection output that is specific to the respective partial scene of the environment, wherein the object detection neural network comprises: a sequential processing subnetwork that processes (i) a current sequential processing subnetwork input and (ii) a preceding sequential processing subnetwork output that is generated by the sequential processing subnetwork from a preceding sub-period, to generate a current sequential processing subnetwork output; and an object detection subnetwork that processes, for the current data received, (i) the current sequential processing subnetwork output and (ii) a preceding object detection output that is generated by the object detection subnetwork from the preceding sub period, to generate the current object detection output. ### Claim 17 (depends on 16) The non-transitory computer storage media of claim 16, wherein: the object detection neural network further comprises a featurization subnetwork that process the current data generated by the sensing system for the sub-period to generate a current intermediate representation; and the object detection subnetwork is also configured to process the current intermediate representation when generating the current object detection output. ### Claim 18 (depends on 16) The non-transitory computer storage media of claim 16, wherein the current sequential processing subnetwork input is the current intermediate representation. ### Claim 19 (depends on 16) The non-transitory computer storage media of claim 16, wherein partitioning the predetermined period of time into the plurality of sub-periods comprises: determining a plurality of candidate sub-periods that are each of different time lengths from each other; computing, for each candidate sub-period, a respective ratio of (i) a time length of the candidate sub-period to (ii) a time length required for processing the current data that is generated by the sensing system during the candidate sub-period using the object detection neural network; determining, based at least on the respective ratios, a determined time length of the sub-period; and partitioning the predetermined period of time into the plurality of sub-periods to each have the determined time length. ### Claim 20 (depends on 16) The non-transitory computer storage media of claim 16, wherein determining the determined time length of the sub-period further comprises, for each candidate sub-period: determining that a measure of accuracy of the current object detection output satisfies a predetermined detection accuracy threshold.