# US 12,191,888 B2 · Systems and methods for compressing and storing sensor data collected by an autonomous vehicle > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,191,888 B2 - Document ID: US12191888B2 - Status: Granted - Kind: B2 - Application number: 17/139,440 - Filed: 2020-12-31 - Published: 2022-04-14 - Granted: 2025-01-07 - Earliest claimed: 2020-10-12 - Assignees: Aurora Operations, Inc. - Applicants: Aurora Operations, Inc. - Inventors: Joshua Oren Silberman; Dillon Collins - Web page: https://patentgrep.com/patent/US12191888B2 ## CPC - H03M 7/3059: {Digital compression and data reduction techniques where the original information is represented by a subset or similar information, e.g. lossy compression} - B60W 50/00: Details of control systems for road vehicle drive control not related to the control of a particular sub-unit {, e.g. process diagnostic or vehicle driver interfaces} - G05D 1/0088: {characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00)} - G05D 1/81: Handing over between on-board automatic and on-board manual control - G06N 20/00: Machine learning - G07C 5/085: {using electronic data carriers} - H03K 19/173: using elementary logic circuits as components - B60W 2050/0043: {Signal treatments, identification of variables or parameters, parameter estimation or state estimation} - G07C 5/0841: {Registering performance data (recording measured values G01D; information storage G11B)} - H03M: CODING; DECODING; CODE CONVERSION IN GENERAL (using fluidic means F15C4/00; optical analogue/digital converters G02F7/00; coding, decoding or code conversion, specially adapted for particular applications, see the relevant subclasses, e.g. G01D, G01R, G06F, G06T, G09G, G10L, G11B, G11C, H04B, H04L, H04M, H04N; ciphering or deciphering for cryptography or other purposes involving the need for secrecy G09C) - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G07C: TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE - H03K: PULSE TECHNIQUE (measuring pulse characteristics G01R; modulating sinusoidal oscillations with pulses H03C; transmission of digital information H04L; discriminator circuits detecting phase difference between two signals by counting or integrating cycles of oscillation H03D3/04; automatic control, starting, synchronisation or stabilisation of generators of electronic oscillations or pulses where the type of generator is irrelevant or unspecified H03L; coding, decoding or code conversion, in general H03M) ## Abstract A vehicle computing system onboard an autonomous vehicle can include one or more processors and one or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations can include obtaining sensor data from one or more sensors of the autonomous vehicle; applying lossy compression to the sensor data to generate compressed sensor data; storing data describing the compressed sensor data; decompressing the compressed sensor data to generate decompressed sensor data; and inputting data describing the decompressed sensor data into an autonomy system comprising one or more machine-learned models. The autonomy system can be configured to control operations of the autonomous vehicle based on the decompressed sensor data. ## Claims ### Claim 1 A vehicle computing system onboard an autonomous vehicle, the vehicle computing system comprising: one or more processors; one or more non-transitory computer-readable media that store instructions for execution by the one or more processors to cause the vehicle computing system to perform operations, the operations comprising: obtaining sensor data from one or more sensors of the autonomous vehicle; applying lossy compression to the sensor data to generate compressed sensor data; storing data describing the compressed sensor data; decompressing the compressed sensor data to generate decompressed sensor data; providing the compressed sensor data to a remote computing system for logged viewing of the compressed sensor data; providing the decompressed sensor data to the remote computing system for live viewing of the decompressed sensor data; inputting data describing the decompressed sensor data into an autonomy system comprising one or more machine-learned models, the autonomy system configured to control operations of the autonomous vehicle based on the decompressed sensor data; and providing a collection of compressed sensor data including the compressed sensor data from the autonomous vehicle and compressed sensor data from other autonomous vehicles as input to the one or more machine-learned models offboard the autonomous vehicles to train the one or more machine-learned models. ### Claim 2 (depends on 1) The vehicle computing system of claim 1, wherein the sensor data comprises image data. ### Claim 3 (depends on 2) The vehicle computing system of claim 2, wherein the operations further comprise, before applying the lossy compression to the sensor data to generate the compressed sensor data, converting the sensor data into a non-linear color space. ### Claim 4 (depends on 1) The vehicle computing system of claim 1, wherein applying the lossy compression to the sensor data to generate the compressed sensor data comprises applying a lossy compression algorithm to individual image frames of the sensor data without applying motion-based compression. ### Claim 5 (depends on 1) The vehicle computing system of claim 1, wherein applying the lossy compression to the sensor data to generate the compressed sensor data comprises applying a block-oriented integer-discrete cosine transform algorithm. ### Claim 6 (depends on 1) The vehicle computing system of claim 1, further comprising a field programmable gate array, and wherein applying the lossy compression to the sensor data to generate the compressed sensor data comprises applying the lossy compression using the field programmable gate array. ### Claim 7 (depends on 6) The vehicle computing system of claim 6, wherein the sensor data comprises raw Bayer data. ### Claim 8 (depends on 1) The vehicle computing system of claim 1, wherein the one or more machine-learned models of the autonomy system were trained using training sensor data that was previously compressed with the lossy compression. ### Claim 9 (depends on 8) The vehicle computing system of claim 8, wherein the training sensor data that was previously compressed with the lossy compression was collected onboard at least one of the autonomous vehicle or another autonomous vehicle. ### Claim 10 (depends on 1) The vehicle computing system of claim 1, wherein applying the lossy compression to the sensor data to generate the compressed sensor data comprises applying an H264 compression codec. ### Claim 11 A method for compressing and storing sensor data collected by an autonomous vehicle, the method comprising: obtaining, by a vehicle computing system comprising one or more computing devices onboard the autonomous vehicle, sensor data from by one or more sensors of the autonomous vehicle; applying, by the vehicle computing system, lossy compression to the sensor data to generate compressed sensor data; storing, by the vehicle computing system, data describing the compressed sensor data; decompressing, by the vehicle computing system, the compressed sensor data to generate decompressed sensor data; providing the compressed sensor data to a remote computing system for logged viewing of the compressed sensor data; providing the decompressed sensor data to the remote computing system for live viewing of the decompressed sensor data; inputting, by the vehicle computing system, data describing the decompressed sensor data into an autonomy system comprising one or more machine-learned models, the autonomy system configured to control operations of the autonomous vehicle based on the decompressed sensor data; and providing a collection of compressed sensor data including the compressed sensor data from the autonomous vehicle and compressed sensor data from other autonomous vehicles as input to the one or more machine-learned models offboard the autonomous vehicles to train the one or more machine-learned models. ### Claim 12 (depends on 11) The method of claim 11, wherein the sensor data comprises image data. ### Claim 13 (depends on 12) The method of claim 12, further comprising, before applying the lossy compression to the sensor data to generate the compressed sensor data, converting, by the vehicle computing system, the sensor data into a non-linear color space. ### Claim 14 (depends on 12) The method of claim 12, wherein applying the lossy compression to the sensor data to generate the compressed sensor data comprises applying, by the vehicle computing system, a lossy compression algorithm to individual image frames of the sensor data without applying motion-based compression. ### Claim 15 (depends on 12) The method of claim 12, wherein applying the lossy compression to the sensor data to generate the compressed sensor data comprises applying the lossy compression using a field programmable gate array. ### Claim 16 (depends on 11) The method of claim 11, wherein applying the lossy compression to the sensor data to generate the compressed sensor data comprises applying, by the vehicle computing system, a block-oriented integer-discrete cosine transform algorithm. ### Claim 17 (depends on 11) The method of claim 11, wherein the sensor data comprises raw Bayer data. ### Claim 18 (depends on 11) The method of claim 11, wherein the one or more machine-learned models of the autonomy system were trained using training sensor data that was previously compressed with the lossy compression. ### Claim 19 (depends on 18) The method of claim 18, wherein the training sensor data that was previously compressed onboard one or more of the autonomous vehicle or another autonomous vehicle. ### Claim 20 A computing system comprising: a vehicle computing system onboard an autonomous vehicle, the vehicle computing system comprising one or more processors and one or more non-transitory computer-readable media that store instructions for execution by the one or more processors to cause the vehicle computing system to perform operations, the operations comprising: obtaining sensor data from by one or more sensors of the autonomous vehicle; applying lossy compression to the sensor data to generate compressed sensor data; storing data describing the compressed sensor data; decompressing the compressed sensor data to generate decompressed sensor data; providing the compressed sensor data to a remote computing system for logged viewing of the compressed sensor data; and providing the decompressed sensor data to the remote computing system for live viewing of the decompressed sensor data; and a training computing system offboard the autonomous vehicle, the training computing system comprising one or more processors and one or more non-transitory computer-readable media that store instructions that are executable by the one or more processors to cause the training computing system to perform operations, the operations comprising: receiving a collection of compressed sensor data including the compressed sensor data from the vehicle computing system and compressed sensor data from other autonomous vehicles; and providing the collection of compressed sensor data as input to one or more machine-learned models to train the one or more machine-learned models. ### Claim 21 (depends on 20) The computing system of claim 20, wherein the sensor data comprises image data.