# US 10,122,783 B2 · Dynamic data-ingestion pipeline > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,122,783 B2 - Document ID: US10122783B2 - Status: Granted - Kind: B2 - Application number: 14/944,934 - Filed: 2015-11-18 - Published: 2017-05-18 - Granted: 2018-11-06 - Earliest claimed: 2015-11-18 - Assignees: Microsoft Technology Licensing, LLC - Applicants: LinkedIn Corporation - Inventors: Lin Qiao; Yinan Li; Sahil Takiar; Ziyang Liu; Narasimha R. Veeramreddy; Min Tu; Ying Dai; Issac Buenrostro; Kapil L. Surlaker; Shirshanka Das; Chavdar Botev; Kenneth D. Goodhope - Web page: https://patentgrep.com/patent/US10122783B2 ## CPC - H04L 67/02: based on web technology, e.g. hypertext transfer protocol [HTTP] - G06F 16/11: File system administration, e.g. details of archiving or snapshots (error detection or correction of the data by redundancy in operations G06F11/14) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract In order to ingest data from an arbitrary source in a set of sources, a computer system accesses predefined configuration instructions. Then, the computer system generates a dynamic data-ingestion pipeline that is compatible with a Hadoop file system based on the predefined configuration instructions. This dynamic data-ingestion pipeline includes a modular arrangement of operators from a set of operators that includes: an extraction operator for extracting the data of interest from the source, a converter operator for transforming the data, and a quality-checker operator for checking the transformed data. Moreover, the computer system receives the data from the source. Next, the computer system processes the data using the dynamic data-ingestion pipeline as the data is received without storing the data in memory for the purpose of subsequent ingestion processing. ## Claims ### Claim 1 A computer-system-implemented method for ingesting data from a source in a set of sources, the method comprising: accessing predefined configuration instructions; generating a dynamic data-ingestion pipeline compatible with a Hadoop file system based on the predefined configuration instructions, wherein the dynamic data-ingestion pipeline includes a modular arrangement of operators from a set of operators that includes: an extraction operator for extracting the data of interest from the source, a converter operator for transforming the data, and a quality-checker operator for checking the transformed data; receiving the data from the source; and processing the data, by a computer system, using the dynamic data-ingestion pipeline implemented on the computer system by: using the converter operator to transform the received data into a common format compatible with the Hadoop file system; wherein the data is processed in real-time as the data is received, without storing the data in memory for the purpose of subsequent ingestion processing, thereby reducing storage requirements of the computer system. ### Claim 2 (depends on 1) The method of claim 1, wherein the set of sources is compatible with one of: a database, a message broker, a distributed key-value storage system, a Simple Storage Service (S3) file system, a first file system on a first network accessible via HyperText Transfer Protocol, a second file system on a second network accessible via File Transfer Protocol, and a local file system. ### Claim 3 (depends on 1) The method of claim 1, wherein the dynamic data-ingestion pipeline egresses the data to a sink in a set of sinks; wherein the sink is compatible with the Hadoop file system; and wherein the set of operators includes a writer operator for outputting the data to the sink. ### Claim 4 (depends on 1) The method of claim 1, wherein the set of operators includes a publisher operator for outputting the data to an output directory. ### Claim 5 (depends on 4) The method of claim 4, wherein the publisher operator outputs the data when all of the operators in the dynamic data-ingestion pipeline are successfully completed. ### Claim 6 (depends on 4) The method of claim 4, wherein the publisher operator outputs the data when a subset of the operators in the dynamic data-ingestion pipeline is successfully completed. ### Claim 7 (depends on 1) The method of claim 1, wherein the processing of the data using the dynamic data-ingestion pipeline is performed as a batch process. ### Claim 8 (depends on 1) The method of claim 1, wherein the processing of the data using the dynamic data-ingestion pipeline involves parallel processing of workunits. ### Claim 9 (depends on 1) The method of claim 1, wherein the quality-checker operator checks one of: a record-level policy, and a task-level policy. ### Claim 10 An apparatus, comprising: one or more processors; memory; and a program module, wherein the program module is stored in the memory and, during operation of the apparatus, is executed by the one or more processors to ingest data from a source in a set of sources, the program module including: instructions for accessing predefined configuration instructions; instructions for generating a dynamic data-ingestion pipeline compatible with a Hadoop file system based on the predefined configuration instructions, wherein the dynamic data-ingestion pipeline includes a modular arrangement of operators from a set of operators that includes: an extraction operator for extracting the data of interest from the source, a converter operator for transforming the data, and a quality-checker operator for checking the transformed data; instructions for receiving the data from the source; and instructions for processing the data using the dynamic data-ingestion pipeline by: using the converter operator to transform the received data into a common format compatible with the Hadoop file system; wherein the data is processed in real-time as the data is received, without storing the data in memory for the purpose of subsequent ingestion processing, thereby reducing storage requirements on the memory. ### Claim 11 (depends on 10) The apparatus of claim 10, wherein the set of sources is compatible with one of: a database, a message broker, a distributed key-value storage system, a Simple Storage Service (S3) file system, a first file system on a first network accessible via HyperText Transfer Protocol, a second file system on a second network accessible via File Transfer Protocol, and a local file system. ### Claim 12 (depends on 10) The apparatus of claim 10, wherein the dynamic data-ingestion pipeline egresses the data to a sink in a set of sinks; wherein the sink is compatible with the Hadoop file system; and wherein the set of operators includes a writer operator for outputting the data to the sink. ### Claim 13 (depends on 10) The apparatus of claim 10, wherein the set of operators includes a publisher operator for outputting the data to an output directory. ### Claim 14 (depends on 13) The apparatus of claim 13, wherein the publisher operator outputs the data when all of the operators in the dynamic data-ingestion pipeline are successfully completed. ### Claim 15 (depends on 13) The apparatus of claim 13, wherein the publisher operator outputs the data when a subset of the operators in the dynamic data-ingestion pipeline is successfully completed. ### Claim 16 (depends on 10) The apparatus of claim 10, wherein the processing of the data using the dynamic data-ingestion pipeline is performed as a batch process. ### Claim 17 (depends on 10) The apparatus of claim 10, wherein the processing of the data using the dynamic data-ingestion pipeline involves parallel processing of workunits. ### Claim 18 (depends on 10) The apparatus of claim 10, wherein the quality-checker operator checks one of: a record-level policy, and a task-level policy. ### Claim 19 A system, comprising: a processing module comprising one or more processors and a non-transitory computer readable medium storing instructions that, when executed by the one or more processors, cause the system to: access predefined configuration instructions; generate a dynamic data-ingestion pipeline compatible with a Hadoop file system based on the predefined configuration instructions, wherein the dynamic data-ingestion pipeline includes a modular arrangement of operators from a set of operators that includes: an extraction operator for extracting the data of interest from the source, a converter operator for transforming the data, and a quality-checker operator for checking the transformed data; receive the data from a source in a set of sources; and process the data using the dynamic data-ingestion pipeline by: using the converter operator to transform the received data into a common format compatible with the Hadoop file system; wherein the data is processed in real-time as the data is received, without storing the data in memory for the purpose of subsequent ingestion processing, thereby reducing storage requirements of the system. ### Claim 20 (depends on 19) The system of claim 19, wherein the dynamic data-ingestion pipeline egresses the data to a sink in a set of sinks; wherein the sink is compatible with the Hadoop file system; and wherein the set of operators includes a writer operator for outputting the data to the sink.