US 7,127,520 B2 · Method and system for transforming input data streams

Abstract

The present system and method transforms an input data stream in a first data format of a plurality of first data formats to an output data stream in a second data format of a plurality of second data formats. A plurality of input connector modules receive respective input data streams and at least one input queue stores the received input data streams. A plurality of job threads is operatively connected to the at least one input queue, each job thread, in parallel with at least one other job thread, formatting a stored input data stream to produce an output data stream. At least one output queue respectively stores the output data streams from the plurality of job threads. A plurality of output connector modules is operatively connected to the at least one output queue, the output connector modules supplying respective output data streams.

Claims27

Dependent claims

A system for transforming an input data stream in a first data format of a plurality of first data formats to an output data stream in a second data format of a plurality of second data formats, comprising:

a parsing model comprising information about message structure;
at least one input queue for storing input data streams;
a plurality of job threads operatively connected to the at least one input queue, each job thread, in parallel with at least one other job thread, formatting a respective stored input data stream to produce an output data stream, each job thread comprising:
a filter for filtering irrelevant data from the respective retrieved input data stream;
an event identifier for identifying events in the input data stream, the events comprising sequences, patterns, or both, in the input data stream;
a field list for each of the identified events, the fields comprising data, variables, or both in the input stream;
a message generator for generating messages associated with each of said identified event in response to the field list for said identified event and said information about message structure;
and
a processor for processing each message in response to said information about message structure to generate meta records, the meta records comprising data that is not formatted for a specific output device;
and
a formatter for formatting the meta records to produce an output data stream;
at least one output queue for respectively storing the output data streams from the plurality of job threads.

9 more elements

a parsing model comprising information about message structure;
a plurality of input connector modules for receiving respective input data streams;
a plurality of input queues for storing received input data streams;
a plurality of job threads operatively connected to at least one of respective input connector modules of the plurality of input connector modules and respective input queues of the plurality of the input queues, each job thread in parallel with at least one other job thread formatting a stored input data stream to produce an output data stream, each job thread comprising:
filter for filtering irrelevant data from the respective retrieved input data stream;
an event identifier for identifying events in the input data stream, the events comprising sequences, patterns, or both, in the input data stream;
a field list for each of the identified events, the fields comprising data, variables, or both in the input stream;
a message generator for generating messages associated with each of said identified event in response to the field list for said identified event and said information about message structure;
and
a processor for processing each message in response to information about message structure to generate meta records, the meta records comprising data that is not formatted for a specific output device;
and
a formatting for formatting the meta records to produce an output data stream;
a plurality of output queues for respectively storing the output data streams from the plurality of job threads;
and
a plurality of output connector modules operatively connected to at least one of the output queues and the job threads, the output connector modules supplying respective output data streams.

12 more elements

a parsing model comprising information about message structure;
a plurality of input connector modules for receiving respective input data streams;
at least one input queues for storing received input data streams;
a plurality of job threads operatively connected to the at least one input queue, each job thread, in parallel with at least one other job thread, formatting a stored input data stream to produce an output data stream, each job thread comprising:
a filter for filtering irrelevant data from the respective retrieved input data stream;
an event identifier for identifying events in the input data stream, the events comprising sequences, patterns, or both, in the input data stream;
a field list for each of the identified events, the fields comprising data, variables, or both in the input stream;
a message generator for generating messages associated with each of said identified event in response to the field list for said identified event and said information about message structure;
and
a processor for processing each message in response to said information about message structure to generate meta records, the meta records comprising data that is not formatted for a specific output device;
and
a formatter for formatting the meta records to produce an output data stream;
at least one output queue for respectively storing the output data streams from the plurality of job threads;
and
a plurality of output connector modules operatively connected to the at least one output queue, the output connector modules supplying respective output data streams.

12 more elements

A method for transforming an input data stream in a first data format of a plurality of first data formats to an output data stream in a second data format of a plurality of second data formats, comprising:

providing a parsing model comprising information about message structure;
receiving input data streams, each of the input data streams being in a respective data format of a plurality of first data formats;
storing the input data streams in at least one input queue;
retrieving, via job threads, stored input data streams from the at least one input queue;
formatting a respective retrieved input data stream that is in a first data format of a plurality of first data formats by a respective job thread, the formatting by the respective job thread of the respective retrieved input data stream being in parallel with formatting by at least one other job thread of another respective retrieved input data stream, each job thread:
filtering irrelevant data from the respective retrieved input data stream;
identifying events in the input data stream, the events comprising sequences, patterns or both, in the input data stream;
creating a field list for each of the identified events, the fields comprising data, variables, or both in the input stream;
and
generating message associated with each of said identified event in response to the field list for said identified event and said information about message structure;
processing each message in response to said information about message structure to generate meta records, the meta records comprising data that is not formatted for a specific output device;
formatting the meta records into an output data stream;
and
storing the output data streams in at least one output queue;
selecting one of the output data streams in the at least one output queue;
and
outputting the selected output data stream.

14 more elements

A computer readable storage medium containing embedded computer program code for transforming an input data stream in a first data format of a plurality of first data formats to an output data stream in a second data format of a plurality of second data formats, the computer readable storage media containing computer program code segments comprising:

a first computer program code segment that provides a parsing model comprising information about message structure;
a second computer program code segment that stores input data streams in at least one input queue;
a third computer program code segment that retrieves, via job threads, input data streams from the at least one input queue;
and
a fourth computer program code segment that formats a respective retrieved input data stream by a respective job thread to produce a respective output data stream, the respective retrieved input data stream being formatted in parallel with at least one other respective retrieved input data stream, each job thread:
filtering irrelevant data from the respective retrieved input data stream;
identifying events in the input data stream, the events comprising sequences, patterns or both, in the input data stream;
creating a field list for each of the identified events, the fields comprising data, variables, or both in the input stream;
and
generating message associated with each of said identified event in response to the field list for said identified event and said information about message structure;
a fifth computer program code segment that processes each message in response to said information about message structure to generate meta records, the meta records comprising data that is not formatted for a specific output device;
a sixth computer program code segment that formats the meta records into an output data stream;
and
a seventh computer program code segment that stores the output data streams in at least one output queue.

11 more elements

Dependencies are inferred from the claim text.

Method and system for transforming input data streams

Life of this patent

Method and system for transforming input data streams

Life of this patent

Abstract

Claims27