Load shedding in a data stream management system

A data stream processing system, comprising:

• a DSMS comprising:
• a data receiving module configured to receive:
• an input data stream from each of a plurality of data stream sources;
• and
• respective location information associated with sets of at least one of the data stream sources:
• a continuous query execution module configured to execute a continuous query against data items received via the input data streams to generate at least one output data stream for output to a client of the DSMS;
• a load shedding module configured to execute a load shedding process when the DSMS is overloaded with data from the input data streams, so as to discard a plurality of the data items one of:
• as received by the DSMS;
• and
• that have been partially processed by the DSMS;
• a learning module configured to determine, when the DSMS is not overloaded and for the location information associated with each of the sets of the at least one data stream source, a respective utility value indicating a utility to the client of the DSMS of data from the set of the at least one data stream source, by processing a feedback signal that is based on the at least one output data stream, the learning module being configured to store the location information in association with the corresponding data utility value;
• and
• the load shedding module being configured to control the load shedding process by using location information received by the data receiving module when the DSMS is overloaded, together with the location information and data utility values stored by the learning module, to identify at least one input data stream whose data items are to be discarded;
• and
• a DSMS client configured to receive and process the at least one output data stream generated by the DSMS, and configured to execute an action when triggered by the processing of at least one data item in the output data stream, the action comprising the sending by the DSMS client of a signal to a recipient different from the DSMS client;
• and
• a feedback loop configured to convey a feedback signal to the DSMS notifying the execution of an action by the DSMS client;
• the learning module of the DSMS is configured to determine, when the DSMS is not overloaded and for the location information associated with each of the sets of at least one data stream source, a respective utility value indicating a utility to the DSMS client of data from the at least one data stream source in the set, by processing the feedback signal conveyed by the feedback loop, the learning module being configured to store the location information in association with the corresponding data utility value.

A method of processing stream data in a data stream management system, DSMS, the method comprising the DSMS:

• receiving an input data stream from each of a plurality of data stream sources and respective location information associated with sets of at least one er-more of the data stream sources;
• executing a continuous query against data items received via the input data streams to generate at least one output data stream for output to a client of the DSMS;
• determining, when the DSMS is not overloaded with data from the input data streams and for the location information associated with each of the sets of at least one data stream source a respective utility value indicating a utility to the client of the DSMS of data from the set of at least one data stream source, by processing a feedback signal that is based on the at least one output data stream;
• storing, in association with the location information associated with each of the sets of at least one data stream source the corresponding data utility value;
• and
• executing a load shedding process when the DSMS is overloaded with data from the input data streams, so as to discard some of the data items as one of received by the DSMS and that have been partially processed by the DSMS;
• and
• the load shedding process being controlled by using location information received when the DSMS is overloaded, together with the stored location information and data utility values, to identify one or more input data streams whose data items are to be discarded.

A non-transitory computer-readable storage medium storing computer program instructions which, when executed by a processor, cause the processor to perform a method of processing stream data in a data stream management system, DSMS, the method comprising the DSMS:

• receiving an input data stream from each of a plurality of data stream sources and respective location information associated with sets of at least one of the data stream sources;
• executing a continuous query against data items received via the input data streams to generate at least one output data stream for output to a client of the DSMS;
• determining, when the DSMS is not overloaded with data from the input data streams and for the location information associated with each of the sets of at least one data stream source, a respective utility value indicating a utility to the client of the DSMS of data from the set of at least one data stream source, by processing a feedback signal that is based on the at least one output data stream;
• storing, in association with the location information associated with each of the sets of at least one data stream source, the corresponding data utility value;
• and
• executing a load shedding process when the DSMS is overloaded with data from the input data streams, so as to discard some of the data items as one of received by the DSMS and that have been partially processed by the DSMS, the load shedding process being controlled by using location information received when the DSMS is overloaded, together with the stored location information and data utility values, to identify one or more input data streams whose data items are to be discarded.