# US 10,372,734 B2 · Controlling operation of a machine by performing reconciliation using a distributed cluster of nodes > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,372,734 B2 - Document ID: US10372734B2 - Status: Granted - Kind: B2 - Application number: 15/835,780 - Filed: 2017-12-08 - Published: 2018-08-09 - Granted: 2019-08-06 - Earliest claimed: 2017-02-04 - Assignees: SAS Institute Inc. - Applicants: SAS Institute Inc. - Inventors: Michele Angelo Trovero; Byron Davis Biggs; Jennifer Leigh Sloan Beeman; Michael James Leonard - Web page: https://patentgrep.com/patent/US10372734B2 ## CPC - G06F 16/285: {Clustering or classification} - G05B 13/0265: {the criterion being a learning criterion} - G06F 16/215: Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors - G06F 16/288: {Entity relationship models} - H04L 69/28: Timers or timing mechanisms used in protocols - H04L 69/40: for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection (management of faults, events, alarms or notifications in data switching networks H04L41/06) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract The operation of a machine can be controlled by performing reconciliation using a cluster of nodes. In one example, a node can receive parent timestamped data from a parent dataset and child timestamped data from child datasets that are children of the parent dataset in a hierarchical relationship. The parent timestamped data and the child timestamped data can relate to an operational characteristic of the machine. The node can generate computer processing-threads. Each computer processing-thread can solve one or more respective reconciliation problems between a parent data point that has a particular timestamp in the parent timestamped data and child data points that also have the particular timestamp in the child timestamp data to generate a reconciled dataset. An operational setting of the machine can then be adjusted based on the reconciled dataset. ## Claims ### Claim 1 A method for performing distributed processing using a cluster of nodes, the method comprising: obtaining parent timestamped data and child timestamped data, wherein the child timestamped data is a child of the parent timestamped data in a hierarchical relationship; performing a first distribution operation involving distributing each parent data-point in the parent timestamped data among the cluster of nodes such that each node in the cluster of nodes is provided with a set of parent data-points that is different from the other nodes in the cluster of nodes; performing a second distribution operation involving, for each parent data-point in the parent timestamped data: identifying a child data-point in the child timestamped data that has the same timestamp as the parent data-point, wherein the parent data-point and the child data-point collectively form a parent-child pair; and transmitting the child data-point to the same node in the cluster of nodes to which the parent data-point was previously transmitted during the first distribution operation, such that the node has the parent-child pair; generating a plurality of computer processing-threads on a single processing device of a particular node in the cluster of nodes, such that each computer processing-thread among the plurality of computer processing-threads is assigned to solve a respective reconciliation problem corresponding to a respective parent-child pair stored on the particular node; and reconciling a difference between the parent data-point and the child data-point in each respective parent-child pair at least in part by executing the plurality of computer processing-threads in parallel on the particular node, to collectively generate a reconciled dataset that is distributed among the cluster of nodes. ### Claim 2 (depends on 1) The method of claim 1, wherein reconciling the difference between the parent data-point and the child data-point comprises adjusting a value of the child data-point to be closer to another value of the parent data-point while maintaining an aggregate value of a plurality of adjusted versions of child data-points within a tolerance range. ### Claim 3 (depends on 1) The method of claim 1, wherein the plurality of computer processing-threads are configured to solve their respective reconciliation problems in parallel to one another. ### Claim 4 (depends on 1) The method of claim 1, wherein each node in the cluster of nodes retrieves a respective set of parent data-points from a remote database concurrently with the other nodes in the cluster of nodes by executing a read command in parallel with the other nodes in the cluster of nodes. ### Claim 5 (depends on 1) The method of claim 1, wherein the parent timestamped data and the child timestamped data are predictive forecasts. ### Claim 6 (depends on 1) The method of claim 1, further comprising generating a predictive forecast using the reconciled dataset. ### Claim 7 (depends on 6) The method of claim 6, further comprising: monitoring a machine using the predictive forecast to identify an event associated with the machine; and adjusting a parameter of the machine based on the event. ### Claim 8 (depends on 6) The method of claim 6, further comprising: detecting an anomaly with a machine using the predictive forecast; and based on detecting the anomaly, performing a task configured to reduce a likelihood of the anomaly occurring. ### Claim 9 (depends on 1) The method of claim 1, further comprising: receiving respective portions of the reconciled dataset from various nodes in the cluster of nodes, each of the respective portions of the reconciled dataset corresponding to different timestamps in the parent timestamped data; combining the respective portions of the reconciled dataset to form the reconciled dataset; and storing the reconciled dataset in a database for later retrieval. ### Claim 10 A cluster of nodes for performing distributed processing, the cluster of nodes comprising: one or more processing devices; and one or more memory devices having instructions executable by the one or more processing devices for causing the one or more processing devices to: obtain parent timestamped data and child timestamped data, wherein the child timestamped data is a child of the parent timestamped data in a hierarchical relationship; perform a first distribution operation involving distribute each parent data-point in the parent timestamped data among the cluster of nodes, such that each node in the cluster of nodes is provided with a set of parent data-points that is different from the other nodes in the cluster of nodes; perform a second distribution operation involving, for each parent data-point in the parent timestamped data: identify a child data-point in the child timestamped data that has the same timestamp as the parent data-point, wherein the parent data-point and the child data-point collectively form a parent-child pair; and transmit the child data-point to the same node in the cluster of nodes to which the parent data-point was previously transmitted during the first distribution operation, such that the node has the parent-child pair; generate a plurality of computer processing-threads on a single processing device of a particular node in the cluster of nodes, such that each computer processing-thread among the plurality of computer processing-threads is assigned to solve a respective reconciliation problem corresponding to a respective parent-child pair stored on the particular node; and reconcile a difference between the parent data-point and the child data-point in each respective parent-child pair at least in part by executing the plurality of computer processing-threads in parallel on the particular node, to collectively generate a reconciled dataset that is distributed among the cluster of nodes. ### Claim 11 (depends on 10) The cluster of nodes of claim 10, wherein reconciling the difference between the parent data-point and the child data-point comprises adjusting a value of the child data-point to be closer to another value of the parent data-point while maintaining an aggregate value of a plurality of adjusted versions of child data-points within a tolerance range. ### Claim 12 (depends on 10) The cluster of nodes of claim 10, wherein the plurality of computer processing-threads are configured to solve their respective reconciliation problems in parallel to one another. ### Claim 13 (depends on 10) The cluster of nodes of claim 10, wherein each node in the cluster of nodes is configured to retrieve a respective set of parent data-points from a remote database concurrently with the other nodes in the cluster of nodes by executing a read command in parallel with the other nodes in the cluster of nodes. ### Claim 14 (depends on 10) The cluster of nodes of claim 10, wherein the parent timestamped data and the child timestamped data are predictive forecasts. ### Claim 15 (depends on 10) The cluster of nodes of claim 10, wherein the one or more memory devices further comprises instructions executable by the one or more processing devices for causing the one or more processing devices to generate a predictive forecast using the reconciled dataset. ### Claim 16 (depends on 15) The cluster of nodes of claim 15, wherein the one or more memory devices further comprises instructions executable by the one or more processing devices for causing the one or more processing devices to: monitor a machine using the predictive forecast to identify an event associated with the machine; and adjust a parameter of the machine based on the event. ### Claim 17 (depends on 15) The cluster of nodes of claim 15, wherein the one or more memory devices further comprises instructions executable by the one or more processing devices for causing the one or more processing devices to: detect an anomaly with a machine using the predictive forecast; and based on detecting the anomaly, perform a task configured to reduce a likelihood of the anomaly occurring. ### Claim 18 (depends on 10) The cluster of nodes of claim 10, wherein the one or more memory devices further comprises instructions executable by the one or more processing devices for causing the one or more processing devices to: receive respective portions of the reconciled dataset from various nodes in the cluster of nodes, each of the respective portions of the reconciled dataset corresponding to different timestamps in the parent timestamped data; combine the respective portions of the reconciled dataset to form the reconciled dataset; and store the reconciled dataset in a database for later retrieval. ### Claim 19 A non-transitory computer readable medium comprising program code executable by one or more processing devices of a cluster of nodes in a distributed computing environment for causing the one or more processing devices to: obtain parent timestamped data and child timestamped data, wherein the child timestamped data is a child of the parent timestamped data in a hierarchical relationship; perform a first distribution operation involving distribute each parent data-point in the parent timestamped data among the cluster of nodes, such that each node in the cluster of nodes is provided with a set of parent data-points that is different from the other nodes in the cluster of nodes; perform a second distribution operation involving, for each parent data-point in the parent timestamped data: identify a child data-point in the child timestamped data that has the same timestamp as the parent data-point, wherein the parent data-point and the child data-point collectively form a parent-child pair; and transmit the child data-point to the same node in the cluster of nodes to which the parent data-point was previously transmitted during the first distribution operation, such that the node has the parent-child pair; generate a plurality of computer processing-threads on a single processing device of a particular node in the cluster of nodes, such that each computer processing-thread among the plurality of computer processing-threads is assigned to solve a respective reconciliation problem corresponding to a respective parent-child pair stored on the particular node; and reconcile a difference between the parent data-point and the child data-point in each respective parent-child pair at least in part by executing the plurality of computer processing-threads in parallel on the particular node, to collectively generate a reconciled dataset that is distributed among the cluster of nodes. ### Claim 20 (depends on 19) The non-transitory computer readable medium of claim 19, wherein reconciling the difference between the parent data-point and the child data-point comprises adjusting a value of the child data-point to be closer to another value of the parent data-point while maintaining an aggregate value of a plurality of adjusted versions of child data-points within a tolerance range. ### Claim 21 (depends on 19) The non-transitory computer readable medium of claim 19, wherein the plurality of computer processing-threads are configured to solve their respective reconciliation problems in parallel to one another. ### Claim 22 (depends on 19) The non-transitory computer readable medium of claim 19, wherein each node in the cluster of nodes is configured to retrieve a respective set of parent data-points from a remote database concurrently with the other nodes in the cluster of nodes by executing a read command in parallel with the other nodes in the cluster of nodes. ### Claim 23 (depends on 19) The non-transitory computer readable medium of claim 19, wherein the parent timestamped data and the child timestamped data are predictive forecasts. ### Claim 24 (depends on 19) The non-transitory computer readable medium of claim 19, further comprising instructions executable by the one or more processing devices for causing the one or more processing devices to generate a predictive forecast using reconciled dataset. ### Claim 25 (depends on 24) The non-transitory computer readable medium of claim 24, further comprising instructions executable by the one or more processing devices for causing the one or more processing devices to: monitor a machine using the predictive forecast to identify an event associated with the machine; and adjust a parameter of the machine based on the event. ### Claim 26 (depends on 24) The non-transitory computer readable medium of claim 24, further comprising instructions executable by the one or more processing devices for causing the one or more processing devices to: detect an anomaly with a machine using the predictive forecast; and based on detecting the anomaly, perform a task configured to reduce a likelihood of the anomaly occurring. ### Claim 27 (depends on 19) The non-transitory computer readable medium of claim 19, further comprising instructions executable by the one or more processing devices for causing the one or more processing devices to: receive respective portions of the reconciled dataset from various nodes in the cluster of nodes, each of the respective portions of the reconciled dataset corresponding to different timestamps in the parent timestamped data; combine the respective portions of the reconciled dataset to form the reconciled dataset; and store the reconciled dataset in a database for later retrieval.