# US 12,771,743 B2 · Data exchange and processing synchronization in distributed systems > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,771,743 B2 - Document ID: US12771743B2 - Status: Granted - Kind: B2 - Application number: 18/311,571 - Filed: 2023-05-03 - Published: 2023-08-31 - Granted: 2026-10-20 - Earliest claimed: 2019-12-18 - Assignees: Hitachi Energy Ltd - Applicants: Hitachi Energy Ltd - Inventors: Mehmet H. Cintuglu; Dmitry Ishchenko - Web page: https://patentgrep.com/patent/US12771743B2 ## CPC - H04L 67/1095: Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes - G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)} - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Systems, methods, techniques and apparatuses of asynchronous communication is distributed systems are disclosed. One exemplary embodiment is a method determining, with a plurality of agent nodes structured to communicate asynchronously in a distributed system, a first set of iterations including an iteration determined by each of the plurality of agent nodes; determining, with a first agent node of the plurality of agent nodes, a local vector clock; receiving, with the first agent node, a first iteration of the first set of iterations and a remote vector clock determined based on the first iteration; updating, with the first agent node, the local vector clock based on the received remote vector clock; and determining a first iteration of a second set of iterations based on the first set of iterations after determining all iterations of the first set of iterations have been received based on the local vector clock. ## Claims ### Claim 1 A method comprising: operating a distributed system including a first agent node and a plurality of agent nodes structured to communicate asynchronously; determining, with each agent node of the plurality of agent nodes, a plurality of iterations of a distributed iterative algorithm separated in time by a plurality of ticker periods, wherein each ticker period comprises a predetermined time interval separating consecutive iterations; determining, with the first agent node, a trust matrix including a plurality of trust factors, each trust factor corresponding to a weight applied to iterations of the plurality of iterations determined by one of the plurality of agent nodes in the distributed iterative algorithm; determining, with the first agent node, a first iteration of the distributed iterative algorithm; receiving, with the first agent node, a portion of the plurality of iterations determined by a portion of the plurality of agent nodes during a first ticker period beginning after the first iteration; updating, with the first agent node, the trust matrix based on the portion of the plurality of iterations; and determining, with the first agent node, a second iteration after the first ticker period based on the portion of the plurality of iterations and the updated trust matrix, wherein: the first iteration, the second iteration, and the plurality of iterations are state estimations; the plurality of agent nodes are microgrid controllers; a final iteration is a final state estimation; and the method further comprises: determining, with the first agent node, a series of iterations including the first iteration and the second iteration until the first agent node determines the final iteration; and operating, with the first agent node, a controllable device of a microgrid based on the final iteration. ### Claim 2 (depends on 1) The method of claim 1, further comprising determining, with the first agent node, a second plurality of iterations including the first iteration and the second iteration, the plurality of iterations being separated in time by a second plurality of ticker periods, wherein the second plurality of ticker periods are each equal in length of time. ### Claim 3 (depends on 2) The method of claim 2, wherein a portion of the ticker periods of the plurality of ticker periods are a length in time different than the length of time of the ticker periods of the second plurality of ticker periods. ### Claim 4 (depends on 1) The method of claim 1, wherein updating the trust matrix based on the portion of the plurality of iterations received during the first ticker period includes decreasing any weight applied to iterations received from a remainder of the plurality of agent nodes that did not transmit an iteration to the first agent node during the first ticker period. ### Claim 5 (depends on 4) The method of claim 4, wherein updating the trust matrix based on the portion of the plurality of iterations received during the first ticker period includes increasing the weight applied to the iterations determined by the first agent node in an amount equal to the amount any weights were decreased. ### Claim 6 (depends on 1) The method of claim 1, further comprising determining a step size based on the portion of the plurality of iterations received during the first ticker period, wherein determining the second iteration after the first ticker period is based in part on the determined step size. ### Claim 7 (depends on 1) The method of claim 1, wherein the first iteration, the second iteration, and the plurality of iterations are determined by the plurality of agent nodes based on recursive least squares. ### Claim 8 A distributed system comprising: a first agent node; a plurality of agent nodes; and a communication network structured to allow the first agent node and the plurality of agent nodes to communicate asynchronously; wherein the plurality of agent nodes each include a first set of instructions stored on a first memory device which, when executed by a first processing device of the corresponding agent node, is effective to determine a plurality of iterations of a distributed iterative algorithm separated in time by a plurality of ticker periods, wherein each ticker period comprises a predetermined time interval separating consecutive iterations, wherein the first agent node is structured to execute a second set of instructions stored on a second memory device of the first agent node with a second processing device of the first agent node effective to: determine a trust matrix including a plurality of trust factors, each trust factor corresponding to a weight applied to iterations of the plurality of iterations determined by one of the plurality of agent nodes in the distributed iterative algorithm, determine a first iteration of the distributed iterative algorithm, receive a portion of the plurality of iterations determined by a portion of the plurality of agent nodes during a first ticker period beginning after the first iteration, update the trust matrix based on the portion of the plurality of iterations, and determine a second iteration after the first ticker period based on the portion of the plurality of iterations and the updated trust matrix, wherein: the first iteration, the second iteration, and the plurality of iterations are state estimations; the plurality of agent nodes are microgrid controllers; a final iteration is a final state estimation; and the second set of instructions is further effective to: determine a series of iterations including the first iteration and the second iteration until the first agent node determines the final iteration; and operate a controllable device of a microgrid based on the final iteration. ### Claim 9 A method comprising: operating a distributed system including a first agent node and a plurality of agent nodes structured to communicate asynchronously; determining, with each agent node of the plurality of agent nodes, a plurality of iterations of a distributed iterative algorithm separated in time by a plurality of ticker periods, wherein each ticker period comprises a predetermined time interval separating consecutive iterations; determining, with the first agent node, a trust matrix including a plurality of trust factors, each trust factor corresponding to a weight applied to iterations of the plurality of iterations determined by one of the plurality of agent nodes in the distributed iterative algorithm; determining, with the first agent node, a first iteration of the distributed iterative algorithm; receiving, with the first agent node, a portion of the plurality of iterations determined by a portion of the plurality of agent nodes during a first ticker period beginning after the first iteration; updating, with the first agent node, the trust matrix based on the portion of the plurality of iterations; and determining, with the first agent node, a second iteration after the first ticker period based on the portion of the plurality of iterations and the updated trust matrix, wherein: the first iteration, the second iteration, and the plurality of iterations are state estimations; a final iteration is a final state estimation; the method further comprises: determining, with the first agent node, a series of iterations including the first iteration and the second iteration until the first agent node determines the final iteration wherein the plurality of agent nodes are microgrid controllers and operating, with the first agent node, a controllable device of a microgrid based on the final iteration. ### Claim 10 (depends on 9) The method of claim 9, wherein the first iteration, the second iteration, and the plurality of iterations are determined by the plurality of agent nodes based on recursive least squares. ### Claim 11 (depends on 10) The method of claim 10, wherein updating the trust matrix based on the portion of the plurality of iterations received during the first ticker period includes increasing the weight applied to iterations determined by the first agent node in an amount equal to the amount any weights were decreased. ### Claim 12 (depends on 9) The method of claim 9, further comprising determining a step size based on the portion of the plurality of iterations received during the first ticker period, wherein determining the second iteration after the first ticker period is based in part on the determined step size. ### Claim 13 (depends on 9) The method of claim 9, further comprising repeatedly determining additional iterations until the plurality of agent nodes determines that the iterations have converged. ### Claim 14 (depends on 9) The method of claim 9, wherein determining the second iteration comprises computing a weighted combination of the portion of the plurality of iterations received during the first ticker period, wherein the trust factors of the trust matrix are used as weights in the weighted combination. ### Claim 15 (depends on 9) The method of claim 9, wherein updating the trust matrix based on the portion of the plurality of iterations comprises: decreasing the trust factors corresponding to agent nodes that did not transmit an iteration during the first ticker period; and increasing the trust factor corresponding to the first agent node by an amount equal to a sum of the decreased trust factors. ### Claim 16 (depends on 9) The method of claim 9, wherein different agent nodes of the plurality of agent nodes determine corresponding iterations at different times such that the distributed iterative algorithm executes asynchronously.