# US 12,010,024 B2 · Transparent migration of virtual network functions > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,010,024 B2 - Document ID: US12010024B2 - Status: Granted - Kind: B2 - Application number: 17/304,907 - Filed: 2021-06-28 - Published: 2021-10-21 - Granted: 2024-06-11 - Earliest claimed: 2018-12-13 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: David Schultz; Shuang Guan; Srinivas Krishnan; Eran Gal; Doron Shaharabani; Yossi Richter; Md Ahsan Arefin - Web page: https://patentgrep.com/patent/US12010024B2 ## CPC - H04L 45/22: {Alternate routing} - G06F 9/45558: {Hypervisor-specific management and integration aspects} - H04L 45/306: {Route determination based on the nature of the carried application} - H04L 45/38: {Flow based routing} - H04L 45/42: Centralised routing - H04L 45/7453: using hashing - H04L 47/125: by balancing the load, e.g. traffic engineering - H04L 61/256: NAT traversal - H04L 63/02: {for separating internal from external traffic, e.g. firewalls} - G06F 2009/4557: {Distribution of virtual machine instances; Migration and load balancing} - G06F 2009/45595: {Network integration; Enabling network access in virtual machine instances} - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A method includes identifying a middlebox receiving network flow and communicating with one or more backend virtual machines. The method also includes receiving flow statistics corresponding to the network flow of the middlebox and determining whether the flow statistics satisfy an offload rule. The offload rule indicates when to migrate the network flow from the middlebox to an end host. When the flow statistics satisfy the offload rule, the method also includes migrating the network flow from the middlebox to the end host. ## Claims ### Claim 1 A computer-implemented method when executed by data processing hardware causes the data processing hardware to perform operations comprising: identifying a middlebox receiving network flow between virtual network endpoints in a virtual network layer; performing, using the middlebox, stateful network functions in the virtual network layer using the received network flow between the virtual network endpoints; communicating with one or more backend virtual machines corresponding to the virtual network endpoints; receiving flow statistics corresponding to the network flow of the middlebox; identifying, based on the flow statistics, an uncommon type of network flow from the network flow, the uncommon type of network flow comprising outlier flow statistics of the flow statistics; identifying one or more connections associated with the uncommon type of network flow; migrating the one or more connections associated with the uncommon type of network flow from the middlebox to an end host; after migrating the one or more connections associated with the uncommon type of network flow from the middlebox to the end host, determining whether the flow statistics satisfy an offload rule, the offload rule indicating when to migrate the network flow from the middlebox to an end host; and when the flow statistics satisfy the offload rule, migrating the network flow from the middlebox to the end host. ### Claim 2 (depends on 1) The method of claim 1, wherein the middlebox comprises at least one of a network address translator (NAT), a load balancer, or a firewall. ### Claim 3 (depends on 2) The method of claim 2, wherein the middlebox is associated with a single network device configured to perform network routing. ### Claim 4 (depends on 1) The method of claim 1, wherein migrating the network flow from the middlebox to the end host comprises initiating an end-host connection table at the end host. ### Claim 5 (depends on 1) The method of claim 1, wherein migrating the network flow from the middlebox to the end host comprises: initiating an end-host connection table at the end host; identifying a network connection request received at the end host; determining that the network connection request corresponds to a new network connection; updating the end-host connection table with the new network connection; and controlling the network flow for the new network connection at the end host. ### Claim 6 (depends on 1) The method of claim 1, wherein migrating the network flow from the middlebox to the end host comprises: initiating an end-host connection table at the end host; identifying a network connection request received at the end host; determining that the network connection request corresponds to an existing network connection not present in the end-host connection table; and communicating the network flow for the existing network connection from the end host to the middlebox. ### Claim 7 (depends on 6) The method of claim 6, wherein migrating the network flow from the middlebox further comprises: transferring a middlebox connection table from the middlebox to the end host; and when the middlebox connection table is transferred from the middlebox to the end host, ceasing communication between the end host and the middlebox. ### Claim 8 (depends on 1) The method of claim 1, wherein the operations further comprise, after migrating the network flow from the middlebox to the end host: determining a reduction in the network flow to the end host during a period of time; and migrating the network flow from the end host to the middlebox. ### Claim 9 (depends on 1) The method of claim 1, wherein the middlebox is configured to communicate with the one or more backend virtual machines based on consistent hashing. ### Claim 10 (depends on 1) The method of claim 1, wherein migrating the network flow from the middlebox to the end host further comprises: identifying that a first health characteristic of each backend virtual machine of the one or more backend virtual machines communicating with the middlebox indicates a healthy state; and determining that a second health characteristic corresponding to the end host matches the healthy state of the first health characteristic. ### Claim 11 A system comprising: data processing hardware; and memory hardware in communication with the data processing hardware, the memory hardware storing instruction that when executed on the data processing hardware cause the data processing hardware to perform operations comprising: identifying a middlebox receiving network flow between virtual network endpoints in a virtual network layer; performing, using the middlebox, stateful network functions in the virtual network layer using the received network flow between the virtual network endpoints, and communicating with one or more backend virtual machines corresponding to the virtual network endpoints; receiving flow statistics corresponding to the network flow of the middlebox; identifying, based on the flow statistics, an uncommon type of network flow from the network flow, the uncommon type of network flow comprising outlier flow statistics of the flow statistics; identifying one or more connections associated with the uncommon type of network flow; migrating the one or more connections associated with the uncommon type of network flow from the middlebox to an end host; after migrating the one or more connections associated with the uncommon type of network flow from the middlebox to the end host, determining whether the flow statistics satisfy an offload rule, the offload rule indicating when to migrate the network flow from the middlebox to the end host; and when the flow statistics satisfy the offload rule, migrating the network flow from the middlebox to the end host. ### Claim 12 (depends on 11) The system of claim 11, wherein the middlebox comprises at least one of a network address translator (NAT), a load balancer, or a firewall. ### Claim 13 (depends on 12) The system of claim 12, wherein the middlebox is associated with a single network device configured to perform network routing. ### Claim 14 (depends on 11) The system of claim 11, wherein migrating the network flow from the middlebox to the end host comprises initiating an end-host connection table at the end host. ### Claim 15 (depends on 11) The system of claim 11, wherein migrating the network flow from the middlebox to the end host comprises: initiating an end-host connection table at the end host; identifying a network connection request received at the end host; determining that the network connection request corresponds to a new network connection; updating the end-host connection table with the new network connection; and controlling the network flow for the new network connection at the end host. ### Claim 16 (depends on 11) The system of claim 11, wherein migrating the network flow from the middlebox to the end host comprises: initiating an end-host connection table at the end host; identifying a network connection request received at the end host; determining that the network connection request corresponds to an existing network connection not present in the end-host connection table; and communicating the network flow for the existing network connection from the end host to the middlebox. ### Claim 17 (depends on 16) The system of claim 16, wherein migrating the network flow from the middlebox further comprises: transferring a middlebox connection table from the middlebox to the end host; and when the middlebox connection table is transferred from the middlebox to the end host, ceasing communication between the end host and the middlebox. ### Claim 18 (depends on 11) The system of claim 11, wherein the operations further comprise, after migrating the network flow from the middlebox to the end host: determining, a reduction in the network flow to the end host during a period of time; and migrating the network flow from the end host to the middlebox. ### Claim 19 (depends on 11) The system of claim 11, wherein the middlebox is configured to communicate with the one or more backend virtual machines based on consistent hashing. ### Claim 20 (depends on 11) The system of claim 11, wherein migrating the network flow from the middlebox to the end host further comprises: identifying that a first health characteristic of each backend virtual machine of the one or more backend virtual machines communicating with the middlebox indicates a healthy state; and determining that a second health characteristic corresponding to the end host matches the healthy state of the first health characteristic.