Method and apparatus for monitoring and maintaining health in a searchable data service

A method, comprising:

• processing query requests received from client applications by a searchable data service implemented on a plurality of computer devices, wherein the searchable data service is configured to locate searchable data service objects in respective searchable indexes that satisfy the query requests, wherein each searchable data service object specifies two or more attributes of a particular entity in a articular one of a plurality of independent data stores on one or more storage devices on a network and separate from the plurality of computer devices that implement the searchable data service, wherein each searchable index stores searchable data service objects for a particular one of the plurality of independent data stores, and wherein the attributes include a unique entity identifier for locating the particular entity in the particular data store;
• wherein the searchable data service comprises a plurality of nodes implemented on the plurality of computer devices, wherein the plurality of nodes are organized into one or more node groups each including a subset of the plurality of nodes, wherein each node in the node group comprises an instance of a group health component of the searchable data service;
• on each node in a node group, the group health component on the node:
• sending group health messages each including collected local health information for the node to at least one other node in the node group;
• and
• receiving group health messages originating from one or more others of the nodes in the node group, wherein each group health message includes local health information for the originating node.

A computer-accessible memory medium storing program instructions, wherein the program instructions are computer-executable to implement a searchable data service on a plurality of nodes, wherein the searchable data service is configured to:

• process query requests received from client applications by a searchable data service implemented on a plurality of computer devices, wherein the searchable data service is configured to locate searchable data service objects in respective searchable indexes that satisfy the query requests, wherein each searchable data service object specifies two or more attributes of a particular entity in a particular one of a plurality of independent data stores on one or more storage devices on a network and separate from the plurality of computer devices that implement the searchable data service, wherein each searchable index stores searchable data service objects for a particular one of the plurality of independent data stores, and wherein the attributes include a unique entity identifier for locating the particular entity in the particular data store;
• wherein the searchable data service comprises a plurality of nodes implemented on the plurality of computer devices, wherein the plurality of nodes are organized into one or more node groups each including a subset of the plurality of nodes, wherein each node in the node group comprises an instance of a group health component of the searchable data service;
• wherein, on each node in a node group, the group health component of the searchable data service is configured to:
• send group health messages each including collected local health information for the node to at least one other node in the node group;
• and
• receive group health messages originating from one or more others of the nodes in the node group, wherein each group health message includes local health information for the originating node.

A searchable data service node, comprising:

• one or more processors;
• a network interface configured to communicatively couple to a network;
• and
• a memory comprising program instructions, wherein the program instructions are executable by the one or more processors to:
• persistently store a partition of a particular one of a plurality of searchable indexes maintained by a searchable data service implemented on a plurality of computer devices, wherein the searchable data service node is implemented on one of the computer devices, wherein the particular searchable index corresponds to a particular one of a plurality of independent data stores on one or more storage devices on the network and separate from the plurality of computer devices that implement the searchable data service, and wherein the partition is configured to store a set of searchable data service objects stored in the particular searchable index;
• process query requests received from client applications via the network to locate searchable data service objects that satisfy the query requests in the particular searchable index, wherein each searchable data service object specifies two or more attributes of a particular entity in the data store corresponding to the particular searchable index, and wherein the attributes include a unique entity identifier for locating the particular entity in the corresponding data store;
• send group health messages each including collected local health information for the searchable data service node to another searchable data service node implemented on another one of the computer devices via the network;
• and
• receive group health messages via the network and originating from the other searchable data service node, wherein each received group health message includes local health information for the other searchable data service node.