Merchandise hierarchy refinement by incorporation of product correlation

A system for of merchandise hierarchy refinement comprising:

• a memory storage device;
• a processor unit in communication with said memory storage device, said processor unit configured to perform a method comprising:
• extracting first data from a predetermined merchandise hierarchy represented as a tree data structure of nodes interconnected by links wherein one or more products at a lowest level node of said hierarchy are assigned as being a member of a category at an upper category level node of said hierarchy and include a link to that category level node, and second data representing transaction records having a plurality of transactions related to the plurality of products;
• clustering said plurality of products based on said plurality of transactions in which a new cluster is generated relating to products of said plurality;
• and
• updating the predetermined merchandise hierarchy representation based on said clustering in which said products of said new cluster are assigned as being members of a new category level node, and a new link generated for connecting the products and corresponding new category level node, wherein said clustering comprises:
• initializing a new membership matrix as representing categories and their member product assignments according to a lowest level of said predefined merchandise hierarchy;
• performing, via a computer simulation, an iterative process to minimize an objective function relating a similarity of complementary information between said plurality of products, wherein at each iteration, assigning each product to a cluster label such that similar items have similar assignments to update said new membership matrix and form a refined merchandise hierarchy that satisfies a consistency constraint based on a ratio of mutual information representing a difference between the predefined merchandise hierarchy and the refined merchandise hierarchy;
• repeating said initializing and performing at a next upper level until a next highest level of said refined merchandise hierarchy is reached;
• and
• outputting said new membership matrix.