Deep multi-modal pairwise ranking model for crowdsourced food data

A health tracking system comprising:

• a database configured to store a plurality of data records, each of the plurality of data records comprising at least a descriptive string and nutritional data regarding a respective consumable item;
• and
• a computer processor in communication with the database, the computer processor being configured to:
• receive a query string;
• retrieve from the database a first data record of the plurality of data records and a second data record of the plurality of data records based on the query string;
• generate (i) a first feature vector based on the descriptive string of the first data record, (ii) a second feature vector based on the descriptive string of the second data record, and (iii) a third feature vector based on the query string, using at least one first embedding function of a machine learning model, the at least one first embedding function being learned in a training process of the machine learning model;
• generate (i) a first nutrition information vector from the nutritional data of the first data record and (ii) a second nutrition information vector from the nutritional data of the second data record;
• generate a third nutrition information vector based on the query string, using a second embedding function of the machine learning model, the second embedding function being learned in the training process of the machine learning model;
• determining which of the first data record and the second data record is more relevant to the query string based on the first feature vector, the second feature vector, the third feature vector, first nutrition information vector, the second nutrition information vector, and the third nutrition information vector;
• and
• transmit a list of data records of the plurality of data records to an electronic device of a user of the health tracking system, the list of data records at least including the first data record and the second data record, a relative sorting of the first data record and the second data record in the list of data records depending on the determination of which of the first data record and the second data record is more relevant to the query string.

A method of operating a health tracking system to train a machine learning model, the method comprising:

• receiving, with a processor of the health tracking system, a plurality of training inputs, each training input including (i) a query string, (ii) a first descriptive string and first nutritional data labeled as corresponding to a correct output, and (iii) a second descriptive string and second nutritional data labeled as corresponding to an incorrect output;
• and
• for each training input:
• generating, with the processor, (i) a first feature vector based on the first descriptive string, (ii) a second feature vector based on the second descriptive string, and (iii) a third feature vector based on the query string, using at least one first embedding function of the machine learning model;
• determining, with the processor, (i) a first nutrition information vector from the first nutritional data and (ii) a second nutrition information vector from the second nutritional data;
• generating, with the processor, a third nutrition information vector based on the query string, using a second embedding function of the machine learning model;
• determining, with the processor, a hinge loss based on the first feature vector, the second feature vector, the third feature vector, first nutrition information vector, the second nutrition information vector, and the third nutrition information vector;
• and
• adjusting, with the processor, parameter values of the machine learning model based on the hinge loss.