System, Method, and Computer Program Product for Streaming Data Mining with Text-Image Joint Embeddings

A system, comprising:

• a memory;
• and
• at least one processor coupled to the memory and configured to:
• obtain a roadway dataset, wherein each record in the roadway dataset includes an image logged by an autonomous vehicle (AV) in a roadway;
• generate, based on a text-image model, an embedding dataset comprising an image embedding for each record in the roadway dataset, each image embedding identifying a point in a high dimension space;
• determine a parametric description of a high probability field which surrounds a query input text in the high dimension space;
• determine at least one input image of an input stream of images that is in the high probability field;
• and
• automatically label the at least one input image based on the query input text.

A non-transitory computer-readable medium having instructions stored thereon that, when executed by at least one computing device comprising one or more processors, cause the at least one computing device to:

• obtain a roadway dataset, wherein each record in the roadway dataset includes an image logged by an autonomous vehicle (AV) in a roadway;
• generate, based on a text-image model, an embedding dataset comprising an image embedding for each record in the roadway dataset, each image embedding identifying a point in a high dimension space;
• determine a parametric description of a high probability field which surrounds a query input text in the high dimension space;
• determine at least one input image of an input stream of images that is in the high probability field;
• and
• automatically label the at least one input image based on the query input text.