Relational model based natural language querying to identify object relationships in scene

A system comprising:

• a memory comprising instructions;
• and
• one or more processors configured to execute the instructions to:
• receive, from a client device, a search query for content, wherein the search query specifies one or more objects and one or more spatial relationships between the one or more objects;
• segment the search query into two or more elements;
• generate a query vector using a computer-operated neural language model;
• compare the query vector to an indexed vector for each of one or more spatial relationships between one or more objects of an image;
• determine a listing of relational images from a collection of images based on a comparison between the query vector and the indexed vector;
• determine a ranking for each image in the listing of relational images;
• and
• provide a set of search results responsive to the search query to the client device, the set of search results including a prioritized listing of multiple relational images based on the ranking, wherein the one or more spatial relationships between the one or more objects comprise a relative spatial position between an object pair selected from the one or more objects in the image, and wherein the indexed vector includes a relational value indicative of the relative spatial position between the object pair in the image.

A non-transitory machine-readable storage medium comprising machine-readable instructions for causing a processor to execute a method, comprising:

• receiving, from a client device, a search query for content, wherein the search query specifies one or more objects and one or more spatial relationships between the one or more objects;
• segmenting the search query into two or more elements;
• generating, based on a segmented search query element, a query vector using a computer-operated neural language model;
• comparing the query vector to an indexed vector for each of one or more spatial relationships between one or more objects of an image;
• determining a listing of relational images from a collection of images based on the comparing;
• determining a ranking for each image in the listing of relational images;
• and
• providing a set of search results responsive to the search query to the client device, the set of search results including a prioritized listing of the relational images based on the determined ranking, wherein the one or more spatial relationships between the one or more objects comprise a relative spatial position between an object pair selected from the one or more objects in the image, and wherein the indexed vector includes a relational value indicative of the relative spatial position between the object pair in the image.