Semantic rearrangement of unknown objects from natural language commands

A system for rearranging objects in a scene, the system comprising:

• a memory storing a point cloud and a natural language instruction;
• and
• at least one processor configured to:
• generate, via a first network, an instance mask based on the natural language instruction, wherein the instance mask identifies a set of query objects in the point cloud;
• generate, via a second network, samples of pose offsets for the set of query objects in accordance with the natural language instruction;
• and
• identify, via a discriminator network, a selected rearrangement of the set of query objects in accordance with scores.

A non-transitory computer readable medium storing instructions that, responsive to being executed by at least one processor, cause a system to:

• receive a point cloud and a natural language instruction;
• generate, via a first network, an instance mask based on the natural language instruction, wherein the instance mask identifies a set of query objects in the point cloud;
• generate, via a second network, samples of pose offsets for the set of query objects in accordance with the natural language instruction;
• and
• identify, via a discriminator network, a selected rearrangement of the set of query objects in accordance with scores.