Interacting with machines using natural language input and an enhanced state graph

An apparatus, comprising:

• a memory to store a binary code of a controller;
• and
• a processing device, operatively coupled to the memory, to:
• obtain a state graph, wherein:
• the state graph represents a set of objects within an environment and a set of positions of the set of objects within the environment;
• the state graph comprises a set of object nodes to represent the set of objects, the set of object nodes being connected by edges indicating a type of an object node;
• and
• the state graph comprises a set of property nodes to represent a set of properties of the set of objects;
• and
• obtain user input data, wherein the user input data is generated based on a natural language input;
• and
• update the state graph based on the user input data to generate an enhanced state graph, wherein the enhanced state graph comprises the set of object nodes, the set of property nodes, additional nodes generated based on the user input data, and updated edges connecting the additional nodes, the set of object nodes, and the set of property nodes;
• provide the enhanced state graph to a planning module configured to generate instructions for operating a mechanical system based on the enhanced state graph.

A non-transitory computer-readable storage medium having instructions stored thereon that, when executed by a processing device, cause the processing device to:

• obtain a state graph, wherein:
• the state graph represents a set of objects within an environment and a set of positions of the set of objects within the environment;
• the state graph comprises a set of object nodes to represent the set of objects, the set of object nodes being connected by edges indicating a type of an object node;
• and
• the state graph comprises a set of property nodes to represent a set of properties of the set of objects;
• and
• obtain user input data, wherein the user input data is generated based on a natural language input;
• and
• update the state graph based on the user input data to generate an enhanced state graph, wherein the enhanced state graph comprises the set of object nodes, the set of property nodes, additional nodes generated based on the user input data, and updated edges connecting the additional nodes. the set of object nodes, and the set of property nodes;
• provide the enhanced state graph to a planning module configured to generate instructions for operating a mechanical system based on the enhanced state graph.