Generating a semantic construction of a physical setting

A device comprising:

• one or more processors;
• a non-transitory memory;
• one or more displays;
• and
• one or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the device to:
• obtain environmental data corresponding to a physical environment;
• determine, based on the environmental data, a bounding surface of the physical environment;
• detect a physical element located within the physical environment based on the environmental data;
• determine a semantic label for the physical element based on at least a portion of the environmental data corresponding to the physical element;
• generate a semantic construction of the physical environment based on the environmental data, wherein the semantic construction of the physical environment includes a representation of the bounding surface, a representation of the physical element and the semantic label for the physical element;
• and
• provide the semantic construction of the physical environment to a virtual intelligent agent (VIA) that generates actions for a graphical object that represents the VIA.

A non-transitory memory storing one or more programs, which, when executed by one or more processors of a device with a display, cause the device to:

• obtain environmental data corresponding to a physical environment;
• determine, based on the environmental data, a bounding surface of the physical environment;
• detect a physical element located within the physical environment based on the environmental data;
• determine a semantic label for the physical element based on at least a portion of the environmental data corresponding to the physical element;
• generate a semantic construction of the physical environment based on the environmental data, wherein the semantic construction of the physical environment includes a representation of the bounding surface, a representation of the physical element and the semantic label for the physical element;
• and
• provide the semantic construction of the physical environment to an objective-effectuator engine that generates actions fora graphical object representing an objective-effectuator that is instantiated in a graphical setting.