Interactive Computing System with Environmental Recognition and Contextual Output

A method for providing multimodal interactive feedback within an augmented interaction space, the method comprising:

• projecting, by one or more projection devices supported by a positioning structure at a first position relative to a surface, visual content onto the surface;
• capturing, by one or more image capture devices supported by the positioning structure and oriented to capture image data of the surface and a region above the surface, the image data of an augmented interaction space, the augmented interaction space defined by an overlapping spatial region in which (i) a field of view of the one or more image capture devices and (ii) a projection area of the one or more projection devices coincide over the surface, the augmented interaction space including both physical objects disposed on or above the surface and projected visual content displayed on the surface by the one or more projection devices;
• obtaining, by one or more processors, a contextual state for an activity being performed within the augmented interaction space, the contextual state based at least in part on one or more physical objects within the augmented interaction space;
• generating, by the one or more processors, based on the contextual state, one or more response outputs including at least a projected visual overlay;
• and
• causing, by the one or more processors, the one or more projection devices to display the projected visual overlay within the augmented interaction space.

A computing system for providing contextual feedback within an augmented interaction space, comprising:

• one or more processors;
• and
• one or more memories coupled to the one or more processors, the one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to:
• establish an augmented interaction space over a surface by causing one or more projection devices, supported by a positioning structure at a first position relative to the surface, to project visual content onto the surface, the augmented interaction space defined by an overlapping spatial region in which (i) a field of view of one or more image capture devices supported by the positioning structure and oriented to capture image data of the surface and a region above the surface and (ii) a projection area of the one or more projection devices coincide over the surface, the augmented interaction space including both physical objects disposed on or above the surface and projected visual content displayed on the surface by the one or more projection devices;
• receive, from the one or more image capture devices, a stream of image data of the augmented interaction space;
• detect, based on the stream of image data, an event within the augmented interaction space, the event including at least a change in position of a physical object on or above the surface;
• obtain, based on at least the detected event, a contextual state update for an activity being performed within the augmented interaction space, the contextual state update based at least in part on the one or more physical objects within the augmented interaction space;
• and
• cause, in response to the contextual state update, the one or more projection devices to modify the projected visual content within the augmented interaction space to reflect the contextual state update.