User input and interface design in augmented reality for use in surgical settings

A system comprising one or more processors, and a non-transitory computer-readable medium including one or more sequences of instructions that, when executed by the one or more processors, cause the system to perform operations comprising:

• generating, within a unified three-dimensional (3D) coordinate space:
• (i) a virtual 3D medical model positioned according to a current model pose, the current model pose representing a position and orientation of the virtual 3D medical model in the unified 3D coordinate space;
• and
• (ii) at least one a virtual 3D hand representation;
• rendering, via an Augmented Reality (AR) headset device, an AR display that includes display of the virtual 3D medical model positioned according to the current model pose and the virtual 3D hand representation;
• detecting a first physical gesture;
• modifying the AR display, via the AR headset device, by rendering display of a type of movement of the virtual 3D hand representation based on the detected first physical gesture;
• identifying selection of a clipping plane virtual interaction based on the type of movement of the virtual 3D hand representation;
• modifying the AR display, via the AR headset device, by rendering a virtual clipping plane in the AR display concurrently displayed, at a first display position, with the virtual 3D medical model;
• detecting a second physical gesture based on a first change of pose data of the AR headset device, the first change of pose data representing a change of a position and orientation of the AR headset device in the unified 3D coordinate space;
• determining a second display position of the virtual clipping plane based on the first change of pose data of the AR headset device;
• determining a first intersection between the virtual 3D clipping plane at the second display position and the rendered virtual 3D medical model at the current model pose;
• and
• identifying one or more portions of non-rendered 3D medical model data for the virtual 3D medical model in the current model pose that map to the first intersection;
• and
• modifying the AR display, via the AR headset device, by rendering the virtual clipping plane at the second display position concurrently with rendering the virtual 3D medical model portraying the identified one or more portions of 3D medical model data that map to the first intersection.

A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein to be executed by one or more processors, the program code including instructions to:

• generating, within a unified three-dimensional (3D) coordinate space:
• (i) a virtual 3D medical model positioned according to a current model pose, the current model pose representing a position and orientation of the virtual 3D medical model in the unified 3D coordinate space;
• and
• (ii) at least one a virtual 3D hand representation;
• rendering, via an Augmented Reality (AR) headset device, an AR display that includes display of the virtual 3D medical model positioned according to the current model pose and the virtual 3D hand representation;
• detecting a first physical gesture;
• modifying the AR display, via the AR headset device, by rendering display of a type of movement of the virtual 3D hand representation based on the detected first physical gesture;
• identifying selection of a clipping plane virtual interaction based on the type of movement of the virtual 3D hand representation;
• modifying the AR display, via the AR headset device, by rendering a virtual clipping plane in the AR display concurrently displayed, at a first display position, with the virtual 3D medical model;
• detecting a second physical gesture based on a first change of pose data of the AR headset device, the first change of pose data representing a change of a position and orientation of the AR headset device in the unified 3D coordinate space;
• determining a second display position of the virtual clipping plane based on the first change of pose data of the AR headset device;
• determining a first intersection between the virtual 3D clipping plane at the second display position and the rendered virtual 3D medical model at the current model pose;
• and
• identifying one or more portions of non-rendered 3D medical model data for the virtual 3D medical model in the current model pose that map to the first intersection;
• and
• modifying the AR display, via the AR headset device, by rendering the virtual clipping plane at the second display position concurrently with rendering the virtual 3D medical model portraying the identified one or more portions of 3D medical model data that map to the first intersection.

A computer-implemented method, comprising:

• generating, within a unified three-dimensional (3D) coordinate space:
• (i) a virtual 3D container;
• (ii) a virtual 3D medical model positioned according to a current model pose within the virtual 3D container, the current model pose representing a position and orientation of the virtual 3D medical model in the unified 3D coordinate space;
• and
• (iii) at least one a virtual 3D hand representation;
• rendering, via an Augmented Reality (AR) headset device, an AR display that includes display of the virtual 3D container and virtual 3D medical model positioned according to the current model pose and the virtual 3D hand representation;
• detecting a first physical gesture;
• modifying the AR display, via the AR headset device, by rendering display of a type of movement of the virtual 3D hand representation based on the detected first physical gesture;
• identifying selection of a cropping virtual interaction based on the type of movement of the virtual 3D hand representation;
• modifying the AR display, via the AR headset device, by rendering a selectable shape in the AR display concurrently displayed on a first surface of the virtual 3D container, the first surface displayed at a first display position;
• detecting a second physical gesture based on a first change of pose data of the AR headset device, the first change of pose data representing a change of a position and orientation of the AR headset device in the unified 3D coordinate space;
• determining a movement of the selectable shape on the first surface of the virtual 3D container based on the first change of pose data of the AR headset device;
• determining a second display position of the first surface of the virtual 3D container resulting from the movement of the selectable shape;
• determining a first intersection between the first surface of the virtual 3D container at the second display position and the rendered virtual 3D medical model at the current model pose;
• and
• identifying one or more portions of non-rendered 3D medical model data for the virtual 3D medical model in the current model pose that map to the first intersection;
• and
• modifying the AR display, via the AR headset device, by rendering the first surface of the virtual 3D container at the second display position concurrently with rendering the virtual 3D medical model portraying the identified one or more portions of 3D medical model data that map to the first intersection.