Marker-less Augmented Reality System for Mammoplasty Pre-Visualization

A method of generating a medical image relative to a present patient, comprising:

• obtaining a first plurality of two-dimensional (2D) images, wherein the first plurality of 2D images comprises images of prior patients before respective surgical procedures;
• generating a first plurality of three-dimensional (3D) models from the first plurality of 2D images;
• obtaining a second plurality of 2D images, wherein the second plurality of 2D images comprises images of the prior patients after their respective surgical procedures;
• generating a second plurality of 3D models from the second plurality of 2D images;
• training a convolutional neural network, wherein a training image for the convolutional neural network is an input 3D model from the first plurality of 3D models and a first ground truth for the input 3D model is a post-surgery 3D model from the second plurality of 3D models, wherein the input 3D model and the post-surgery 3D model are derived from the same prior patient of the prior patients;
• obtaining at least one 2D image of the present patient;
• generating at least one 3D model of the present patient from the at least one 2D image of the present patient;
• applying the at least one 3D model of the present patient as a first input to the convolutional neural network in an inference stage;
• obtaining patient-specific parameters corresponding to a proposed surgical procedure;
• applying the patient-specific parameters derived from the proposed surgical procedure as a second input to the convolutional neural network in the inference stage;
• and
• outputting, from the convolutional neural network, an inferred post-surgery 3D model of the present patient given the patient-specific parameters.

A method of generating a modified view image, of a user body part of a user, from a user input image depicting the user body part, the method comprising:

• generating a first set of images comprising a first plurality of two-dimensional images from the first set of images;
• generating a first set of semantic landmarks in images of the first set of images;
• training a first convolutional neural network (CNN) with inputs of the first set of images with a first ground truth of the first set of semantic landmarks, wherein training is to train the first CNN to form a first trained CNN to output one or more output semantic landmarks related to an input image without requiring a ground truth of the input image;
• obtaining the user input image depicting the user body part;
• generating, using the first trained CNN with the user input image as a CNN input, a second set of semantic landmarks representing landmarks of the user body part;
• obtaining a mesh model for the first set of images, wherein the mesh model is data structure that corresponds to a sampling of a three-dimensional model of instances of sampled body parts of persons other than the user;
• generating, from the mesh model and the second set of semantic landmarks, a user body part mesh, wherein the user body part mesh is an approximation of a three-dimensional model of the user body part depicted in the user input image;
• obtaining a target body part mesh data structure, distinct from the user body part mesh;
• and
• generating the modified view image of the user body part, modified to reflect differences between the target body part mesh data structure and the user body part mesh while retaining at least some texture of the user body part from the user input image.