Patch-based scene segmentation using neural networks

An image segmentation system comprising one or more processors, one or more non-transitory digital storage media coupled to the processors and storing one or more sequences of program instructions which, when executed using the one or more processors, cause the one or more processors to perform:

• using one or more computing devices, receiving a digital image comprising test image;
• using the one or more computing devices, creating, based on the test image, a plurality of grid patches;
• using the one or more computing devices, receiving a plurality of classifiers that have been trained to identify one or more materials of a plurality of materials;
• for each patch of the plurality of grid patches, before each pixel of the patch, of the plurality of grid patches, is labeled with a label:
• determining whether a pixel-based-size of the patch matches a pixel-based-size of a classifier of the plurality of classifiers;
• in response to determining that the pixel-based-size of the patch does not match the pixel-based-size of the classifier in the plurality of classifiers,
• generating a resized patch by either downsizing or upsizing the patch, and replacing the patch with the resized patch in the plurality of grid patches;
• applying the classifier or a different classifier, from the plurality of classifiers, to the patch to determine whether labels can be assigned to pixels of the patch;
• in response to determining that one or more pixels of the patch do not have assigned labels, repeating resizing of the patch until all pixels of the patch have assigned labels;
• using the one or more computing devices, generating, based on labels assigned to pixels of the plurality of grid patches, a grid of labels for the test image;
• based on, at least in part, the grid of labels, generating a segmentation mask for the test image;
• based on, at least in part, the segmentation mask, determining a set of materials, of the plurality of materials, that is depicted in the test image;
• based on, at least in part, the set of materials that has been determined, determining a work status of activities taking place on a site for which the test image was received, the site comprising one or more of a construction site, an industrial site, a warehouse site, a storage site, a commercial site or a residential site.

One or more non-transitory digital storage media storing one or more sequences of program instructions which, when executed using one or more processors, cause the one or more processors to perform:

• using one or more computing devices, receiving a digital image comprising test image;
• using the one or more computing devices, creating, based on the test image, a plurality of grid patches;
• using the one or more computing devices, receiving a plurality of classifiers that have been trained to identify one or more materials of a plurality of materials;
• for each patch of the plurality of grid patches, before each pixel of the patch, of the plurality of grid patches, is labeled with a label:
• determining whether a pixel-based-size of the patch matches a pixel-based-size of a classifier of the plurality of classifiers;
• in response to determining that the pixel-based-size of the patch does not match the pixel-based-size of the classifier in the plurality of classifiers,
• generating a resized patch by either downsizing or upsizing the patch, and replacing the patch with the resized patch in the plurality of grid patches;
• applying the classifier or a different classifier, from the plurality of classifiers, to the patch to determine whether labels can be assigned to pixels of the patch;
• in response to determining that one or more pixels of the patch do not have assigned labels, repeating resizing of the patch until all pixels of the patch have assigned labels;
• using the one or more computing devices, generating, based on labels assigned to pixels of the plurality of grid patches, a grid of labels for the test image;
• based on, at least in part, the grid of labels, generating a segmentation mask for the test image;
• based on, at least in part, the segmentation mask, determining a set of materials, of the plurality of materials, that is depicted in the test image;
• based on, at least in part, the set of materials that has been determined, determining a work status of activities taking place on a site for which the test image was received, the site comprising one or more of a construction site, an industrial site, a warehouse site, a storage site, a commercial site or a residential site.