Carpet detection method, movement control method, and mobile machine using the same

A method for controlling a mobile machine having an RGB-D camera to move, comprising:

• at least one or more processors of the mobile machine,
• obtaining, through the RGB-D camera, one or more RGB-D image pairs, wherein each RGB-D image pair comprises an RGB image and a depth image, and the depth image comprises data for representing a distance to scene objects in the depth image;
• detecting one or more carpet areas in the RGB image of the one or more RGB-D image pairs and generating a first 2D hounding box to mark each of the one or more carpet areas using a first deep learning model based on a first data set containing a carpet class, wherein the first data set comprises a plurality of images of carpets in various scenes;
• detecting one or more carpet-curl areas in the RGB image of the one or more RGB-D image pairs and generating a second 2D bounding box to mark each of the one or more carpet-curl areas using a second deep learning model based on a second data set containing a carpet-curl class, wherein the second data set comprises a plurality of images of carpet-curls in various scenes;
• generating a group of carpet points corresponding to each of the one or more carpet areas in the RGB image of each of the one or more RGB-D image pairs by matching each pixel of the RGB image within the first 2D bounding box corresponding to the carpet area to each pixel in the depth image of the RGB-D image pair;
• generating a group of carpet-curl points corresponding to each of the one or more carpet-curl areas in the RGB image of each of the one or more RGB-D image pairs by matching each pixel of the RGB image within the second 2D bounding box corresponding to the carpet-curl area to each pixel in the depth image of the RGB-D image pair;
• taking the generated group of carpet points corresponding to each of the one or more carpet areas in the RGB image of each of the one or more RGB-D image pairs as a carpet;
• obtaining one or more carpet-curls each corresponding to each of the one or more carpet-curl areas in the RGB image of each of the one or more RGB-D image pairs based on the generated group of carpet-curl points corresponding to each of the one or more carpet-curl areas, and taking each of the one or more carpet-curls as an obstacle;
• and
• controlling the mobile machine to make a movement with reference to the carpet corresponding to each of the one or more carpet areas in the RGB image of each of the one or more RGB-D image pairs and avoid the obstacle.

A mobile machine, comprising:

• an RGB-D camera;
• one or more processors;
• and
• memory storing one or more programs configured to be executed by the one or more processors, wherein the one or more programs include instructions to:
• obtain, through an RGB-D camera, one or more RGB-D image pairs, wherein each RGB-D image pair comprises an RGB image and a depth image;
• detect one or more carpet areas in the RGB image of the one or more RGB-D image pairs and generate a first 2D bounding box to mark each of the one or more carpet areas using a first deep learning model based on a first data set containing a carpet class, wherein the first data set comprises a plurality of images of carpets in various scenes;
• detect one or more carpet-curl areas in the RGB image of the one or more RGB-D image pairs and generate a second 2D bounding box to mark each of the one or more carpet-curl areas using a second deep learning model based on a second data set containing a carpet-curl class, wherein the second data set comprises a plurality of images of carpet-curls in various scenes;
• generate a group of carpet points corresponding to each of the one or more carpet areas in the RGB image of each of the one or more RGB-D image pairs by matching each pixel of the RGB image within the first 2D bounding box corresponding to the carpet area to each pixel in the depth image of the RGB-D image pair;
• generate a group of carpet-curl points corresponding to each of the one or more carpet-curl areas in the RGB image of each of the one or more RGB-D image pairs by matching each pixel of the RGB image within the second 2D bounding box corresponding to the carpet-curl area to each pixel in the depth image of the RGB-D image pair;
• and
• control the mobile machine to move and avoid one or more carpet-curls, according to the generated group of carpet points corresponding to each of the one or more carpet areas in the RGB image of each of the one or more RGB-D image pairs, and the generated group of carpet-curl points corresponding to each of the one or more carpet-curl areas in the RGB image of each of the one or more RGB-D image pairs.