# US 12,001,216 B2 · Carpet detection method, movement control method, and mobile machine using the same > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,001,216 B2 - Document ID: US12001216B2 - Status: Granted - Kind: B2 - Application number: 17/334,834 - Filed: 2021-05-31 - Published: 2022-12-01 - Granted: 2024-06-04 - Earliest claimed: 2021-05-31 - Assignees: Ubkang (Qingdao) Technology Co., Ltd. - Applicants: Ubtech North America Research and Development Center Corp; Ubtech Robotics Corp Ltd - Inventors: Chuqiao Dong; Dan Shao; Zhen Xiu; Dejun Guo; Huan Tan - Web page: https://patentgrep.com/patent/US12001216B2 ## CPC - G05D 1/0248: {in combination with a laser (lasers per se H01S)} - G05D 1/628: following the obstacle profile, e.g. a wall or undulated terrain - G05D 1/249: from positioning sensors located off-board the vehicle, e.g. from cameras - G05D 1/65: Following a desired speed profile - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/09: Supervised learning - G06T 7/55: from multiple images - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G06V 10/82: using neural networks - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G06N 3/08: Learning methods - G06T 2207/10024: Color image - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30261: Obstacle - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Carpet detection using an RGB-D camera and mobile machine movement control based thereon are disclosed. Carpets and carpet curls are detected by obtaining a RGB-D image pair including an RGB image and a depth image through an RGB-D camera, detecting carpet and carpet-curl areas in the RGB image and generating a 2D bounding box to mark each area using a deep learning model, and generating groups of carpet and carpet-curt points corresponding to each of the carpet and carpet-curl areas by matching each pixel of the RGB image within each 2D bounding box corresponding to the carpet and carpet curl areas to each pixel in the depth image. ## Claims ### Claim 1 A carpet detection method, comprising: obtaining, through an RGB-D camera, one or more RGB-D image pairs, wherein each RGB-D image pair comprises and RGB image and a depth image, and the depth image comprises data for representing a distance to scene objects in the depth image; detecting one or more carpet areas in the RGB image of the one or more RGB-D image pairs and generating a first 2D bounding box to mark each of the one or more carpet areas using a first deep learning model based on a first data set containing a carpet class, wherein the first data set comprises a plurality of images of carpets in various scenes; detecting one or more carpet-curl areas in the RGB image of the one or more RGB-D image pairs and generating a second 2D bounding box to mark each of the one or more carpet-curl areas using a second deep learning model based on a second data set containing a carpet-curl class, wherein the second data set comprises a plurality of images of carpet-curls in various scenes; generating a group of carpet points corresponding to each of the one or more carpet areas in the RGB image of each of the one or more RGB-D image pairs by matching each pixel of the RGB image within the first 2D bounding box corresponding to the carpet area to each pixel in the depth image of the RGB-D image pair; generating a group of carpet-curl points corresponding to each of the one or more carpet-curl areas in the RGB image of each of the one or more RGB-D image pairs by matching each pixel of the RGB image within the second 2D bounding box corresponding to the carpet-curl area to each pixel in the depth image of the RGB-D image pair; and controlling a mobile machine to move and avoid one or more carpet-curls, according to the generated group of carpet points corresponding to each of the one or more carpet areas in the RGB image of each of the one or more RGB-D image pairs, and the generated group of carpet-curl points corresponding to each of the one or more carpet-curl areas in the RGB image of each of the one or more RGB-D image pairs. ### Claim 2 (depends on 1) The method of claim 1, wherein controlling the mobile machine to move and avoid the one or more carpet-curls, according to the generated group of carpet points corresponding to each of the one or more carpet areas in the RGB image of each of the one or more RGB-D image pairs, and the generated group of carpet-curl points corresponding to each of the one or more carpet-curl areas in the RGB image of each of the one or more RGB-D image pairs comprises: taking the generated group of carpet points corresponding to each of the one or more carpet areas in the RGB image of each of the one or more RGB-D image pairs as a carpet; obtaining the one or more carpet-curls each corresponding to each of the one or more carpet-curl areas in the RGB image of each of the one or more RGB-D image pairs based on the generated group of carpet-curl points corresponding to each of the one or more carpet-curl areas, and take each of the one or more carpet-curls as an obstacle; and controlling the mobile machine to make a movement with reference to the carpet corresponding to each of the one or more carpet areas in the RGB image of each of the one or more RGB-D image pairs and avoid the obstacle. ### Claim 3 (depends on 2) The method of claim 2, further comprising: generating a 3D bounding box to mark the generated group of carpet-curl points corresponding to each of the one or more carpet-curl areas in the RGB image of each of the one or more RGB-D image pairs; and taking the generated 3D bounding box as the obstacle. ### Claim 4 (depends on 1) The method of claim 1, wherein the method further comprises: planing a trajectory, and controlling the mobile machine to move and avoid obstacles according to the trajectory, wherein the obstacles comprise the one or more carpet-curls, and an obstacle detected by fusing sensor data received from different sensors of the mobile machine. ### Claim 5 (depends on 4) The method of claim 4, wherein the method further comprises: in response to each directional change of the mobile machine, returning to perform the step of obtaining, through an RGB-D camera, the one or more RGB-D image pairs. ### Claim 6 A method for controlling a mobile machine having an RGB-D camera to move, comprising: at least one or more processors of the mobile machine, obtaining, through the RGB-D camera, one or more RGB-D image pairs, wherein each RGB-D image pair comprises an RGB image and a depth image, and the depth image comprises data for representing a distance to scene objects in the depth image; detecting one or more carpet areas in the RGB image of the one or more RGB-D image pairs and generating a first 2D hounding box to mark each of the one or more carpet areas using a first deep learning model based on a first data set containing a carpet class, wherein the first data set comprises a plurality of images of carpets in various scenes; detecting one or more carpet-curl areas in the RGB image of the one or more RGB-D image pairs and generating a second 2D bounding box to mark each of the one or more carpet-curl areas using a second deep learning model based on a second data set containing a carpet-curl class, wherein the second data set comprises a plurality of images of carpet-curls in various scenes; generating a group of carpet points corresponding to each of the one or more carpet areas in the RGB image of each of the one or more RGB-D image pairs by matching each pixel of the RGB image within the first 2D bounding box corresponding to the carpet area to each pixel in the depth image of the RGB-D image pair; generating a group of carpet-curl points corresponding to each of the one or more carpet-curl areas in the RGB image of each of the one or more RGB-D image pairs by matching each pixel of the RGB image within the second 2D bounding box corresponding to the carpet-curl area to each pixel in the depth image of the RGB-D image pair; taking the generated group of carpet points corresponding to each of the one or more carpet areas in the RGB image of each of the one or more RGB-D image pairs as a carpet; obtaining one or more carpet-curls each corresponding to each of the one or more carpet-curl areas in the RGB image of each of the one or more RGB-D image pairs based on the generated group of carpet-curl points corresponding to each of the one or more carpet-curl areas, and taking each of the one or more carpet-curls as an obstacle; and controlling the mobile machine to make a movement with reference to the carpet corresponding to each of the one or more carpet areas in the RGB image of each of the one or more RGB-D image pairs and avoid the obstacle. ### Claim 7 (depends on 6) The method of claim 6, wherein the taking the group of carpet points corresponding to each of the one or more carpet areas in the RGB image of each of the one or more RGB-D image pairs as the carpet comprises: combining all the generated groups of carpet points into the carpet; and the controlling the mobile machine to make the movement with reference to the carpet corresponding to each of the one or more carpet areas in the RGB image of each of the one or more RGB-D image pairs and avoid the obstacle comprises: controlling the mobile machine to make the movement with reference to the combined carpet and avoid the obstacle. ### Claim 8 (depends on 6) The method of claim 6, wherein the obtaining the one or more carpet curls each corresponding to each of the one or more carpet-curl areas in the RGB image of each of the one or more RGB-D image pairs based on the group of carpet-curl points, and taking each of the one or more carpet-curls as the obstacle comprises: generating a 3D bounding box to mark the generated group of carpet-curl points corresponding to each of the one or more carpet-curl areas in the RGB image of each of the one or more RGB-D image pairs; and taking the generated 3D bounding box as the obstacle. ### Claim 9 (depends on 6) The method of claim 6, wherein the controlling the mobile machine to make the movement with reference to the carpet and avoid the obstacle comprises: planning a to avoiding the obstacle; controlling the mobile machine to move according to the trajectory by actuating one or more motors of the mobile machine according to the trajectory; and keeping a current speed of the mobile machine by adjusting one or more parameters of the one or more motors in response to the mobile machine being on the carpet. ### Claim 10 (depends on 6) The method of claim 6, wherein the first deep learning model and the second deep learning model are based on YOLO algorithm. ### Claim 11 (depends on 10) The method of claim 10, wherein the first deep learning model is trained using the first data set comprising a COCO data set. ### Claim 12 A mobile machine, comprising: an RGB-D camera; one or more processors; and memory storing one or more programs configured to be executed by the one or more processors, wherein the one or more programs include instructions to: obtain, through an RGB-D camera, one or more RGB-D image pairs, wherein each RGB-D image pair comprises an RGB image and a depth image; detect one or more carpet areas in the RGB image of the one or more RGB-D image pairs and generate a first 2D bounding box to mark each of the one or more carpet areas using a first deep learning model based on a first data set containing a carpet class, wherein the first data set comprises a plurality of images of carpets in various scenes; detect one or more carpet-curl areas in the RGB image of the one or more RGB-D image pairs and generate a second 2D bounding box to mark each of the one or more carpet-curl areas using a second deep learning model based on a second data set containing a carpet-curl class, wherein the second data set comprises a plurality of images of carpet-curls in various scenes; generate a group of carpet points corresponding to each of the one or more carpet areas in the RGB image of each of the one or more RGB-D image pairs by matching each pixel of the RGB image within the first 2D bounding box corresponding to the carpet area to each pixel in the depth image of the RGB-D image pair; generate a group of carpet-curl points corresponding to each of the one or more carpet-curl areas in the RGB image of each of the one or more RGB-D image pairs by matching each pixel of the RGB image within the second 2D bounding box corresponding to the carpet-curl area to each pixel in the depth image of the RGB-D image pair; and control the mobile machine to move and avoid one or more carpet-curls, according to the generated group of carpet points corresponding to each of the one or more carpet areas in the RGB image of each of the one or more RGB-D image pairs, and the generated group of carpet-curl points corresponding to each of the one or more carpet-curl areas in the RGB image of each of the one or more RGB-D image pairs. ### Claim 13 (depends on 12) The mobile machine of claim 12, the one or more programs further include instructions to: take the generated group of carpet points corresponding to each of the one or more carpet areas in the RGB image of each of the one or more RGB-D image pairs as a carpet; obtain one or more carpet-curls each corresponding to each of the one or more carpet-curl areas in the RGB image of each of the one or more RGB-D image pairs based on the generated group of carpet-curl points corresponding to each of the one or more carpet-curl areas, and take each of the one or more carpet-curls as an obstacle; and control the mobile machine to make a movement with reference to the carpet corresponding to each of the one or more carpet areas in the RGB image of each of the one or more RGB-D image pairs and avoid the obstacle. ### Claim 14 (depends on 13) The mobile machine of claim 13, wherein the taking the group of carpet points corresponding to each of the one or more carpet areas in the RGB image of each of the one or more RGB-D image pairs as the carpet comprises: combining all the generated groups of carpet points into the carpet; and the controlling the mobile machine to make the movement with reference to the carpet corresponding to each of the one or more carpet areas in the RGB image of each of the one or more RGB-D image pairs and avoid the obstacle comprises: controlling the mobile machine to make the movement with reference to the combined carpet and avoid the obstacle. ### Claim 15 (depends on 14) The mobile machine of claim 14, wherein the controlling the mobile machine to make the movement with reference to the carpet and avoid the obstacle comprises: planning a trajectory avoiding the obstacle; controlling the mobile machine to move according to the trajectory by actuating one or more motors of the mobile machine according to the trajectory; and keeping a current speed of the mobile machine by adjusting one or more parameters of the one or more motors in response to the mobile machine being on the carpet. ### Claim 16 (depends on 13) The mobile machine of claim 13, the obtaining the one or more carpet-curls each corresponding to each of the one or more carpet-curl areas in the RGB image of each of the one or more RGB-D image pairs based on the group of carpet-curl points, and taking each of the one or more carpet-curls as the obstacle comprises: generating, a 3D bounding box to mark the generated group of carpet-curl points corresponding to each of the one or more carpet-curl areas in the RGB image of each of the one or more RGB-D image pairs; and taking the generated 3D bounding box as the obstacle. ### Claim 17 (depends on 12) The mobile machine of claim 12, the one or more programs further include instructions to: display the RGB image of each of the one or more RGB-D image pairs and the generated first 2D box corresponding to each of the one or more carpet areas in the RGB image for marking the carpet area on a display of the mobile machine. ### Claim 18 (depends on 12) The mobile machine of claim 12, the one or more programs further include instructions to: display the RGB image of each of the one or more RGB-D image pairs and the generated second 2D box corresponding to each of the one or more carpet-curl areas in the RGB image for marking the carpet-curl area on a display of the mobile machine. ### Claim 19 (depends on 12) The mobile machine of claim 12, wherein the first deep learning model and the second deep learning model are based on YOLO algorithm. ### Claim 20 (depends on 12) The mobile machine of claim 12, wherein the first deep learning model is trained using the first data set comprising a COCO data set.