# US 12,110,665 B2 · Control system and control method for controlling work machine in accordance with image data > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,110,665 B2 - Document ID: US12110665B2 - Status: Granted - Kind: B2 - Application number: 17/056,136 - Filed: 2019-08-26 - Published: 2021-10-07 - Granted: 2024-10-08 - Earliest claimed: 2018-09-10 - Assignees: Komatsu Ltd. - Applicants: Komatsu Ltd. - Inventors: Masanori Aizawa; Kenjiro Shimada; Yuki Tsuzuki - Web page: https://patentgrep.com/patent/US12110665B2 ## CPC - E02F 9/261: {Surveying the work-site to be treated} - E02F 9/262: {with follow-up actions to control the work tool, e.g. controller} - E02F 3/434: {providing automatic sequences of movements, e.g. automatic dumping or loading, automatic return-to-dig} - E02F 3/437: {providing automatic sequences of movements, e.g. linear excavation, keeping dipper angle constant} - E02F 3/439: {Automatic repositioning of the implement, e.g. automatic dumping, auto-return (E02F3/438 takes precedence)} - E02F 9/265: {with follow-up actions (e.g. control signals sent to actuate the work tool)} - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/09: Supervised learning - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - H04N 7/181: {for receiving images from a plurality of remote sources} - H04N 23/61: based on recognised objects - G06T 2207/20081: Training; Learning - G06T 2207/30196: Human being; Person - E02F: DREDGING; SOIL-SHIFTING (winning peat E21C49/00) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A system includes a work machine, a camera, and a processor. The camera captures an image including a periphery of the work machine. A processor acquires image data indicative of a captured image captured by a camera. The processor acquires, from image data, a specific object present within the captured image and a distance from the work machine to the specific object by performing image analysis using an object detection model. The object detection model is trained an image of the specific object and the distance to the specific object. The processor controls the work machine based on the distance from the work machine to the specific object when the specific object is detected in the captured image. ## Claims ### Claim 1 A system comprising: a work machine; a camera configured to capture an image including a periphery of the work machine; and a processor configured to acquire image data indicative of a captured image captured by the camera, acquire, from the image data, at least one specific object present in the captured image and a distance from the work machine to the at least one specific object by performing image analysis using an object detection model trained using an image of the at least one specific object and the distance to the at least one specific object, determine whether the at least one specific object is one specific object or a plurality of specific objects, upon determining that the at least one specific object is one specific object, set the distance from the work machine to the one specific object as a distance parameter and control the work machine based on the distance parameter, and upon determining that the at least one specific object is a plurality of specific objects, select a shortest distance among distances from the work machine to each of the plurality of specific objects as the distance parameter and control the work machine based on the distance parameter. ### Claim 2 (depends on 1) The system according to claim 1, wherein the processor is further configured to when the distance parameter is equal to or greater than a predetermined distance threshold, control the work machine in a first control mode in which the work machine performs a predetermined work automatically, and when the distance parameter is less than the predetermined distance threshold, control the work machine in a second control mode that is different from the first control mode. ### Claim 3 (depends on 2) The system according to claim 2, wherein the processor is further configured to acquire a plurality of sets of image data which represent a plurality of captured images of the work machine in different directions and, when the specific object is detected in each of the plurality of captured images by image analysis using the object detection model, determine the distance parameter for each of the plurality of captured images and, when at least one distance parameter among the distance parameters of each of the plurality of captured images is less than the predetermined distance threshold, control the work machine in the second control mode. ### Claim 4 (depends on 2) The system according to claim 2, wherein the work machine includes an output device, and the processor is further configured to cause the output device to output a warning in the second control mode. ### Claim 5 (depends on 2) The system according to claim 2, wherein the processor is further configured to limit actions of the work machine in the second control mode more than in the first control mode. ### Claim 6 (depends on 2) The system according to claim 2, wherein the processor is further configured to stop the work machine in the second control mode. ### Claim 7 A system comprising: a work machine; a camera configured to capture an image including a periphery of the work machine; and a processor configured to acquire image data indicative of a captured image captured by the camera, acquire, from the image data, a specific object present in the captured image and a distance from the work machine to the specific object by performing image analysis using an object detection model trained using an image of the specific object and the distance to the specific object, control the work machine based on the distance from the work machine to the specific object when the specific object is detected in the captured image, acquire the distance for each of a plurality of predetermined time periods, the plurality of time periods including a current time period and a previous time period occurring before the current time period, and when a differential between the distance in the current time period and the distance in the previous time period is greater than a predetermined threshold, correct the distance in the current time period so as to reduce the differential. ### Claim 8 (depends on 1) The system according to claim 1, wherein the camera is a single lens camera. ### Claim 9 (depends on 1) The system according to claim 1, wherein the specific object includes a person. ### Claim 10 A method executed by a processor for controlling a work machine, the method comprising: acquiring image data indicative of a captured image including a periphery of the work machine; acquiring, from the image data, at least one specific object present in the captured image and a distance from the work machine to the at least one specific object by performing image analysis using an object detection model trained using an image of the at least one specific object and the distance to the at least one specific object; determining whether the at least one specific object is one specific object or a plurality of specific objects, upon determining that the at least one specific object is one specific object, setting the distance from the work machine to the one specific object as a distance parameter and controlling the work machine based on; and upon determining that the at least one specific object is a plurality of specific objects, selecting a shortest distance among distances from the work machine to the plurality of specific objects as the distance parameter and controlling the work machine based on the distance parameter. ### Claim 11 (depends on 10) The method according to claim 10, further comprising: when the distance parameter is equal to or greater than a predetermined distance threshold, controlling the work machine in a first control mode in which the work machine performs a predetermined work automatically; and when the distance parameter is less than the predetermined distance threshold, controlling the work machine in a second control mode that is different from the first control mode. ### Claim 12 (depends on 11) The method according to claim 11, further comprising: acquiring a plurality of sets of image data representing a plurality of captured images of the work machine in different directions; when the specific object is detected in each of the plurality of captured images by image analysis using the object detection model, determining the distance parameter for each of the plurality of captured images; and when at least one distance parameter among the distance parameters of each of the plurality of captured images is less than the predetermined distance threshold, controlling the work machine in the second control mode. ### Claim 13 (depends on 11) The method according to claim 11, wherein the work machine includes an output device, and the controlling the work machine in the second control mode includes causing the output device to output a warning. ### Claim 14 (depends on 11) The method according to claim 11, wherein the controlling the work machine in the second control mode includes limiting actions of the work machine more than in the first control mode. ### Claim 15 (depends on 11) The method according to claim 11, wherein the controlling the work machine in the second control mode includes stopping the work machine. ### Claim 16 (depends on 10) The method according to claim 10, wherein the specific object includes a person.