# US 2025/0085676 A1 · Device control method, apparatus and system, and electronic device and storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2025/0085676 A1 - Document ID: US20250085676A1 - Status: Published application - Kind: A1 - Application number: 18/728,465 - Filed: 2023-01-12 - Published: 2025-03-13 - Earliest claimed: 2022-01-12 - Assignees: Lumi United Technology Co., Ltd. - Applicants: Lumi United Technology Co., Ltd. - Inventors: Jing LU; Jiabo YANG - Web page: https://patentgrep.com/patent/US20250085676A1 ## CPC - G05B 15/02: electric - G06F 9/451: Execution arrangements for user interfaces - Y02P 90/02: Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS] - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - Y02P: CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS ## Abstract A device control method, comprising: displaying a device control interface corresponding to a scenario picture of a target scenario; in the device control interface, displaying a control assembly corresponding to at least one target device in the target scenario, wherein the control assembly is bound to a target device which is identified on the basis of an image in the target scenario; and in response to a control operation which is triggered for the control assembly, sending a corresponding control instruction to the target device, which corresponds to the control assembly, so as to control the target device to execute the control instruction. ## Claims ### Claim 1 A device control method executed by an electronic device, wherein the method comprises: showing a device control interface corresponding to a scenario picture of a target scenario; in the device control interface, displaying a control component corresponding to at least one target device in the target scenario, wherein the control component is bound to a target device identified in an image based on the target scenario; and in response to a control operation triggered for the control component, sending a corresponding control instruction to the target device corresponding to the control component, so as to control the target device to execute the control instruction. ### Claim 2 (depends on 1) The method according to claim 1, wherein the in the device control interface, displaying a control component corresponding to at least one target device in the target scenario comprises: displaying a scenario picture corresponding to a real target scenario in the device control interface; and displaying, in the scenario picture, a control component corresponding to at least one target device in the real target scenario. ### Claim 3 (depends on 1) The method according to claim 1, wherein the method further comprises: displaying a device status entrance corresponding to the target device in the control component; and showing current device status information corresponding to the target device in response to a trigger operation for the device status entrance. ### Claim 4 (depends on 3) The method according to claim 3, wherein the method further comprises: dynamically updating the device status information in the device control interface when identifying that the device status information of the target device generates change based on an image collected in the target scenario. ### Claim 5 (depends on 1) The method according to claim 1, wherein the device control interface comprises a two-dimensional scenario picture or a three-dimensional scenario picture corresponding to a real target scenario. ### Claim 6 A device control method executed by an electronic device, wherein the method comprises: performing target identification for an image collected in a target scenario to obtain a target device in the image; acquiring devices that have been added in the target scenario and matching the target device in the image with the devices that have been added; and if there is a device that has been added matching with the target device, binding the target device with a control component corresponding to the matching device that has been added in a device control interface corresponding to a scenario picture of the target scenario, wherein the control component is configured to control the target device in the device control interface. ### Claim 7 (depends on 6) The method according to claim 6, wherein the performing target identification for an image collected in a target scenario to obtain a target device in the image comprises: acquiring the image collected in the target scenario; and performing target identification for the image to obtain a device type corresponding to the target device in the image and a device position of the target device in the target scenario; the acquiring devices that have been added in the target scenario and matching the target device in the image with the devices that have been added comprises: acquiring device types and device positions of the devices that have been added in the target scenario; and matching the device type and the device position of the target device with the device types and the device positions of the devices that have been added. ### Claim 8 (depends on 7) The method according to claim 7, wherein the matching the device type and the device position of the target device with the device types and the device positions of the devices that have been added comprises: in the devices that have been added, determining target added devices of which device positions match with the device position of the target device; and in the target added devices, determining a device of which the device type matches with the device type of the target device as the device that has been added matching with the target device. ### Claim 9 (depends on 6) The method according to claim 6, wherein the target device in the image is obtained by performing identification through a trained deep learning model; the trained deep learning model is obtained through model training steps; the model training steps comprise: acquiring sample images collected in multiple kinds of real environments, wherein the sample images include sample labels labeled with positions and types of sample target devices; performing target identification for the sample images through an initial deep learning model to obtain an identification result of a sample target device in the sample images, wherein the identification result includes a position and a type of the sample target device; and adjusting parameters of the initial deep learning model based on difference between the identification result and the sample labels and continuing to train, and stop training when training conditions are met to obtain the trained deep learning model. ### Claim 10 A device control system comprising a terminal and a server; wherein the server is configured to: perform target identification for an image collected in a target scenario to obtain a target device in the image; acquire devices that have been added in the target scenario and match the target device in the image with the devices that have been added; if there is a device that has been added matching with the target device, bind the target device with a control component corresponding to the matching device that has been added in a device control interface corresponding to a scenario picture of the target scenario, wherein the control component is configured to control the target device in the device control interface; the terminal is configured to: show the device control interface corresponding to the scenario picture of the target scenario; in the device control interface, display a control component corresponding to at least one target device in the target scenario, wherein the control component is bound to a target device identified based on an image in the target scenario; and in response to a control operation triggered for the control component, send a corresponding control instruction to the target device corresponding to the control component, so as to control the target device to execute the control instruction. ### Claim at position 11 11-20. (canceled) ### Claim 21 (depends on 1) The method according to claim 1, wherein the control component is stacked and displayed on the at least one target device corresponding to the control component, or displayed beside the at least one target device corresponding to the control component. ### Claim 22 (depends on 1) The method according to claim 1, wherein the scenario picture comprises a real picture or a virtual picture reflecting the target scenario. ### Claim 23 (depends on 22) The method according to claim 22, wherein the real picture is obtained based on images collected by an image collecting module. ### Claim 24 (depends on 22) The method according to claim 22, wherein the virtual picture is obtained by modeling for the target scenario. ### Claim 25 (depends on 1) The method according to claim 1, wherein the control component comprises at least one of a physical control component and a virtual control component, and is configured to directly implement status display and/or functional control for the at least one target device corresponding to the control component in the device control interface. ### Claim 26 (depends on 7) The method according to claim 7, further comprising: monitoring whether an image picture of the image changes; if the image picture does not change, reducing a frequency of collecting the image. ### Claim 27 (depends on 6) The method according to claim 6, wherein the performing target identification for an image collected in a target scenario to obtain a target device in the image comprises: acquiring the image collected in the target scenario; and performing target identification for the image to obtain device types corresponding to a plurality of target devices in the image and device positions of the plurality of target devices in the target scenario; the acquiring devices that have been added in the target scenario and matching the target device in the image with the devices that have been added comprises: acquiring device types and device positions of the devices that have been added in the target scenario; and respectively matching the device types and the device positions of the plurality of target device with the device types and the device positions of the devices that have been added. ### Claim 28 (depends on 6) The method according to claim 6, further comprising: simultaneously displaying a plurality of control components respectively corresponding to the plurality of target devices in the device control interface, wherein each of the plurality of control components is stacked and displayed on a target device corresponding to the control component, or displayed beside a target device corresponding to the control component. ### Claim 29 (depends on 6) The method according to claim 6, wherein the control component comprises at least one of a physical control component and a virtual control component, and is configured to directly implement status display and/or functional control for the target device corresponding to the control component in the device control interface. ### Claim 30 (depends on 6) The method according to claim 6, further comprising: acquiring the number of the device that has been added matching with the target device; if the number of the device that has been added matching with the target device is more than one, showing a device selection interface, and displaying devices that have been added matching with the target device in the device selection interface; and according to a selection operation received by the device selection interface, binding the target device with a control component corresponding to a device that has been added corresponding to the selection operation.