# US 10,479,653 B2 · Monitoring system of a passenger conveyor, a passenger conveyor, and a monitoring method thereof > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,479,653 B2 - Document ID: US10479653B2 - Status: Granted - Kind: B2 - Application number: 15/663,456 - Filed: 2017-07-28 - Published: 2018-02-01 - Granted: 2019-11-19 - Earliest claimed: 2016-07-29 - Assignees: Otis Elevator Company - Applicants: Otis Elevator Company - Inventors: Qiang Li; Jianwei Zhao; ZhaoXia Hu; LongWen Wang; Hui Fang; Zhen Jia; Anna Su - Web page: https://patentgrep.com/patent/US10479653B2 ## CPC - B66B 25/006: {Monitoring for maintenance or repair (for security reasons B66B29/005)} - B66B 21/02: Escalators - B66B 25/003: {Methods or algorithms therefor} - B66B 29/005: {Applications of security monitors} - H05B 47/125: by using cameras - Y02B 20/40: Control techniques providing energy savings, e.g. smart controller or presence detection - B66B: ELEVATORS; ESCALATORS OR MOVING WALKWAYS ({apparatus for raising or lowering persons on theatrical stages or the like A63J5/12} ; funicular railbound systems with rigid ground-supported tracks and cable traction, e.g. cliff railways, B61B9/00; arrangements of ammunition handlers in vessels B63G3/00; hoists, lifts, or conveyors for loading or unloading in general B65G; braking or detent devices controlling normal movements of winding drums or barrels B66D; ship-lifting devices E02C; garages for many vehicles with mechanical means for lifting vehicles E04H6/12; hoists for feeding ammunition or projectiles to launching apparatus or to loading mechanisms F41A9/00) - H05B: ELECTRIC HEATING; ELECTRIC LIGHT SOURCES NOT OTHERWISE PROVIDED FOR; CIRCUIT ARRANGEMENTS FOR ELECTRIC LIGHT SOURCES, IN GENERAL - Y02B: CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO BUILDINGS, e.g. HOUSING, HOUSE APPLIANCES OR RELATED END-USER APPLICATIONS ## Abstract A monitoring system includes: an imaging sensor configured to sense a lighting device of the passenger conveyor to acquire image data; and a processing apparatus configured to process the image data, the processing apparatus being configured to include: a background acquisition module configured to acquire a background model based on sensed image data; a foreground detection module configured to compare image data sensed in real time with the background model to obtain a foreground object; and a working condition judgment module configured to perform data processing at least based on the foreground object to judge whether the lighting device is in a normal state. Then, a failure or an abnormal lighting state of the lighting device can be timely diagnosed so that maintenance is performed. ## Claims ### Claim 1 A monitoring system of, comprising: an imaging sensor configured to sense a lighting device of the passenger conveyor to acquire image data; a microcontroller configured to process the image data to monitor whether the lighting device is in a normal state, the microcontroller having a plurality of modules, wherein the plurality of modules includes: a background acquisition module configured to acquire a background model based on the image data sensed when the lighting device is in a normal state or in an abnormal state; a foreground detection module configured to compare image data sensed in real time with the background model to obtain a foreground object; and a working condition judgment module configured to perform data processing at least based on the foreground object to judge whether the lighting device is in a normal state. ### Claim 2 (depends on 1) The monitoring system of claim 1, wherein the microcontroller further comprises: a foreground feature extraction module configured to extract a corresponding lighting state foreground feature from the foreground object; wherein the working condition judgment module judges, based on the lighting state foreground feature, whether the lighting device is in a normal state. ### Claim 3 (depends on 2) The monitoring system of claim 2, wherein the lighting state foreground feature extracted by the foreground feature extraction module comprises color and/or a light intensity of the lighting device, and/or a light intensity of a monitored area; and the working condition judgment module judges, based on the color and/or the light intensity of the lighting device and/or the light intensity of the monitored area, whether the lighting device is normal. ### Claim 4 (depends on 2) The monitoring system of claim 2, wherein the working condition judgment module is further configured to judge, based on the lighting state foreground feature of the foreground object, whether a specific lighting device is normal, and if the judgment result is “yes”, indicate the judgment, based on the image data processed currently, on whether the lighting device is in a normal state or indicate the judgment result of whether the lighting device corresponding to the image data processed currently is in a normal state. ### Claim 5 (depends on 2) The monitoring system of claim 2, wherein the working condition judgment module is further configured to determine that the lighting device is in an abnormal state, when a judgment of at least two successive pieces of image data indicate that the lighting device is in an abnormal state. ### Claim 6 (depends on 2) The monitoring system of claim 2, wherein the imaging sensor senses at intervals of a predetermined time period to acquire image data of the lighting device in the predetermined time period, and provides the image data for the microcontroller for data processing. ### Claim 7 (depends on 2) The monitoring system of claim 2, wherein the background acquisition module is configured to acquire a first background model based on image data sensed when the lighting device is in a lighting state, or acquire a second background model based on image data sensed when the lighting device is in a non-lighting state, or acquire a third background model group based on image data sensed when the lighting device is in a lighting state with multiple specific light intensities. ### Claim 8 (depends on 2) The monitoring system of claim 2, wherein there are multiple imaging sensors, which are arranged along a length direction of the passenger conveyor. ### Claim 9 (depends on 2) The monitoring system of claim 2, wherein the monitoring system further comprises an alarm unit, and the working condition judgment module triggers the alarm unit to work when determining that the lighting device is abnormal. ### Claim 10 (depends on 2) The monitoring system of claim 2, wherein, the monitoring system further comprises a learning module, the learning module being configured to monitor a lighting state foreground feature of the lighting device of the passenger conveyor under a daily operation condition, to update the background model acquired by the background acquisition module. ### Claim 11 (depends on 1) The monitoring system of claim 1, wherein the imaging sensor is integrated in an RGB-D sensor, and the RGB-D sensor further comprises a depth sensing sensor. ### Claim 12 (depends on 1) The monitoring system of claim 1, wherein the microcontroller is further configured to trigger, when the working condition judgment module determines that the lighting device is in an abnormal state, outputting of a signal to the passenger conveyor and/or an elevator maintenance center of a building to prevent occurrence of a safety accident. ### Claim 13 (depends on 1) The monitoring system of claim 1, wherein the lighting device comprises one or any combination of a traffic flow lamp, a comb lamp, a step lamp, a handrail lamp, and a barrier lamp. ### Claim 14 A monitoring method for a passenger conveyor, comprising: an image acquisition step: sensing a lighting device of the passenger conveyor to acquire image data; a background acquisition step: acquiring a background model based on image data sensed when the lighting device is in a normal state or in an abnormal state; a foreground detection step: comparing image data sensed in real time with the background model to obtain a foreground object; and a working condition judgment step: performing data processing at least based on the foreground object to judge whether the lighting device is in a normal state. ### Claim 15 (depends on 14) The monitoring method of claim 14, further comprising: a foreground feature extraction step: extracting a corresponding lighting state foreground feature from the foreground object according to the lighting device; wherein, in the working condition judgment step, whether the lighting device is in a normal state is judged based on the lighting state foreground feature. ### Claim 16 (depends on 15) The monitoring method of claim 15, wherein, the extracted lighting state foreground feature comprises color and/or a light intensity of the lighting device, and/or a light intensity of a monitored area; and in the working condition judgment step whether the lighting device is normal is judged based on the color and/or the light intensity of the lighting device and/or the light intensity of the monitored area. ### Claim 17 (depends on 16) The monitoring method of claim 16, wherein, in the working condition judgment step, when judging whether the lighting device is in a normal state is judging whether the lighting device emits light, whether the lighting device is in a normal state is judged based on the color and/or the light intensity of the lighting device. ### Claim 18 (depends on 16) The monitoring method of claim 16, wherein, in the working condition judgment step, when judging whether the lighting device is in a normal state is judging whether lighting brightness of the lighting device is suitable, whether the lighting device is in a normal state is judged based on the light intensity of the lighting device. ### Claim 19 (depends on 16) The monitoring method of claim 16, wherein, in the working condition judgment step, when judging whether the lighting device is in a normal state is judging whether lighting brightness of the lighting device is suitable, whether the lighting device is in a normal state is judged based on the light intensity and the color of the lighting device. ### Claim 20 (depends on 16) The monitoring method of claim 16, wherein, in the working condition judgment step, when judging whether the lighting device is in a normal state is judging whether lighting brightness of the lighting device is suitable, whether the lighting device is in a normal state is judged based on the light intensity of the monitored area. ### Claim 21 (depends on 16) The monitoring method of claim 16, wherein, in the foreground feature extraction step, the extracted lighting state foreground feature further comprises a position of the lighting device; and in the working condition judgment step, whether the lighting device is in a normal state is judged based on one or more of the color, the light intensity and the position of the lighting device and/or the light intensity of the monitored area. ### Claim 22 (depends on 15) The monitoring method of claim 15, wherein the working condition judgment step further comprises judging, based on the lighting state foreground feature of the foreground object, whether a specific lighting device is normal, and if the judgment result is “yes”, indicating the judgment, based on the image data processed currently, on whether the lighting device is in a normal state or indicating the judgment result of whether the lighting device corresponding to the image data processed currently is in a normal state. ### Claim 23 (depends on 15) The monitoring method of claim 15, wherein the working condition judgment step further comprises determining, when a judgment result of at least two successive pieces of image data is that the lighting device is in a same abnormal state that the lighting device is in the abnormal state. ### Claim 24 (depends on 15) The monitoring method of claim 15, wherein the image acquisition step further comprises sensing at intervals of a predetermined time period to acquire image data of the lighting device in the predetermined time period, and providing the image data for the processing apparatus for data processing. ### Claim 25 (depends on 15) The monitoring method of claim 15, wherein the background acquisition step further comprises acquiring a first background model based on image data sensed when the lighting device is in a lighting state, or acquiring a second background model based on image data sensed when the lighting device is in a non-lighting state, or acquiring a third background model group based on image data sensed when the lighting device is in a lighting state with multiple specific light intensities. ### Claim 26 (depends on 15) The monitoring method of claim 15, further comprising an alarming step: triggering an alarm unit to work when it is determined based on the working condition judgment step that the lighting device is abnormal. ### Claim 27 (depends on 15) The monitoring method of claim 15, further comprising, a learning step: monitoring a lighting state foreground feature of the lighting device of the passenger conveyor under a daily operation condition, to update the background model acquired in the background acquisition step. ### Claim 28 (depends on 1) A passenger conveyor, comprising the monitoring system of claim 1. ### Claim 29 A monitoring system of a passenger conveyor, comprising: an imaging sensor configured to sense a lighting device of the passenger conveyor to acquire image data; a microcontroller configured to process the image data to monitor whether the lighting device is in a normal state, the microcontroller is operable to: acquiring a background model, using a background acquisition module, based on image data sensed when the lighting device is in a normal state or in an abnormal state; compare the image data sensed in real time with the background model, via a foreground detection module, to obtain a foreground object; and perform data processing at least based on the foreground object, using a working condition judgment module, to judge whether the lighting device is in a normal state.