Intelligent assistive blind guidance system and method based on audio-visual fusion and haptic feedback

An intelligent assistive blind guidance system based on audio-visual fusion and haptic feedback, wherein the system comprises a robotic guide dog and a guide cane, the robotic guide dog comprises a robotic guide dog main body and a quadruped drive device, and the robotic guide dog main body comprises:

• a vision module, configured to sense surrounding environment information in real time;
• a voice interaction module, configured to:
• perform voice interaction with a user, receive a voice instruction of the user, and feed back a voice navigation prompt based on a walking prompt instruction;
• a data processing module, configured to:
• construct a surrounding environment map based on received surrounding environment information, and generate a planned path and a movement strategy with reference to the voice instruction of the user, the movement strategy comprising a current required walking speed;
• and
• a control module, configured to generate, based on the planned path and the movement strategy with reference to the voice instruction of the user and a pushing and pulling force of the user, a walking prompt instruction and a drive instruction used for driving the quadruped drive device;
• wherein, a top handle of the guide cane comprises a haptic feedback module, the haptic feedback module comprises a vibration device and a force sensor configured to sense the pushing and pulling force of the user, and the vibration device is configured to provide haptic vibration feedback based on the walking prompt instruction;
• the generating a movement strategy comprises:
• calculating an environment sparsity based on a quantity of pedestrians, a pedestrian speed, and a distance between the robotic guide dog and a nearest obstacle that are detected in real time in a current scenario, wherein a calculation formula is:
• A
• =
• 1
• k
• 1
• ·
• N
• +
• k
• 2
• ·
• V
• avg
• +
• k
• 3
• ·
• 1
• d
• min
• ;
• wherein, N is a quantity of pedestrians detected in a current environment, Vavg is an average speed of pedestrians in the current environment, the pedestrian in the environment is detected by using a camera and a light detection and ranging (LiDAR), and a speed vi of each pedestrian is calculated, to further calculate an average speed of pedestrians in a crowd by using
• V
• avg
• =
• 1
• N
• ⁢
• ∑
• i
• =
• 1
• N
• v
• i
• ;
• dmin is the distance between the robotic guide dog and the nearest obstacle, and is detected and obtained by using the LiDAR or an ultrasonic sensor;
• and k1, k2, and k3 are weight coefficients, used for adjusting impact of the quantity of pedestrians, the pedestrian speed, and the distance from the obstacle on the environment sparsity;
• calculating a current real-time speed of a blind user based on a motion state of the robotic guide dog;
• and
• generating a current required walking speed, that is, a recommended walking speed of the user based on the environment sparsity with reference to a preset reference walking speed of the user, wherein the recommended walking speed of the user comprises a minimum safe speed Vmin and a maximum safe speed Vmax recommended for the user to walk:
• V
• min
• =
• V
• base
• ·
• (
• 1
• -
• α
• ·
• (
• 1
• -
• A
• )
• )
• ;
• V
• max
• =
• V
• base
• ·
• (
• 1
• +
• β
• ·
• A
• )
• ;
• wherein, Vbase is a reference walking speed of the user;
• and, a and B are adjustment coefficients, used for controlling a safe speed range.