Auxiliary sensing method and system based on sensory substitution

An auxiliary sensing system based on a sensory substitution, comprising:

• a first module configured to establish a discrete horizontal plane included angle and a normal vector included angle in a three-dimensional space as well as response functions corresponding to different discrete distances, and to establish an acoustic coding library of an object;
• a second module configured to obtain a surrounding environmental video in real time by using a visual input unit;
• a third module configured to obtain object information and a category of a current environment by performing an object detection on each frame of a visual image based on the surrounding environmental video obtained, and to calculate a three-dimensional coordinate position of each object from the visual input unit through each frame of visual image information;
• a fourth module configured to establish a three-dimensional sound field of the object according to the visual image information, the object information and the three-dimensional coordinate position obtained in combination with the acoustic coding library;
• and
• a fifth module configured to adjust, for objects of different categories, the three-dimensional sound field of the each object according to movement information and variations of images, and to play the three-dimensional sound field of the each object in real time;
• wherein the third module is configured to perform:
• importing the each frame of the visual image in a real-time video into a deep neural network, detecting the object by using the deep neural network to obtain an image position and a category of the each object in each frame of the visual input unit, and distinguishing between a fixed obstacle and a non-fixed obstacle;
• segmenting an object frame to obtain an object image in a current frame;
• abstracting the object image in the object frame to obtain a hash value Hof the object image in the current frame;
• calculating a Hamming distance between a hash value of a left image and a hash value of a right image to obtain an image frame corresponding to a same object in the left image and the right image through the Hamming distance;
• detecting whether the same object in the current frame and a previous frame of a detection image exists, marking the object as a non-disappearing object if the same object exists, calculating a displacement vector of the same object, and marking an object beyond an image visual field as a disappeared object;
• performing a corner-point analysis on the object in the image frame of the left image and the right image to obtain a corner-point matrix of the object;
• calculating a hash value of the corner-point matrix corresponding to the left image and the right image;
• clustering by importing the hash value of the corner-point matrix corresponding to the left image and the right image into a same space;
• searching for, through clustering points with a closest hash value of each corner-point matrix for a pairwise coupling;
• connecting pairwise coupled points;
• calculating, according to an inertial measurement unit, a vertical bias angle from a horizontal plane generated by a head up and down movement of a current user, a horizontal heading angle generated by moving the head of the user from side to side between a plane perpendicular to a center line of the visual input unit in a previous state and the plane perpendicular to the center line of the visual input unit in a current state, and a horizontal rolling angle formed by a lateral inclination and a normal vector of the horizontal plane;
• removing a corner point of corner points where a paired connecting line of connecting lines is a non-horizontal connecting line;
• calculating distances of all objects with respect to the visual input unit according to selected corner points and the connecting lines;
• calculating a vertical height of the object with respect to the horizontal plane of the visual input unit according to a calculated distance of the object with respect to the visual input unit and a vertical bias between the object and a center position of the visual input unit;
• and
• calculating a horizontal bias distance of the object with respect to a center of the visual input unit according to the calculated distance of the object with respect to the visual input unit as well as a horizontal bias between the object and the center position of the visual input unit.