# US 11,443,151 B2 · Driving assistant system, electronic device, and operation method thereof > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,443,151 B2 - Document ID: US11443151B2 - Status: Granted - Kind: B2 - Application number: 16/874,994 - Filed: 2020-05-15 - Published: 2021-03-04 - Granted: 2022-09-13 - Earliest claimed: 2019-09-03 - Assignees: Samsung Electronics Co., Ltd. - Applicants: Samsung Electronics Co., Ltd. - Inventors: Sangsoo Ko; Byeoungsu Kim; Sanghyuck Ha - Web page: https://patentgrep.com/patent/US11443151B2 ## CPC - G06V 10/764: using classification, e.g. of video objects - G06F 18/253: {of extracted features} - G06V 10/82: using neural networks - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G06V 20/647: {by matching two-dimensional images to three-dimensional objects} - H04N 13/128: Adjusting depth or disparity - H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) - G06V 2201/08: Detecting or categorising vehicles - H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals} - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Systems and method for automated driving assistance are described. A driving assistant system may include a first sensor unit configured to provide an input image including another vehicle and a background; an image processor configured to generate a plurality of pyramid images by down-sampling the input image, acquire a depthmap including depth values from the first sensor unit to the other vehicle and the background, generate a plurality of pieces of mask data with different average depths of the depth values based on the depthmap, and output a plurality of masked images representing different average distances from the first sensor unit based on the plurality of pieces of mask data and the plurality of pyramid images; a feature extractor configured to output feature data of each of the plurality of masked images; and a detector configured to detect the other vehicle included in the input image based on the feature data. ## Claims ### Claim 1 A driving assistant system comprising: a first sensor unit configured to provide an input image including another vehicle and a background; an image processor configured to generate a plurality of pyramid images by down-sampling the input image, identify a depthmap including depth values to the other vehicle and the background, generate a plurality of pieces of mask data having different average depths of the depth values based on the depthmap, and output a plurality of masked images representing different average distances based on the plurality of pieces of mask data and the plurality of pyramid images; a feature extractor configured to output feature data of each of the plurality of masked images; and a detector configured to detect the other vehicle included in the input image based on the feature data. ### Claim 2 (depends on 1) The driving assistant system of claim 1, wherein the first sensor unit comprises a stereo camera, the input image comprises a left-eye image and a right-eye image, and the image processor is further configured to acquire the depthmap based on a comparison of the left-eye image and the right-eye image. ### Claim 3 (depends on 2) The driving assistant system of claim 2, wherein the image processor is further configured to generate a region of interest (RoI) based on the input image and mask a region remaining by excluding the Ro from the plurality of pyramid images. ### Claim 4 (depends on 1) The driving assistant system of claim 1, wherein the plurality of pyramid images comprises a first pyramid image and a second pyramid image, and the image processor is further configured to generate the first pyramid image by down-sampling the input image and to generate the second pyramid image by down-sampling the first pyramid image. ### Claim 5 (depends on 4) The driving assistant system of claim 4, wherein the plurality of pieces of mask data comprises first mask data and second mask data having a smaller average depth than the first mask data, the plurality of masked images comprises a first masked image and a second masked image, and the image processor is further configured to resize the first mask data and the second mask data so as to correspond to the first pyramid image and the second pyramid image, output the first masked image in which the first mask data is applied to the first pyramid image, and output the second masked image in which the second mask data is applied to the second pyramid image. ### Claim 6 (depends on 1) The driving assistant system of claim 1, wherein the image processor is further configured to mask a region remaining by excluding at least one of the other vehicle, a road and a pedestrian from the input image and generate the plurality of pyramid images based on a masked image. ### Claim 7 (depends on 1) The driving assistant system of claim 1, wherein the plurality of masked images comprises a first masked image and a second masked image, the feature extractor is further configured to output first feature data generated based on the first masked image and second feature data generated based on the second masked image, and the detector is further configured to receive concatenated data generated by concatenating the first feature data and the second feature data. ### Claim 8 (depends on 7) The driving assistant system of claim 7, further comprising: a down-sampler configured to down-sample the second feature data and output down-sampled feature data; and a buffer configured to output the concatenated data by concatenating the first feature data and the down-sampled feature data. ### Claim 9 (depends on 1) The driving assistant system of claim 1, further comprising: a second sensor unit comprising a first distance sensor and a second distance sensor having a shorter sensing distance than the first distance sensor, the detector further comprising a first sub-detector and a second sub-detector; a first sensor fusing unit configured to fuse first detection data output from the first sub-detector and first sensing data output from the first distance sensor and output first fusing data; and a second sensor fusing unit configured to fuse second detection data output from the second sub-detector and second sensing data output from the second distance sensor and output second fusing data, wherein the first detection data is generated based on an image including an object of a farther distance than that of the second detection data. ### Claim 10 (depends on 9) The driving assistant system of claim 9, further comprising a third distance sensor having a shorter sensing distance than the second distance sensor, wherein the third distance sensor outputs third sensing data, and the second sensor fusing unit is further configured to fuse the second detection data, the second sensing data, and the third sensing data. ### Claim 11 (depends on 9) The driving assistant system of claim 9, further comprising a merging unit configured to acquire three-dimensional information of the other vehicle based on the first fusing data and the second fusing data.