# US 2025/0157230 A1 · Method and apparatus for three-dimensional object perception > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2025/0157230 A1 - Document ID: US20250157230A1 - Status: Published application - Kind: A1 - Application number: 18/635,249 - Filed: 2024-04-15 - Published: 2025-05-15 - Earliest claimed: 2023-11-13 - Assignees: Samsung Electronics Co., Ltd. - Applicants: Samsung Electronics Co., Ltd. - Inventors: Dongwook LEE; Chanho EOM; Byung In YOO; Hyunjeong LEE - Web page: https://patentgrep.com/patent/US20250157230A1 ## CPC - G06V 20/64: Three-dimensional [3D] objects - G06V 20/647: {by matching two-dimensional images to three-dimensional objects} - G06V 10/22: by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition - G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion - G06V 10/40: Extraction of image or video features - G06V 10/761: {Proximity, similarity or dissimilarity measures} - G06V 10/764: using classification, e.g. of video objects - G06V 10/811: {the classifiers operating on different input data, e.g. multi-modal recognition} - G06V 10/82: using neural networks - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method for (3D) object detection includes: receiving an input image with respect to a 3D space, an input point cloud with respect to the 3D space, and an input language with respect to a target object in the 3D space; using an encoding model to generate candidate image features of partial areas of the input image, a point cloud feature of the input point cloud, and a linguistic feature of the input language; selecting a target image feature corresponding to the linguistic feature from among the candidate image features based on similarity scores of similarities between the candidate image features and the linguistic feature; generating a decoding output by executing a multi-modal decoding model based on the target image feature and the point cloud feature; and detecting a 3D bounding box corresponding to the target object by executing an object detection model based on the decoding output. ## Claims ### Claim 1 A method of detecting a three-dimensional (3D) object, the method comprising: receiving an input image with respect to a 3D space, an input point cloud with respect to the 3D space, and an input language with respect to a target object in the 3D space; using an encoding model to generate candidate image features of partial areas of the input image, a point cloud feature of the input point cloud, and a linguistic feature of the input language; selecting a target image feature corresponding to the linguistic feature from among the candidate image features based on similarity scores of similarities between the candidate image features and the linguistic feature; generating a decoding output by executing a multi-modal decoding model based on the target image feature and the point cloud feature; and detecting a 3D bounding box corresponding to the target object by executing an object detection model based on the decoding output. ### Claim 2 (depends on 1) The method of claim 1, wherein the generating of the candidate image features, the point cloud feature, and the linguistic feature comprises: generating the linguistic feature corresponding to the input language by a language encoding model performing inference on the input language; generating candidate image features corresponding to partial areas of the input image by executing an image encoding model and a region proposal model based on the input image; and generating a point cloud feature corresponding to the input point cloud by executing a point cloud encoding model based on the input point cloud. ### Claim 3 (depends on 1) The method of claim 1, further comprising generating extended expressions each comprising (i) a position field that indicates a geometric characteristic of the target object based on the input language and comprising (ii) a class field indicates a class of the target object, and wherein the linguistic feature is generated based on the extended expressions. ### Claim 4 (depends on 3) The method of claim 3, wherein objects of a same class and with different geometric characteristics are distinguished from each other based on the position fields of the extended expressions. ### Claim 5 (depends on 3) The method of claim 3, wherein the position field is learned through training. ### Claim 6 (depends on 1) The method of claim 1, wherein the generating of the decoding output comprises: generating image tokens by segmenting the target image feature; generating point cloud tokens by segmenting the point cloud feature; generating first position information indicating relative positions of the respective image tokens; generating second position information indicating relative positions of the respective point cloud tokens; and executing the multi-modal decoding model with key data and value data based on the image tokens, the point cloud tokens, the first position information, and the second position information. ### Claim 7 (depends on 6) The method of claim 6, wherein the generating of the decoding output further comprises executing the multi-modal decoding model with query data based on detection guide information indicating detection position candidates with a possibility of detecting the target object in the 3D space. ### Claim 8 (depends on 7) The method of claim 7, wherein the detection position candidates are distributed non-uniformly. ### Claim 9 (depends on 7) The method of claim 7, wherein the multi-modal decoding model generates the decoding output by extracting a correlation from the target image feature, the point cloud feature, and the detection guide information. ### Claim 10 (depends on 1) A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1. ### Claim 11 An electronic device comprising: one or more processors; and a memory storing instructions configured to cause the one or more processors to: receive an input image with respect to a three-dimensional (3D) space, an input point cloud with respect to the 3D space, and an input language with respect to a target object in the 3D space; use an encoding model to generate candidate image features of partial areas of the input image, a point cloud feature of the input point cloud, and a linguistic feature of the input language; select a target image feature corresponding to the linguistic feature from the candidate image features based on similarity scores of similarities between the candidate image features and the linguistic feature; generate a decoding output by executing a multi-modal decoding model based on the target image feature and the point cloud feature; and detect a 3D bounding box corresponding to the target object by executing an object detection model based on the decoding output. ### Claim 12 (depends on 11) The electronic device of claim 11, wherein, the instructions are further configured to cause the one or more processors to: generate a linguistic feature corresponding to the input language by a language encoding model performing inference on the input language, generate candidate image features corresponding to partial areas of the input image by executing an image encoding model and a region proposal model based on the input image, and generate a point cloud feature corresponding to the input point cloud by executing a point cloud encoding model based on the input point cloud. ### Claim 13 (depends on 11) The electronic device of claim 11, wherein the instructions are further configured to cause the one or more processors to generate extended expressions each comprising (i) a position field indicating a geometric characteristic of the target object based on the input language and (ii) a class field indicating a class of the target object are generated, and wherein the linguistic feature is generated based on the extended expressions. ### Claim 14 (depends on 13) The electronic device of claim 13, wherein objects of a same class with different geometric characteristics are distinguished from each other based on the position field. ### Claim 15 (depends on 13) The electronic device of claim 13, wherein the position field is learned through training. ### Claim 16 (depends on 11) The electronic device of claim 11, wherein, the instructions are further configured to cause the one or more processors to: generate image tokens by segmenting the target image feature, generate point cloud tokens by segmenting the point cloud feature, generate first position information indicating relative positions of the respective image tokens, generate second position information indicating relative positions of the respective point cloud tokens, and execute the multi-modal decoding model with key data and value data based on the image tokens, the point cloud tokens, the first position information, and the second position information. ### Claim 17 (depends on 16) The electronic device of claim 16, wherein the instructions are further configured to cause the one or more processors to execute the multi-modal decoding model with query data based on detection guide information indicating detection position candidates with a possibility of detecting the target object in the 3D space. ### Claim 18 (depends on 17) The electronic device of claim 17, wherein the detection position candidates are non-uniformly distributed. ### Claim 19 (depends on 17) The electronic device of claim 17, wherein the multi-modal decoding model is configured to generate the decoding output by extracting a correlation from the target image feature, the point cloud feature, and the detection guide information. ### Claim 20 A vehicle comprising: a camera configured to generate an input image with respect to a three-dimensional (3D) space; a light detection and ranging (lidar) sensor configured to generate an input point cloud with respect to the 3D space; one or more processors configured to: receive the input image with respect to the 3D space, the input point cloud with respect to the 3D space, and an input language with respect to a target object in the 3D space; use an encoding model to generate candidate image features of partial areas of the input image, a point cloud feature of the input point cloud, and a linguistic feature of the input language; select a target image feature corresponding to the linguistic feature from the candidate image features based on similarity scores of similarities between the candidate image features and the linguistic feature; generate a decoding output by executing a multi-modal decoding model based on the target image feature and the point cloud feature; and detect a 3D bounding box corresponding to the target object by executing an object detection model based on the decoding output; and a control system configured to control the vehicle based on the 3D bounding box.