# US 12,223,740 B2 · Image processing method, network training method, and related device > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,223,740 B2 - Document ID: US12223740B2 - Status: Granted - Kind: B2 - Application number: 17/975,922 - Filed: 2022-10-28 - Published: 2023-02-16 - Granted: 2025-02-11 - Earliest claimed: 2020-04-30 - Assignees: Huawei Technologies Co., Ltd. - Applicants: Huawei Technologies Co., Ltd. - Inventors: Xinhai Zhao; Zhen Yang; Wei Zhang - Web page: https://patentgrep.com/patent/US12223740B2 ## CPC - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/09: Supervised learning - G06T 7/73: using feature-based methods - G06V 10/22: by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G06V 10/82: using neural networks - G06V 20/64: Three-dimensional [3D] objects - G06N 3/048: Activation functions - G06T 2207/10016: Video; Image sequence - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G06T 2207/30261: Obstacle - G06V 2201/08: Detecting or categorising vehicles - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract This application provides an image processing method, a network training method, and a related device, and relates to image processing technologies in the artificial intelligence field. The method includes: inputting a first image including a first vehicle into an image processing network to obtain a first result output by the image processing network, where the first result includes location information of a two-dimensional (2D) bounding frame of the first vehicle, coordinates of a wheel of the first vehicle, and a first angle of the first vehicle, and the first angle of the first vehicle indicates an included angle between a side line of the first vehicle and a first axis of the first image; and generating location information of a three-dimensional (3D) outer bounding box of the first vehicle based on the first result. ## Claims ### Claim 1 An image processing method, wherein the method comprising: obtaining a first image comprising a first vehicle; inputting the first image into an image processing network to obtain a first result output by the image processing network, wherein when a side face of the first vehicle is exposed in the first image, the first result comprises location information of a two-dimensional (2D) bounding frame of the first vehicle, coordinates of a wheel of the first vehicle, and a first angle of the first vehicle; the first angle of the first vehicle indicates an included angle between a side line of the first vehicle and a first axis of the first image; the side line of the first vehicle is an intersection line between the exposed side face of the first vehicle and a ground plane on which the first vehicle is located; and the first axis of the first image is parallel to one side of the first image; and generating location information of a three-dimensional (3D) outer bounding box of the first vehicle based on the location information of the 2D bounding frame of the first vehicle, the coordinates of the wheel, and the first angle, wherein the location information of the 3D outer bounding box of the first vehicle comprises coordinates of at least two first points, the at least two first points are all located on a side of the 3D outer bounding box of the first vehicle, the side of the 3D outer bounding box of the first vehicle is located using two first points in the at least two first points, and the 3D outer bounding box of the first vehicle is located using the coordinates of the at least two first points. ### Claim 2 (depends on 1) The method according to claim 1, wherein when only the side face of the first vehicle is exposed in the first image, the at least two first points comprise two intersection points between the side line of the first vehicle and the 2D bounding frame of the first vehicle. ### Claim 3 (depends on 2) The method according to claim 2, wherein the generating location information of a 3D outer bounding box of the first vehicle based on the location information of the 2D bounding frame, the coordinates of the wheel, and the first angle comprises: generating location information of the side line of the first vehicle based on the coordinates of the wheel of the first vehicle and the first angle of the first vehicle; and generating coordinates based on the location information of the side line of the first vehicle and the location information of the 2D bounding frame of the first vehicle, to obtain the coordinates of the at least two first points. ### Claim 4 (depends on 1) The method according to claim 1, wherein when the side face and a main face of the first vehicle are exposed in the first image, the first result further comprises location information of a boundary of the first vehicle and a second angle of the first vehicle; the boundary is a boundary between the side face and the main face; the main face of the first vehicle is a front face or rear face of the first vehicle; the second angle of the first vehicle indicates an included angle between a main side line of the first vehicle and the first axis of the first image; and the main side line of the first vehicle is an intersection line between the exposed main face of the first vehicle and the ground plane on which the first vehicle is located; and the at least two first points comprise a first intersection point, a second intersection point, and a third intersection point; the first intersection point is an intersection point between the side line of the first vehicle and the boundary of the first vehicle, and the first intersection point is a vertex of the 3D outer bounding box of the first vehicle; the second intersection point is an intersection point between the side line of the first vehicle and the 2D bounding frame of the first vehicle; and the third intersection point is an intersection point between the main side line of the first vehicle and the 2D bounding frame of the first vehicle. ### Claim 5 (depends on 4) The method according to claim 4, wherein the generating location information of a three-dimensional (3D) outer bounding box of the first vehicle based on the location information of the 2D bounding frame of the first vehicle, the coordinates of the wheel, and the first angle comprises: generating location information of the side line of the first vehicle based on the coordinates of the wheel of the first vehicle and the first angle of the first vehicle; generating coordinates of the first intersection point based on the location information of the side line of the first vehicle and the location information of the boundary of the first vehicle; generating coordinates of the second intersection point based on the location information of the side line of the first vehicle and the location information of the 2D bounding frame of the first vehicle; generating location information of the main side line of the first vehicle based on the coordinates of the first intersection point and the second angle of the first vehicle; and generating coordinates of the third intersection point based on the location information of the main side line of the first vehicle and the location information of the 2D bounding frame of the first vehicle. ### Claim 6 (depends on 1) The method according to claim 1, further comprising: when the side face of the first vehicle is exposed in the first image, generating an orientation angle of the first vehicle relative to a vehicle equipped with an image processing apparatus based on coordinates of a first point. ### Claim 7 (depends on 6) The method according to claim 6, wherein before the generating an orientation angle based on coordinates of a first point, the method further comprises: generating a distance between the first point and the vehicle equipped with the image processing apparatus based on the coordinates of the first point and a ground plane assumption principle; and the generating an orientation angle based on coordinates of a first point comprises: upon determining, based on the distance between the first point and the vehicle equipped with the image processing apparatus, that a distance between the first vehicle and the vehicle equipped with the image processing apparatus does not exceed a preset-threshold, generating the orientation angle based on the coordinates of the first point according to a first calculation rule; or upon determining, based on the distance between the first point and the vehicle equipped with the image processing apparatus, that a distance between the first vehicle and the vehicle equipped with the image processing apparatus exceeds a threshold, generating the orientation angle based on the coordinates of the first point according to a second calculation rule, wherein the second calculation rule and the first calculation rule are different. ### Claim 8 (depends on 7) The method according to claim 7, wherein when a distance between any one of the at least two first points and the vehicle equipped with the image processing apparatus does not exceed the threshold, it is determined that the distance between the first vehicle and the vehicle equipped with the image processing apparatus does not exceed the threshold; or when a distance between any one of the at least two first points and the vehicle equipped with the image processing apparatus exceeds the threshold, it is determined that the distance between the first vehicle and the vehicle equipped with the image processing apparatus exceeds the threshold. ### Claim 9 (depends on 7) The method according to claim 7, wherein the generating the orientation angle based on the coordinates of the first point according to a first calculation rule comprises: generating three-dimensional coordinates of the first point in a vehicle body coordinate system based on the coordinates of the first point and the ground plane assumption principle, wherein a coordinate system origin of the vehicle body coordinate system is located inside the vehicle equipped with the image processing apparatus; and generating the orientation angle based on the three-dimensional coordinates of the first point. ### Claim 10 (depends on 7) The method according to claim 7, wherein the generating the orientation angle based on the coordinates of the first point according to a second calculation rule comprises: generating the location information of the side line of the first vehicle based on the coordinates of the first point and the first angle of the first vehicle, and generating coordinates of a vanishing point based on the location information of the side line of the first vehicle and location information of a vanishing line of the first image, wherein the vanishing point is an intersection point between the side line of the first vehicle and the vanishing line of the first image; and generating the orientation angle based on the coordinates of the vanishing point and a two-point perspective principle. ### Claim 11 An image processing apparatus, comprising at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the image processing apparatus to: obtain a first image including a first vehicle; input the first image into an image processing network to obtain a first result output by the image processing network, wherein when a side face of the first vehicle is exposed in the first image, the first result comprises location information of a two-dimensional (2D) bounding frame of the first vehicle, coordinates of a wheel of the first vehicle, and a first angle of the first vehicle; the first angle of the first vehicle indicates an included angle between a side line of the first vehicle and a first axis of the first image; the side line of the first vehicle is an intersection line between the exposed side face of the first vehicle and a ground plane on which the first vehicle is located; and the first axis of the first image is parallel to one side of the first image; and generate location information of a three-dimensional (3D) outer bounding box of the first vehicle based on the location information of the 2D bounding frame of the first vehicle, the coordinates of the wheel, and the first angle, wherein the location information of the 3D outer bounding box of the first vehicle comprises coordinates of at least two first points, the at least two first points are all located on a side of the 3D outer bounding box of the first vehicle, the side of the 3D outer bounding box of the first vehicle is located using two first points in the at least two first points, and the 3D outer bounding box of the first vehicle is located using the coordinates of the at least two first points. ### Claim 12 (depends on 11) The apparatus according to claim 11, wherein when only the side face of the first vehicle is exposed in the first image, the at least two first points comprise two intersection points between the side line of the first vehicle and the 2D bounding frame of the first vehicle. ### Claim 13 (depends on 12) The apparatus according to claim 12, wherein the generation of the location information of the 3D outer bounding box of the first vehicle based on the location information of the 2D bounding frame of the first vehicle, the coordinates of the wheel, and the first angle comprises: generating location information of the side line of the first vehicle based on the coordinates of the wheel of the first vehicle and the first angle of the first vehicle; and generating coordinates based on the location information of the side line of the first vehicle and the location information of the 2D bounding frame of the first vehicle, to obtain the coordinates of the at least two first points. ### Claim 14 (depends on 11) The apparatus according to claim 11, wherein when the side face and a main face of the first vehicle are exposed in the first image, the first result further comprises location information of a boundary of the first vehicle and a second angle of the first vehicle; the boundary is a boundary between the side face and the main face; the main face of the first vehicle is a front face or rear face of the first vehicle; the second angle of the first vehicle indicates an included angle between a main side line of the first vehicle and the first axis of the first image; and the main side line of the first vehicle is an intersection line between the exposed main face of the first vehicle and the ground plane on which the first vehicle is located; and the at least two first points comprise a first intersection point, a second intersection point, and a third intersection point; the first intersection point is an intersection point between the side line of the first vehicle and the boundary of the first vehicle, and the first intersection point is a vertex of the 3D outer bounding box of the first vehicle; the second intersection point is an intersection point between the side line of the first vehicle and the 2D bounding frame of the first vehicle; and the third intersection point is an intersection point between the main side line of the first vehicle and the 2D bounding frame of the first vehicle. ### Claim 15 (depends on 14) The apparatus according to claim 14, wherein the generation of the location information of the 3D outer-boundary box of the first vehicle based on the location information of the 2D bounding frame of the first vehicle, the coordinates of the wheel, and the first angle comprises: generating location information of the side line of the first vehicle based on the coordinates of the wheel of the first vehicle and the first angle of the first vehicle; generating coordinates of the first intersection point based on the location information of the side line of the first vehicle and the location information of the boundary of the first vehicle; generating coordinates of the second intersection point based on the location information of the side line of the first vehicle and the location information of the 2D bounding frame of the first vehicle; generating location information of the main side line of the first vehicle based on the coordinates of the first intersection point and the second angle of the first vehicle; and generating coordinates of the third intersection point based on the location information of the main side line of the first vehicle and the location information of the 2D bounding frame of the first vehicle. ### Claim 16 (depends on 11) The apparatus according to claim 11, wherein the instructions, when executed by the at least one processor, further cause the apparatus to: when the side face of the first vehicle is exposed in the first image, generate an orientation angle of the first vehicle relative to a vehicle equipped with the image processing apparatus based on coordinates of a first point. ### Claim 17 (depends on 16) The apparatus according to claim 16, wherein the instructions, when executed by the at least one processor, further cause the apparatus to generate a distance between the first point and the vehicle equipped with the image processing apparatus based on the coordinates of the first point and a ground plane assumption principle; and the generation of the orientation angle based on the coordinates of the first point comprises: upon determining, based on the distance between the first point and the vehicle equipped with the image processing apparatus, that a distance between the first vehicle and the vehicle equipped with the image processing apparatus does not exceed a threshold, generating the orientation angle based on the coordinates of the first point according to a first calculation rule; or upon determining, based on the distance between the first point and the vehicle equipped with the image processing apparatus, that a distance between the first vehicle and the vehicle equipped with the image processing apparatus exceeds a threshold, generating the orientation angle based on the coordinates of the first point according to a second calculation rule, wherein the second calculation rule and the first calculation rule are different. ### Claim 18 (depends on 17) The apparatus according to claim 17, wherein when a distance between any one of the at least two first points and the vehicle equipped with the image processing apparatus does not exceed the threshold, it is determined that the distance between the first vehicle and the vehicle equipped with the image processing apparatus does not exceed the threshold; or when a distance between any one of the at least two first points and the vehicle equipped with the image processing apparatus exceeds the threshold, it is determined that the distance between the first vehicle and the vehicle equipped with the image processing apparatus exceeds the threshold. ### Claim 19 (depends on 17) The apparatus according to claim 17, wherein the generation of the orientation angle based on the coordinates of the first point according to the first calculation rule comprises: generating three-dimensional coordinates of the first point in a vehicle body coordinate system based on the coordinates of the first point and the ground plane assumption principle, wherein a coordinate system origin of the vehicle body coordinate system is located inside the vehicle equipped with the image processing apparatus; and generating the orientation angle based on the three-dimensional coordinates of the first point. ### Claim 20 A non-transitory computer-readable storage medium, storing a program that, when executed by a computer, enable the computer to perform operations comprising: obtaining a first image comprising a first vehicle; inputting the first image into an image processing network to obtain a first result output by the image processing network, wherein when a side face of the first vehicle is exposed in the first image, the first result comprises location information of a two-dimensional (2D) bounding frame of the first vehicle, coordinates of a wheel of the first vehicle, and a first angle of the first vehicle; the first angle of the first vehicle indicates an included angle between a side line of the first vehicle and a first axis of the first image; the side line of the first vehicle is an intersection line between the exposed side face of the first vehicle and a ground plane on which the first vehicle is located; and the first axis of the first image is parallel to one side of the first image; and generating location information of a three-dimensional (31) outer bounding box of the first vehicle based on the location information of the 2D bounding frame of the first vehicle, the coordinates of the wheel, and the first angle, wherein the location information of the 3D outer bounding box of the first vehicle comprises coordinates of at least two first points, the at least two first points are all located on a side of the 3D outer bounding box of the first vehicle, the side of the 3D outer bounding box of the first vehicle is located using two first points in the at least two first points, and the 3D outer bounding box of the first vehicle is located using the coordinates of the at least two first points.