# US 11,308,639 B2 · Tool and method for annotating a human pose in 3D point cloud data > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,308,639 B2 - Document ID: US11308639B2 - Status: Granted - Kind: B2 - Application number: 16/692,901 - Filed: 2019-11-22 - Published: 2020-09-17 - Granted: 2022-04-19 - Earliest claimed: 2019-03-12 - Assignees: Volvo Car Corporation - Applicants: Volvo Car Corporation - Inventors: Saudin Botonjic; Sihao Ding; Andreas Wallin - Web page: https://patentgrep.com/patent/US11308639B2 ## CPC - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G01S 7/4802: {using analysis of echo signal for target characterisation; Target signature; Target cross-section} - G01S 7/4808: {Evaluating distance, position or velocity data} - G01S 17/04: Systems determining the presence of a target - G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders - G01S 17/89: for mapping or imaging - G01S 17/931: of land vehicles - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 18/2413: based on distances to training or reference patterns - G06T 7/73: using feature-based methods - G06T 15/20: Perspective computation - G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition} - G06T 2200/24: involving graphical user interfaces [GUIs] - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/20101: Interactive definition of point of interest, landmark or seed - G06T 2207/20132: Image cropping - G06T 2207/30196: Human being; Person - G06T 2219/004: Annotating, labelling - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method and apparatus for annotating point cloud data. An apparatus may be configured to cause display of the point cloud data, label points in the point cloud data with a plurality of annotation points, the plurality of annotation points corresponding to points on a human body, move, in response to a user input, one or more of the annotation points to define a human pose and create annotated point cloud data, and output the annotated point cloud data. ## Claims ### Claim 1 A method comprising: causing, by one or more processors, display of at least one frame of multiple frames of point cloud data; labeling, by the one or more processors, points in the at least one frame of the point cloud data with a plurality of annotation points, the plurality of annotation points corresponding to points on a human body; causing, by the one or more processors, a video display of the multiple frames of the point cloud data, wherein the video display may be controlled to be forward, fast forward, pause, and reverse; moving, by the one or more processors, and in response to a user input, one or more of the annotation points to define a human pose and create annotated point cloud data; and outputting, by the one or more processors, the annotated point cloud data for training by a neural network. ### Claim 2 (depends on 1) The method of claim 1, wherein labeling, by the one or more processors, points in the at least one frame of the point cloud data with the plurality of annotation points comprises: estimating, by the one or more processors, a position of a potential human pose in the at least one frame of the point cloud data; and labeling, by the one or more processors, the annotation points to correspond with the estimated position of the potential human pose. ### Claim 3 (depends on 1) The method of claim 1, wherein moving, by the one or more processors, and in response to the user input, one or more of the annotation points to define the human pose and create annotated point cloud data comprises: receiving, by the one or more processors, a selection of a single annotation point of the plurality of annotation points; moving, by the one or more processors, and in response to the user input, only the single annotation point to define a portion of the human pose. ### Claim 4 (depends on 1) The method of claim 1, wherein moving, by the one or more processors, and in response to the user input, one or more of the annotation points to define the human pose and create annotated point cloud data comprises: receiving, by the one or more processors, a selection of two or more annotation points of the plurality of annotation points; moving, by the one or more processors, and in response to the user input, only the two or more annotation points to define a portion of the human pose. ### Claim 5 (depends on 1) The method of claim 1, further comprising: displaying an image that corresponds to the at least one frame of the point cloud data at the same time as displaying the point cloud data; cropping, by the one or more processors, the point cloud data and the image around a region of a potential human pose in the point cloud data; and causing, by the one or more processors, display of the cropped region. ### Claim 6 (depends on 5) The method of claim 5, further comprising: causing, by the one or more processors, display of the cropped region at a plurality of perspectives. ### Claim 7 (depends on 1) The method of claim 1, wherein the plurality of annotation points includes annotation points corresponding to a top of a head, a center of a neck, a right hip, a left hip, a right shoulder, a right elbow, a right hand, a right knee, a right foot, a left shoulder, a left elbow, a left hand, a left knee, and a left foot, and wherein groups of the annotation points correspond to limbs of a person, the method further comprising: causing, by the one or more processors, display of lines between the annotation points to define the limbs, including displaying different limbs using different colors. ### Claim 8 (depends on 1) The method of claim 1, further comprising: adding, by the one or more processors, an action label for each of the multiple frames of the point cloud data and to the human pose. ### Claim 9 (depends on 1) The method of claim 1, further comprising: training, by the one or more processors, the neural network with the annotated point cloud data, wherein the neural network is configured to estimate a pose of a person from LiDAR point cloud data. ### Claim 10 An apparatus comprising: a memory configured to store point cloud data; and one or more processors in communication with the memory, the one or more processors configured to: cause display of at least one frame of multiple frames of the point cloud data; label points in the at least one frame of the point cloud data with a plurality of annotation points, the plurality of annotation points corresponding to points on a human body; cause a video display of the multiple frames of the point cloud data, wherein the video display may be controlled to be forward, fast forward, pause, and reverse; move, in response to a user input, one or more of the annotation points to define a human pose and create annotated point cloud data; and output the annotated point cloud data for training by a neural network. ### Claim 11 (depends on 10) The apparatus of claim 10, wherein to label points in the at least one frame of the point cloud data with the plurality of annotation points, the one or more processors are further configured to: estimate a position of a potential human pose in the at least one frame of the point cloud data; and label the annotation points to correspond with the estimated position of the potential human pose. ### Claim 12 (depends on 10) The apparatus of claim 10, wherein to move, in response to the user input, one or more of the annotation points to define the human pose and create annotated point cloud data, the one or more processors are further configured to: receive a selection of a single annotation point of the plurality of annotation points; move, in response to the user input, only the single annotation point to define a portion of the human pose. ### Claim 13 (depends on 10) The apparatus of claim 10, wherein to move, in response to the user input, one or more of the annotation points to define the human pose and create annotated point cloud data, the one or more processors are further configured to: receive a selection of two or more annotation points of the plurality of annotation points; move, in response to the user input, only the two or more annotation points to define a portion of the human pose. ### Claim 14 (depends on 10) The apparatus of claim 10, wherein the one or more processors are further configured to: display an image that corresponds to the at least one frame of the point cloud data at the same time as displaying the point cloud data; crop the point cloud data and the image around a region of a potential human pose in the point cloud data; and cause display of the cropped region. ### Claim 15 (depends on 14) The apparatus of claim 14, wherein the one or more processors are further configured to: cause display of the cropped region at a plurality of perspectives. ### Claim 16 (depends on 10) The apparatus of claim 10, wherein the plurality of annotation points include annotation points corresponding to a top of a head, a center of a neck, a right hip, a left hip, a right shoulder, a right elbow, a right hand, a right knee, a right foot, a left shoulder, a left elbow, a left hand, a left knee, and a left foot, and wherein groups of the annotation points correspond to limbs of a person, and wherein the one or more processors are further configured to: cause display of lines between the annotation points to define the limbs, including cause display of different limbs using different colors. ### Claim 17 (depends on 10) The apparatus of claim 10, wherein the one or more processors are further configured to: add an action label for each of the multiple frames of the point cloud data and to the human pose. ### Claim 18 (depends on 10) The apparatus of claim 10, wherein the one or more processors are further configured to: train the neural network with the annotated point cloud data, wherein the neural network is configured to estimate a pose of a person from LiDAR point cloud data. ### Claim 19 An apparatus comprising: means for causing display of at least one frame of multiple frames of point cloud data; means for labeling points in the at least one frame of the point cloud data with a plurality of annotation points, the plurality of annotation points corresponding to points on a human body; means for causing a video display of the multiple frames of the point cloud data, wherein the video display may be controlled to be forward, fast forward, pause, and reverse; means for moving, in response to a user input, one or more of the annotation points to define a human pose and create annotated point cloud data; and means for outputting the annotated point cloud data for training by a neural network. ### Claim 20 A non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors to: cause display of at least one frame of multiple frames of point cloud data; label points in the at least one frame of the point cloud data with a plurality of annotation points, the plurality of annotation points corresponding to points on a human body; cause a video display of the multiple frames of the point cloud data, wherein the video display may be controlled to be forward, fast forward, pause, and reverse; move, in response to a user input, one or more of the annotation points to define a human pose and create annotated point cloud data; and output the annotated point cloud data for training by a neural network.