# US 12,602,819 B2 · Image processing apparatus, feature map generating apparatus, learning model generation apparatus, image processing method, and computer-readable recording medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,819 B2 - Document ID: US12602819B2 - Status: Granted - Kind: B2 - Application number: 18/274,102 - Filed: 2021-06-28 - Published: 2024-10-10 - Granted: 2026-04-14 - Earliest claimed: 2021-06-28 - Assignees: NEC Corporation - Applicants: NEC Corporation - Inventors: Yadong Pan - Web page: https://patentgrep.com/patent/US12602819B2 ## CPC - G06T 7/73: using feature-based methods - G06V 10/771: Feature selection, e.g. selecting representative features from a multi-dimensional feature space - G06T 2207/20044: Skeletonization; Medial axis transform - G06T 2207/20081: Training; Learning - G06T 2207/30196: Human being; Person - G06T 2207/30232: Surveillance - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract An image processing apparatus includes a feature map generation unit and a grouping unit. The feature map generation unit generate a first feature map that specifies a horizontal location of the person in an image in a horizontal direction and a second feature map that specifies a vertical location of the person in the image in a vertical direction. The grouping unit uses locations of joints detected in the image in the horizontal direction and vertical direction, the first feature map, and the second feature map, to group each of the joints to a corresponding person. ## Claims ### Claim 1 An image processing apparatus comprising: at least one memory storing instructions; and at least one processor configured to execute the instructions to: generate a first feature map that specifies horizontal locations of people in an image in a horizontal direction and a second feature map that specifies vertical locations of the people in the image in a vertical direction, and use locations of joints detected in the image in the horizontal direction and the vertical direction, the first feature map, and the second feature map, to group each of the joints to corresponding people, among the people, wherein the at least one processor is further configured to execute the instructions to: generate, as the first feature map, a map that is composed of the same number of pixels as the pixels constituting the image, and assign numerical values of the people in the horizontal direction to each of the pixels in areas corresponding to the people, generate, as the second feature map, a map that is composed of the same number of pixels as the pixels constituting the image, and assign numerical values of the people in the vertical direction to each of the pixels in areas corresponding to the people, specify, for the joints detected from the image, for each of the people in the image, the numerical value of the pixel corresponding to the joint in the first feature map and the numerical value of the pixel corresponding to the joint in the second feature map, calculate, for the joints detected from the image, for each of the people in the image, a distance between the joint and a corresponding person, by using the two numerical values specified, the numerical value indicating the location of the corresponding person in the horizontal direction, and the numerical value indicating the location of the corresponding person in the vertical direction, and determine, for the joints detected from the image, the corresponding person corresponding to the joint based on the calculated distance, and groups based on the determination result. ### Claim 2 (depends on 1) The image processing apparatus according to claim 1, wherein a numerical value indicating a location of a person in the horizontal direction is a value that indicates the ratio of a distance in the horizontal direction from an origin of the first feature map to a reference point of the person with respect to a horizontal length of the first feature map, and wherein a numerical value indicating a location of the person in the vertical direction is a value that indicates the ratio of a distance in the vertical direction from an origin of the second feature map to the reference point of the person with respect to a vertical length of the second feature map. ### Claim 3 (depends on 1) The image processing apparatus according to claim 1, wherein the at least one processor is further configured to execute the instructions to: generate the first feature map and the second feature map by using a learning model in which a relationship between an image, the first feature map, and the second feature map is machine-learned. ### Claim 4 (depends on 1) The image processing apparatus according to claim 1, wherein the at least one processor is further configured to execute the instructions to: determine the corresponding person corresponding to the joint based on the calculated distance not exceeding a threshold value and that a plurality of joints of the same type do not correspond to the same person. ### Claim 5 An image processing method comprising: generating a first feature map that specifies horizontal locations of people in an image in a horizontal direction and a second feature map that specifies vertical locations of the people in the image in a vertical direction, using locations of joints detected in the image in the horizontal direction and the vertical direction, the first feature map, and the second feature map, to group each of the joints to corresponding people, among the people, wherein the generating the first feature map comprises generating, as the first feature map, a map that is composed of the same number of pixels as the pixels constituting the image, and assigns numerical values of the people in the horizontal direction to each of the pixels in areas corresponding to the people, wherein the generating the second feature map comprises generating, as the second feature map, a map that is composed of the same number of pixels as the pixels constituting the image, and assigns numerical values of the people in the vertical direction to each of the pixels in areas corresponding to the people, and wherein the method further comprises: specifying, for the joints detected from the image, for each of the people in the image, the numerical value of the pixel corresponding to the joint in the first feature map and the numerical value of the pixel corresponding to the joint in the second feature map; calculating, for the joints detected from the image, for each of the people in the image, a distance between the joint and a corresponding person, by using the two numerical values specified, the numerical value indicating the location of the corresponding person in the horizontal direction, and the numerical value indicating the location of the corresponding person in the vertical direction; and determining, for the joints detected from the image, the corresponding person corresponding to the joint based on the calculated distance, and groups based on the determination result. ### Claim 6 (depends on 5) The image processing method according to claim 5, wherein a numerical value indicating a location of a person in the horizontal direction is a value that indicates the ratio of a distance in the horizontal direction from an origin of the first feature map to a reference point of the person with respect to a horizontal length of the first feature map, and wherein a numerical value indicating a location of the person in the vertical direction is a value that indicates the ratio of a distance in the vertical direction from an origin of the second feature map to the reference point of the person with respect to a vertical length of the second feature map. ### Claim 7 (depends on 5) The image processing method according to claim 5, wherein, in the feature map generation, generating the first feature map and the second feature map by using a learning model in which a relationship between an image, the first feature map, and the second feature map is machine-learned. ### Claim 8 (depends on 5) The image processing method according to claim 5, wherein, in the grouping, determining the corresponding person corresponding to the joint based on the calculated distance not exceeding a threshold value and that a plurality of joints of the same type do not correspond to the same person. ### Claim 9 A non-transitory computer-readable recording medium that includes a program, the program including instructions that cause a computer to carry out: generating a first feature map that specifies horizontal locations of people in an image in a horizontal direction and a second feature map that specifies vertical locations of the people in the image in a vertical direction, using locations of joints detected in the image in the horizontal direction and the vertical direction, the first feature map, and the second feature map, to group each of the joints to corresponding people, among the people, wherein the generating the first feature map comprises generating, as the first feature map, a map that is composed of the same number of pixels as the pixels constituting the image, and assigns numerical values of the people in the horizontal direction to each of the pixels in areas corresponding to the people, wherein the generating the second feature map comprises generating, as the second feature map, a map that is composed of the same number of pixels as the pixels constituting the image, and assigns numerical values of the people in the vertical direction to each of the pixels in areas corresponding to the people, and wherein the program further includes instructions that cause the computer to carry out: specifying, for the joints detected from the image, for each of the people in the image, the numerical value of the pixel corresponding to the joint in the first feature map and the numerical value of the pixel corresponding to the joint in the second feature map; calculating, for the joints detected from the image, for each of the people in the image, a distance between the joint and a corresponding person, by using the two numerical values specified, the numerical value indicating the location of the corresponding person in the horizontal direction, and the numerical value indicating the location of the corresponding person in the vertical direction; and determining, for the joints detected from the image, the corresponding person corresponding to the joint based on the calculated distance, and groups based on the determination result. ### Claim 10 (depends on 9) The non-transitory computer-readable recording medium according to claim 9, wherein a numerical value indicating a location of a person in the horizontal direction is a value that indicates the ratio of a distance in the horizontal direction from an origin of the first feature map to a reference point of the person with respect to a horizontal length of the first feature map, and wherein a numerical value indicating a location of the person in the vertical direction is a value that indicates the ratio of a distance in the vertical direction from an origin of the second feature map to the reference point of the person with respect to a vertical length of the second feature map. ### Claim 11 (depends on 9) The non-transitory computer-readable recording medium according to claim 9, wherein, in the feature map generation, generating the first feature map and the second feature map by using a learning model in which a relationship between an image, the first feature map, and the second feature map is machine-learned. ### Claim 12 (depends on 9) The non-transitory computer-readable recording medium according to claim 9, wherein, in the grouping, determining the corresponding person corresponding to the joint based on the calculated distance not exceeding a threshold value and that a plurality of joints of the same type do not correspond to the same person.