# US 12,182,197 B2 · Image processing apparatus, image processing method, and non-transitory storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,182,197 B2 - Document ID: US12182197B2 - Status: Granted - Kind: B2 - Application number: 18/009,155 - Filed: 2020-06-10 - Published: 2023-07-06 - Granted: 2024-12-31 - Earliest claimed: 2020-06-10 - Assignees: NEC Corporation - Applicants: NEC Corporation - Inventors: Noboru Yoshida; Yadong Pan; Ryo Kawai; Jianquan Liu; Shoji Nishimura - Web page: https://patentgrep.com/patent/US12182197B2 ## CPC - G06F 16/5854: {using shape and object relationship} - G06F 16/532: Query formulation, e.g. graphical querying - G06T 7/62: of area, perimeter, diameter or volume - G06T 7/73: using feature-based methods - G06T 7/75: {involving models} - G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components - G06V 10/761: {Proximity, similarity or dissimilarity measures} - G06T 2207/20044: Skeletonization; Medial axis transform - G06T 2207/30196: Human being; Person - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract The present invention provides an image processing apparatus including: an image acquisition unit acquiring a candidate of a query image; a skeletal structure detection unit detecting a two-dimensional skeletal structure of a person included in a candidate of the query image; a query evaluation unit computing an evaluation value of a candidate of the query image, based on a detection result of the two-dimensional skeletal structure; a selection unit selecting a query image out of candidates of the query image, based on the evaluation value; a feature value computation unit computing a feature value of the two-dimensional skeletal structure detected from the query image; and a retrieval unit retrieving an analysis target image including a person in a pose similar to a pose of a person included in the query image out of analysis target images, based on a degree of similarity of the computed feature value. ## Claims ### Claim 1 An image processing apparatus comprising: at least one memory storing one or more instructions; and at least one processor configured to execute the one or more instructions to: acquire a plurality of query image candidates; detect, for each query image candidate, a two-dimensional skeletal structure of a person included in the each query image candidate; compute, for each query image candidate, an evaluation value based on the two-dimensional skeletal structure of the person included in the each query image candidate, the evaluation value of each query image candidate indicating an extent to which a feature value computable from the each query image candidate is indicative of a pose of the person; select, from among the query image candidates, a query image to be used for image search, based on the evaluation value computed for each query image candidate; compute a feature value of the two-dimensional skeletal structure of the person included in the query image candidate selected as the query image; and perform the image search by retrieving, from among a plurality of images, a target image including a feature value with a highest degree of similarity to the computed feature value, as a target image including a pose most similar to the pose of the person in the query image candidate selected as the query image, wherein the evaluation value of each query image candidate is computed by calculating a height pixel count of a standing person in the each query image candidate, and normalizing the calculated height pixel count to yield the evaluation value. ### Claim 2 (depends on 1) The image processing apparatus according to claim 1, wherein the processor is further configured to execute the one or more instructions to: output the evaluation value of each query image candidate; accept user input specifying the query image from among the query image candidates; and select the query image from among the query image candidates as specified by the user input. ### Claim 3 (depends on 2) The image processing apparatus according to claim 2, wherein the processor is further configured to execute the one or more instructions to: output the evaluation value for each query image candidate in such a way that the values for the query image candidates are comparable; accept user input specifying one or more of the query images from among the query image candidates; and select the query one or more of the query images from among the query image candidates as specified by the user input. ### Claim 4 (depends on 1) The image processing apparatus according to claim 1, wherein the processor is further configured to execute the one or more instructions to select, from among the query image candidates, a query image candidate that the evaluation value of which satisfies a criterion, as the query image. ### Claim 5 (depends on 1) The image processing apparatus according to claim 1, wherein the processor is further configured to execute the one or more instructions to: extract a plurality of keypoints of a body; and compute, for each query image candidate, the evaluation value based on at least one of a number of the extracted keypoints and a confidence level of each extracted keypoint. ### Claim 6 (depends on 5) The image processing apparatus according to claim 5, wherein the processor is further configured to execute the one or more instructions to compute, for each query image candidate, the evaluation value based on a weight value of each keypoint that is set based on user input. ### Claim 7 (depends on 6) The image processing apparatus according to claim 6, wherein the processor is further configured to execute the one or more instructions to: determine part of the keypoints by accepting user input for enclosing part of the body of the person by an image frame in indicating the body and determining each keypoint included in the frame, accepting user input for specifying the part of the keypoints in an image indicating the body of the person and the keypoints of the body and determining the specified part of the keypoints, or accepting user input for specifying part of the body of the person by a name of the part of the body and determining the keypoints included in the specified part of the body; and set the weight value of each keypoint of the determined part of the keypoints based on user input. ### Claim 8 (depends on 1) The image processing apparatus according to claim 1, wherein the processor is further configured to execute the one or more instructions to compute, for each query image candidate, the evaluation value based on a size of the person in an image. ### Claim 9 (depends on 1) The image processing system according to claim 1, wherein the height pixel count is calculated by totaling lengths of bones from head to foot out of bones of the two-dimensional skeletal structure. ### Claim 10 (depends on 1) The image processing system according to claim 1, wherein the height pixel count is calculated by using a human-body model including a relationship between a length of each of a plurality of bones of the two-dimensional skeletal structure and a length of a whole body, where the length of the whole body is a height in two-dimensional image space. ### Claim 11 (depends on 1) The image processing system according to claim 1, wherein the height pixel count is calculated by fitting a three-dimensional human-body model to a two-dimensional skeletal structure. ### Claim 12 An image processing method comprising, performed by a computer and comprising: acquiring a plurality of query image candidates; detecting, for each query image candidate, a two-dimensional skeletal structure of a person included in the each query image candidate; computing, for each query image candidate, an evaluation value, based on the two-dimensional skeletal structure of the person included in the each query image candidate, the evaluation value of each query image candidate indicating an extent to which a feature value computable from the each query image candidate is indicative of a pose of the person; selecting, from among the query image candidates, a query image to be used for image search, based on the evaluation value computed for each query image candidate; computing a feature value of the two-dimensional skeletal structure of the person included in the query image candidate selected as the query image; and performing the image search by retrieving, from among a plurality of images, a target image including a feature value with a highest degree of similarity to the computed feature value, as a target image including a pose most similar to the pose of the person in the query image candidate selected as the query image, wherein the evaluation value of each query image candidate is computed by calculating a height pixel count of a standing person in the each query image candidate, and normalizing the calculated height pixel count to yield the evaluation value. ### Claim 13 A non-transitory storage medium storing a program executable by a computer to: acquire a plurality of query image candidates; detect, for each query image candidate, a two-dimensional skeletal structure of a person included in the each query image candidate; compute, for each query image candidate, an evaluation value, based on the two-dimensional skeletal structure of the person included in the each query image candidate, the evaluation value of each query image candidate indicating an extent to which a feature value computable from the each query image candidate is indicative of a pose of the person; select, from among the query image candidates, a query image to be used for image search, based on the evaluation value computed for each query image candidate; compute a feature value of the two-dimensional skeletal structure of the person included in the query image candidate selected as the query image; and perform the image search by retrieving, from among a plurality of images, a target image including a feature value with a highest a degree of similarity to the computed feature value, as a target image including a pose most similar to the pose of the person in the query image candidate selected as the query image, wherein the evaluation value of each query image candidate is computed by calculating a height pixel count of a standing person in the each query image candidate, and normalizing the calculated height pixel count to yield the evaluation value.