# US 11,308,315 B2 · Information processing apparatus, control method, and program > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,308,315 B2 - Document ID: US11308315B2 - Status: Granted - Kind: B2 - Application number: 16/650,544 - Filed: 2017-09-27 - Published: 2020-09-10 - Granted: 2022-04-19 - Earliest claimed: 2017-09-27 - Assignees: NEC Corporation - Applicants: NEC Corporation - Inventors: Ryoma Oami - Web page: https://patentgrep.com/patent/US11308315B2 ## CPC - G06T 7/62: of area, perimeter, diameter or volume - G01B 11/022: {by means of tv-camera scanning} - G01B 11/0608: {Height gauges} - G06T 7/292: Multi-camera tracking - G06T 7/60: Analysis of geometric attributes - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06V 10/462: {Salient features, e.g. scale invariant feature transforms [SIFT]} - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands - G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition} - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G06T 2207/10016: Video; Image sequence - G06T 2207/30196: Human being; Person - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract An information processing apparatus (2000) includes a detection unit (2020), a state estimation unit (2040), and a height estimation unit (2080). The detection unit (2020) detects a target person from a video frame. The state estimation unit (2040) estimates a state of the detected target person. The height estimation unit (2080) estimates a height of the person on the basis of a height of the target person in the video frame in a case where the estimated state satisfies a predetermined condition. ## Claims ### Claim 1 An information processing system comprising: a processor; and memory storing executable instructions that, when executed by the processor, causes the processor to perform as: a detection unit that detects a person from a video frame; a state estimation unit that estimates a state of a target person using a result of the detection; a body height estimation unit that estimates a body height of the target person on a basis of a height of the target person in the video frame in a case where the state of the target person satisfies a predetermined condition, and does not estimate a body height of the target person on the basis of a height of the target person in the video frame in a case where the state of the target person does not satisfy the predetermined condition; and a temporary-height estimation unit that estimates a temporary-height of a target person on the basis of the height of the target person in the video frame, wherein the detection unit detects the person from a first video frame generated at a first time point, the information processing system further comprises: an estimated position computation unit that estimates a position of each tracking target person at the first time point using tracking information that shows a history of information related to the tracking target person detected from the video frame generated before the first time point, and an update unit that associates the person detected from a first video frame with the tracking target person by comparing a position of each person detected from the first video frame in a real world or an image with a position of each tracking target person in the real world or the image at the first time point, and updates the tracking information using a result of the association, wherein the update unit estimates the position of the person detected from the first video frame in the real world using the estimated body height of the person in a case where the person is upright, and estimates the position of the person detected from the first video frame in the real world using the temporary-height of the person in a case where the person is not upright. ### Claim 2 (depends on 1) The information processing system according to claim 1, wherein the state which the target person may take includes an upright movement state, an upright standstill state, and a non-upright standstill state, and the predetermined condition is satisfied in a case where the state of the target person is the upright movement state or the upright standstill state. ### Claim 3 (depends on 1) The information processing system according to claim 1, wherein the body height estimation unit computes an estimated body height of the target person by computing a height of the target person in a real world from each of a plurality of video frames in which the estimated state satisfies the predetermined condition, and performing a statistic process on the plurality of computed heights. ### Claim 4 (depends on 3) The information processing system according to claim 3, wherein the body height estimation unit performs the statistic process on the height of the target person in the real world computed from each of the plurality of video frames by applying a weight based on an angle of depression, which is determined by the target person and a direction of the camera generating the video frame, and resolution of the target person in the video frame. ### Claim 5 (depends on 3) The information processing system according to claim 3, wherein the state which the target person may take includes an upright movement state, an upright standstill state, and a non-upright standstill state, and the state estimation unit further estimates at least one of a degree of openness of legs of the target person or a direction of the target person, and in the statistic process, in a case where the state of the target person is the upright movement state, a higher weight is applied to the height of the target person in the real world as the degree of openness of the legs of the target person is decreased, or a higher weight is applied to the height of the target person in the real world as a degree to which the direction of the target person is different from a direction of the camera capturing the video frame is increased. ### Claim 6 (depends on 3) The information processing system according to claim 3, wherein the state estimation unit computes the height of the target person in the real world from each of a plurality of video frames in which the target person is included, and decides a threshold for determining whether or not the person is upright on the basis of the computed height, and for the video frame that is subsequently generated, determines whether or not the target person is upright in the video frame by computing the height of the target person in the real world and comparing the height with the threshold, and the predetermined condition is satisfied in a case where the target person is upright. ### Claim 7 (depends on 3) The information processing system according to claim 3, wherein the state which the target person may take includes an upright movement state, an upright standstill state, and a non-upright standstill state, and the body height estimation unit estimates the body height by performing weighting in consideration of the state of the target person, computes an approximate value of the body height using a time of the upright movement state as a reference, and estimates the body height by increasing a weight of an observation value that is stable at a value close to the approximate value of the body height before and after the upright movement state. ### Claim 8 (depends on 1) The information processing system according to claim 1, wherein the state which the target person may take includes an upright movement state, an upright standstill state, and a non-upright standstill state, and even in a case where a foot position of the person is not seen, the temporary-height estimation unit computes an observation value of the temporary-height even from only a head part position in the upright standstill state and the non-upright standstill state by assuming that the foot position in the image does not move. ### Claim 9 (depends on 1) The information processing system according to claim 1, wherein the detection unit generates a person detection result for each camera by performing a person detection process on each of the images generated by two or more cameras, and for a tracking target person who is present in a region observable by a plurality of cameras, the temporary-height estimation unit and the body height estimation unit compare position coordinates that are included in a tracking result based on a detection result of the image that is acquired from each camera with a generation time difference less than a predetermined time, and correct observation values of the body height and the temporary-height such that a distance between the coordinates is decreased. ### Claim 10 (depends on 1) The information processing system according to claim 1, wherein the temporary-height estimation unit and the body height estimation unit set the predetermined time to be short to an extent that the person can be regarded as being present at the same position when the person is in an upright movement state, and set the predetermined time to be long to an extent that the person can be regarded as being present at the same position and a change in pose can be regarded as not occurring in a case where the person is in an upright standstill state or a non-upright standstill state. ### Claim 11 A control method executed by a computer, the method comprising: detecting a person from a video frame; estimating a state of a target person using a result of the detection; estimating a body height of the target person on the basis of a height of the target person in the video frame in a case where the state of the target person satisfies a predetermined condition, and does not estimate a body height of the target person on the basis of a height of the target person in the video frame in a case where the state of the target person does not satisfy the predetermined condition; and estimating a temporary-height of the target person on the basis of the height of the target person in the video frame, wherein the detection detects the person from a first video frame generated at a first time point, the information processing method further comprising: estimating a position of each tracking target person at the first time point using tracking information that shows a history of information related to the tracking target person detected from the video frame generated before the first time point, and associating the person detected from the first video frame with the tracking target person by comparing a position of each person detected from the first video frame in a real world or the image with a position of each tracking target person in the real world or the image at the first time point, and updates the tracking information using a result of the association, and estimating the position of the person detected from the first video frame in the real world using the estimated body height of the person in a case where the person is upright, and estimates the position of the person detected from the first video frame in the real world using the temporary-height of the person in a case where the person is not upright. ### Claim 12 (depends on 11) The control method according to claim 11, wherein the state which the target person may take includes an upright movement state, an upright standstill state, and a non-upright standstill state, and the predetermined condition is satisfied in a case where the state of the target person is the upright movement state or the upright standstill state. ### Claim 13 (depends on 11) The control method according to claim 11, wherein an estimated body height of the target person is computed by computing a height of the target person in a real world from each of a plurality of video frames in which the estimated state satisfies the predetermined condition, and performing a statistic process on the plurality of computed heights. ### Claim 14 (depends on 13) The control method according to claim 13, wherein the statistic process on the height of the target person in the real world computed from each of the plurality of video frames is performed by applying a weight based on an angle of depression, which is determined by the target person and a direction of the camera generating the video frame and resolution of the target person in the video frame. ### Claim 15 (depends on 13) The control method according to claim 13, wherein the state which the target person may take includes an upright movement state, an upright standstill state, and a non-upright standstill state, in the state estimation, at least one of a degree of openness of legs of the target person or a direction of the target person is further estimated, and in the statistic process, in a case where the state of the target person is the upright movement state, a higher weight is applied to the height of the target person in the real world as the degree of openness of the legs of the target person is decreased, or a higher weight is applied to the height of the target person in the real world as a degree to which the direction of the target person is different from a direction of the camera capturing the video frame is increased. ### Claim 16 (depends on 13) The control method according to claim 13, wherein in the state estimation, the height of the target person in the real world is computed from each of a plurality of video frames in which the target person is included, and a threshold for determining whether or not the person is upright is decided on the basis of the computed height, and for the video frame that is subsequently generated, whether or not the target person is upright in the video frame is determined by computing the height of the target person in the real world and comparing the height with the threshold, and the predetermined condition is satisfied in a case where the target person is upright. ### Claim 17 (depends on 13) The control method according to claim 13, wherein the state which the target person may take includes an upright movement state, an upright standstill state, and a non-upright standstill state, and in the body height estimation, the body height is estimated by performing weighting in consideration of the state of the target person, an approximate value of the body height is computed using a time of the upright movement state as a reference, and the body height is estimated by increasing a weight of an observation value that is stable at a value close to the approximate value of the body height before and after the upright movement state. ### Claim 18 (depends on 11) A non-transitory computer-readable storage medium storing a program causing a computer to execute: the control method of claim 11.