# US 2023/0054973 A1 · Information processing apparatus, information processing method, and information processing program > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2023/0054973 A1 - Document ID: US20230054973A1 - Status: Published application - Kind: A1 - Application number: 17/792,327 - Filed: 2021-02-05 - Published: 2023-02-23 - Earliest claimed: 2020-02-06 - Assignees: Sony Group Corporation - Applicants: Sony Group Corporation - Inventors: Hayato NISHIOKA; Takanori OKU; Shinichi FURUYA - Web page: https://patentgrep.com/patent/US20230054973A1 ## CPC - G06T 7/292: Multi-camera tracking - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G06T 7/593: from stereo images - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 7/73: using feature-based methods - G10H 1/0008: {Associated control or indicating means} - G06T 2207/10016: Video; Image sequence - G06T 2207/10024: Color image - G06T 2207/10048: Infrared image - G06T 2207/20076: Probabilistic image processing - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30196: Human being; Person - G06T 2207/30241: Trajectory - G10H 2220/455: Camera input, e.g. analyzing pictures from a video camera and using the analysis results as control data - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G10H: ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE ## Abstract An information processing apparatus (100) includes an estimation unit (132) that estimates time-series information regarding a posture of a finger on the basis of image information including an operation of the finger with respect to an object including a contact operation of the finger with respect to the object and the object. The estimation unit (132) estimates the time-series information regarding the posture of the finger by using a first machine learning model learned to estimate the time-series information regarding the posture of the finger on the basis of the image information including the operation of the finger and the object. ## Claims ### Claim 1 An information processing apparatus comprising: an estimation unit that estimates time-series information regarding a posture of a finger on the basis of image information including an operation of the finger with respect to an object including a contact operation of the finger with respect to the object and the object. ### Claim 2 (depends on 1) The information processing apparatus according to claim 1, wherein the estimation unit estimates the time-series information regarding the posture of the finger by using a first machine learning model learned to estimate the time-series information regarding the posture of the finger on the basis of the image information including the operation of the finger and the object. ### Claim 3 (depends on 1) The information processing apparatus according to claim 1, wherein the estimation unit estimates, as the time-series information regarding the posture of the finger, time-series information of a position, a speed, an acceleration, or a trajectory of a feature point of each joint of the finger or each fingertip, palm, back of hand, or wrist, or an angle, an angular velocity, or an angular acceleration of each joint of the finger. ### Claim 4 (depends on 1) The information processing apparatus according to claim 1, wherein the image information is image information photographed by a high-speed monochrome camera or a high-speed infrared camera. ### Claim 5 (depends on 1) The information processing apparatus according to claim 1, wherein the image information is a plurality of pieces of image information acquired by a plurality of cameras installed so as to photograph the object from a plurality of different directions. ### Claim 6 (depends on 5) The information processing apparatus according to claim 5, wherein the plurality of cameras is attached to a gate-shaped structure surrounding the object, and each of the plurality of pieces of image information is the plurality of pieces of image information photographed in a state where the finger is illuminated by a light source installed in the vicinity of each of the cameras. ### Claim 7 (depends on 1) The information processing apparatus according to claim 1, wherein the image information is a plurality of pieces of image information photographed by three or more cameras installed on both sides of the object and above the object. ### Claim 8 (depends on 1) The information processing apparatus according to claim 1, wherein the image information is image information photographed with a range from a fingertip of the finger to a wrist as a photographing range. ### Claim 9 (depends on 1) The information processing apparatus according to claim 1, wherein the estimation unit estimates the time-series information regarding the posture of the finger on the basis of image information of a back of a hand performing an operation of the finger. ### Claim 10 (depends on 9) The information processing apparatus according to claim 9, wherein the estimation unit estimates the time-series information regarding the posture of the finger by using a second machine learning model learned to estimate the time-series information regarding the posture of the finger on the basis of the image information of the back of the hand performing the operation of the finger. ### Claim 11 (depends on 1) The information processing apparatus according to claim 1, wherein the estimation unit estimates the time-series information regarding the posture of the finger of a user on the basis of sensing data detected by a plurality of IMU sensors installed on a thumb and a back of a hand of the user. ### Claim 12 (depends on 1) The information processing apparatus according to claim 1, wherein the estimation unit estimates the time-series information regarding the posture of the finger of a user on the basis of the image information photographed by a wearable camera attached to a wrist of the user. ### Claim 13 (depends on 12) The information processing apparatus according to claim 12, wherein the wearable camera further includes an IMU sensor, and the estimation unit estimates the time-series information regarding the posture of the finger based on sensing data detected by the IMU sensor. ### Claim 14 (depends on 1) The information processing apparatus according to claim 1, wherein the estimation unit estimates the time-series information regarding the posture of the finger in contact with the object on the basis of sensing data detected by a contact sensor that detects a contact operation of the finger with respect to the object. ### Claim 15 (depends on 14) The information processing apparatus according to claim 14, wherein the estimation unit estimates the time-series information regarding the posture of the finger in contact with the object on the basis of position information of the object before the contact operation of the finger with respect to the object is performed, a change amount of a position of the object before and after the contact operation of the finger with respect to the object is performed, and contact position information of the finger with respect to the object. ### Claim 16 (depends on 14) The information processing apparatus according to claim 14, wherein the estimation unit estimates an angle of a PIP joint of the finger on the basis of a distance between an MP joint and the PIP joint of the finger, a distance between the PIP joint and a fingertip of the finger, a position of the MP joint of the finger, and a position of the fingertip of the finger as the time-series information regarding the posture of the finger in contact with the object. ### Claim 17 (depends on 1) The information processing apparatus according to claim 1, wherein the object is a keyboard, and the operation of the finger with respect to the object is a key hitting operation of the finger with respect to the keyboard or a moving operation of moving a position of the finger with respect to the keyboard. ### Claim 18 (depends on 1) The information processing apparatus according to claim 1, further comprising a provision unit configured to provide the time-series information regarding the posture of the finger estimated by the estimation unit to a user. ### Claim 19 An information processing method comprising: allowing a computer to estimate time-series information regarding a posture of a finger on the basis of image information including an operation of the finger with respect to an object including a contact operation of the finger with respect to the object and the object. ### Claim 20 A program for causing a computer to function as an estimation unit that estimates time-series information regarding a posture of a finger on the basis of image information including an operation of the finger with respect to an object including a contact operation of the finger with respect to the object and the object.