Forecasting multiple poses based on a graphical image
AssigneeAdobe Systems Incorporated
InventorsJimei Yang, Yu-Wei Chao, Scott Cohen, Brian Price
Life of this patent
Estimating the term…
Abstract
A forecasting neural network receives data and extracts features from the data. A recurrent neural network included in the forecasting neural network provides forecasted features based on the extracted features. In an embodiment, the forecasting neural network receives an image, and features of the image are extracted. The recurrent neural network forecasts features based on the extracted features, and pose is forecasted based on the forecasted features. Additionally or alternatively, additional poses are forecasted based on additional forecasted features.
Claims20
A method of forecasting multiple positions of a subject depicted by an image, the method comprising:
- receiving, by a forecasting neural network, an image depicting a subject, wherein the forecasting neural network comprises an encoder neural network, a recurrent neural network, and a decoder neural network;
- extracting, by the encoder neural network, a feature of the received image;
- providing the extracted feature to the recurrent neural network;
- determining, by the recurrent neural network, a first modification to the extracted feature;
- determining, by the recurrent neural network and based on the first modification, a second modification to the extracted feature;
- generating, by the recurrent neural network, a first forecasted feature based on the determined first modification and a second forecasted feature based on the determined second modification;
- providing the first forecasted feature and the second forecasted feature to the decoder neural network;
- and
- generating, by the decoder neural network, a first set of keypoints and a second set of keypoints, wherein each keypoint in the first set of keypoints indicates a forecasted position of a respective portion of the image subject and each keypoint in the second set of keypoints indicates a second forecasted position of the respective portion of the image subject.
A non-transitory computer-readable medium embodying program code for producing multiple poses from an input image, the program code comprising instructions which, when executed by a processor, cause the processor to perform operations comprising:
- receiving, by a forecasting neural network, an image depicting a subject, wherein the forecasting neural network comprises an encoder neural network, a recurrent neural network, and a decoder neural network;
- extracting, by the encoder neural network, a feature of the received image;
- providing the extracted feature to the recurrent neural network;
- determining, by the recurrent neural network, a first modification to the extracted feature;
- determining, by the recurrent neural network and based on the first modification, a second modification to the extracted feature;
- generating, by the recurrent neural network, a first forecasted feature based on the determined first modification and a second forecasted feature based on the determined second modification;
- providing the first forecasted feature and the second forecasted feature to the decoder neural network;
- and
- generating, by the decoder neural network, a first set of keypoints and a second set of keypoints, wherein each keypoint in the first set of keypoints indicates a forecasted position of a respective portion of the image subject and each keypoint in the second set of keypoints indicates a second forecasted position of the respective portion of the image subject.
A system for producing multiple poses from an input image, the system comprising:
- a means for receiving, by a forecasting neural network, an image depicting a subject, wherein the forecasting neural network comprises an encoder neural network, a recurrent neural network, and a decoder neural network;
- a means for extracting, by the encoder neural network, a feature of the received image;
- a means for providing the extracted feature to the recurrent neural network;
- a means for determining, by the recurrent neural network, a first modification to the extracted feature;
- a means for determining, by the recurrent neural network and based on the first modification, a second modification to the extracted feature;
- a means for generating, by the recurrent neural network, a first forecasted feature based on the determined first modification and a second forecasted feature based on the determined second modification;
- a means for providing the first forecasted feature and the second forecasted feature to the decoder neural network;
- and
- a means for generating, by the decoder neural network, a first set of keypoints and a second set of keypoints, wherein each keypoint in the first set of keypoints indicates a forecasted position of a respective portion of the image subject and each keypoint in the second set of keypoints indicates a second forecasted position of the respective portion of the image subject.
Dependencies are inferred from the claim text.