# US 11,113,887 B2 · Generating three-dimensional content from two-dimensional images > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,113,887 B2 - Document ID: US11113887B2 - Status: Granted - Kind: B2 - Application number: 16/428,982 - Filed: 2019-06-01 - Published: 2019-09-19 - Granted: 2021-09-07 - Earliest claimed: 2018-01-08 - Assignees: Verizon Patent and Licensing Inc. - Applicants: Verizon Patent and Licensing Inc. - Inventors: Daniel Kopeinigg; Arthur van Hoff; Philip Lee; Solmaz Hajmohammadi; Sourabh Khire; Simion Venshtain; Andrew Walkingshaw - Web page: https://patentgrep.com/patent/US11113887B2 ## CPC - G06Q 10/40: Business processes related to social networking or social networking services - G06F 18/24: Classification techniques - G06F 30/20: Design optimisation, verification or simulation (optimisation, verification or simulation of circuit designs G06F30/30) - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 20/00: Machine learning - G06T 7/55: from multiple images - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 7/73: using feature-based methods - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06V 10/82: using neural networks - G06V 20/42: {of sport video content} - G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition} - G06T 2207/10016: Video; Image sequence - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30196: Human being; Person - G06T 2207/30221: Sports video; Sports image - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method includes receiving two-dimensional video streams from a plurality of cameras, the two-dimensional video streams including multiple angles of a sporting event. The method further includes determining boundaries of the sporting event from the two-dimensional video streams. The method further includes identifying a location of a sporting object during the sporting event. The method further includes identifying one or more players in the sporting event. The method further includes identifying poses of each of the one or more players during the sporting event. The method further includes generating a three-dimensional model of the sporting event based on the boundaries of the sporting event, the location of the sporting object during the sporting event, and the poses of each of the one or more players during the sporting event. The method further includes generating a simulation of the three-dimensional model. ## Claims ### Claim 1 A method comprising: receiving two-dimensional video streams from a plurality of cameras, the two-dimensional video streams including multiple angles of a live event; determining boundaries of the live event from the two-dimensional video streams; identifying a location of a live object during the live event; identifying one or more players in the live event; identifying poses of each of the one or more players during the live event; identifying one or more social network profiles that correspond to the one or more players in the live event; generating a three-dimensional model of the live event based on the boundaries of the live event, the location of the live object during the live event, and the poses of each of the one or more players during the live event; generating a simulation of the three-dimensional model; providing a presentation of the simulation of the three-dimensional model as an augmentation of real-world content in a cross-reality view presented to a user, the real-world content including a presentation of the live event that is different from the presentation of the simulation of the three-dimensional model in the cross-reality view; and providing, together with the presentation of the simulation of the three-dimensional model and as an additional augmentation of the real-world content in the cross-reality view presented to the user, a presentation of one or more links to the one or more social network profiles of the one or more players and a presentation of an additional three-dimensional model, the additional three-dimensional model being of buttons for controlling the presentation of the simulation. ### Claim 2 (depends on 1) The method of claim 1, further comprising: providing the simulation to a cross reality system that includes augmented reality glasses, wherein the simulation is overlaid onto real-world elements included in the real-world content. ### Claim 3 (depends on 1) The method of claim 1, further comprising: generating a machine learning model, wherein the machine learning model is used to determine the boundaries, identify the location of the live event, identify the poses of each of the one or more players, generate the three-dimensional model, and generate the simulation. ### Claim 4 (depends on 3) The method of claim 3, wherein the machine learning model is generated by at least one of a deep neural network, a convolutional neural network, and a recurrent neural network. ### Claim 5 (depends on 1) The method of claim 1, wherein identifying the poses of the one or more players during the live event includes generating a skeletal model of each of the one or more players and the skeletal model predicts a location of corresponding body parts that are obscured in image frames of the two-dimensional video streams. ### Claim 6 (depends on 5) The method of claim 5, further comprising: classifying people that are in the two-dimensional video streams including the one or more players; wherein predicting the location of corresponding body parts that are obscured in the image frames includes applying a classification of the people to determine whether the corresponding body parts are obscured by the one or more players or an other person. ### Claim 7 (depends on 6) The method of claim 6, further comprising: identifying one or more additional social network profiles that correspond to additional people that are in the two-dimensional video streams, wherein the additional people include spectators to the live event; and providing one or more additional links to the one or more additional social network profiles within the simulation. ### Claim 8 (depends on 1) The method of claim 1, further comprising generating the additional three-dimensional model of buttons based on a type of the live event. ### Claim 9 (depends on 8) The method of claim 8, wherein the additional three-dimensional model includes at least one button that is specific to the type of the live event and that is for selecting an event in the live event for playback in the presentation of the simulation. ### Claim 10 (depends on 1) The method of claim 1, wherein the simulation of the three-dimensional model and the additional three-dimensional model are presented in a user interface that includes an option to create a bookmark of part of the live event for later viewing. ### Claim 11 A system comprising: one or more processors coupled to a memory; a processing module stored in the memory and executable by the one or more processors, the processing module operable to receive two-dimensional video streams from a plurality of cameras, the two-dimensional video streams including multiple angles of a live event; a machine learning module stored in the memory and executable by the one or more processors, the machine learning module operable to generate a machine learning model to determine boundaries of the live event from the two-dimensional video streams and identify a location of a live object during the live event; a tracking module stored in the memory and executable by the one or more processors, the tracking module operable to, based on the machine learning model, identify one or more players in the live event, identify one or more social network profiles that correspond to the one or more players in the live event, and identify poses of each of the one or more players during the live event; a simulation module stored in the memory and executable by the one or more processors, the simulation module operable to, based on the machine learning model, generate a three-dimensional model of the live event based on the boundaries of the live event, the location of the live object during the live event, and the poses of each of the one or more players during the live event and generate a simulation of the three-dimensional model; and a user interface module stored in memory and executable by the one or more processors, the user interface module operable to provide a presentation of the simulation of the three-dimensional model as an augmentation of real-world content in a cross-reality view presented to a user, the real-world content including a presentation of the live event that is different from the presentation of the simulation of the three-dimensional model in the cross-reality view; wherein the user interface module is operable to provide, together with the presentation of the simulation of the three-dimensional model and as an additional augmentation of the real-world content in the cross-reality view presented to the user, a presentation of one or more links to the one or more social network profiles of the one or more players and a presentation of an additional three-dimensional model, the additional three-dimensional model being of buttons for controlling the presentation of the simulation. ### Claim 12 (depends on 11) The system of claim 11, wherein the simulation of the three-dimensional model and the additional three-dimensional model are overlaid onto real-world elements included in the real-world content. ### Claim 13 (depends on 12) The system of claim 12, wherein the machine learning model is generated by at least one of a deep neural network, a convolutional neural network, and a recurrent neural network. ### Claim 14 (depends on 11) The system of claim 11, wherein identifying the poses of the one or more players during the live event includes generating a skeletal model of each of the one or more players and the skeletal model predicts a location of corresponding body parts that are obscured in image frames of the two-dimensional video streams. ### Claim 15 A non-transitory computer storage medium encoded with instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: receiving two-dimensional video streams from a plurality of cameras, the two-dimensional video streams including multiple angles of a live event; determining boundaries of the live event from the two-dimensional video streams; identifying a location of a live object during the live event; identifying one or more players in the live event; identifying poses of each of the one or more players during the live event; identifying the one or more social network profiles that correspond to the one or more players in the live event; generating a three-dimensional model of the live event based on the boundaries of the live event, the location of the live object during the live event, and the poses of each of the one or more players during the live event; generating a simulation of the three-dimensional model; providing a presentation of the simulation of the three-dimensional model as an augmentation of real-world content in a cross-reality view presented to a user, the real-world content including a presentation of the live event that is different from the presentation of the simulation of the three-dimensional model in the cross-reality view; and providing, together with the presentation of the simulation of the three-dimensional model and as an additional augmentation of the real-world content in the cross-reality view presented to the user, a presentation of one or more links to the one or more social network profiles of the one or more players and a presentation of an additional three-dimensional model, the additional three-dimensional model being of buttons for controlling the presentation of the simulation. ### Claim 16 (depends on 15) The non-transitory computer storage medium of claim 15, wherein the operations further comprise: providing the simulation to a cross reality system that includes augmented reality glasses by way of which the simulation is overlaid onto real-world elements included in the real-world content. ### Claim 17 (depends on 15) The non-transitory computer storage medium of claim 15, wherein the operations further comprise: generating a machine learning model, wherein the machine learning model is used to determine the boundaries, identify the location of the live event, identify the poses of each of the one or more players, generate the three-dimensional model, and generate the simulation. ### Claim 18 (depends on 17) The non-transitory computer storage medium of claim 17, wherein the machine learning model is generated by at least one of a deep neural network, a convolutional neural network, and a recurrent neural network. ### Claim 19 (depends on 15) The non-transitory computer storage medium of claim 15, wherein identifying the poses of the one or more players during the live event includes generating a skeletal model of each of the one or more players and the skeletal model predicts a location of corresponding body parts that are obscured in image frames of the two-dimensional video streams.