# US 12,008,790 B2 · Encoding three-dimensional data for processing by capsule neural networks > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,790 B2 - Document ID: US12008790B2 - Status: Granted - Kind: B2 - Application number: 16/836,028 - Filed: 2020-03-31 - Published: 2021-03-25 - Granted: 2024-06-11 - Earliest claimed: 2019-09-24 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Nitish Srivastava; Ruslan Salakhutdinov; Hanlin Goh - Web page: https://patentgrep.com/patent/US12008790B2 ## CPC - G06T 9/002: {using neural networks} - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/047: Probabilistic or stochastic networks - G06N 3/0475: Generative networks - G06N 3/08: Learning methods - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G06T 9/001: {Model-based coding, e.g. wire frame} - H04N 13/111: Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation - H04N 13/161: Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A method includes defining a geometric capsule that is interpretable by a capsule neural network, wherein the geometric capsule includes a feature representation and a pose. The method also includes determining multiple viewpoints relative to the geometric capsule and determining a first appearance representation of the geometric capsule for each of the multiple viewpoints. The method also includes determining a transform for each of the multiple viewpoints that moves each of the multiple viewpoints to a respective transformed viewpoint and determining second appearance representations that each correspond to one of the transformed viewpoints. The method also includes combining the second appearance representations to define an agreed appearance representation. The method also includes updating the feature representation for the geometric capsule based on the agreed appearance representation. ## Claims ### Claim 1 A method, comprising: defining a geometric capsule, wherein the geometric capsule is an encoded form of three-dimensional data that is interpretable by a capsule neural network, and the geometric capsule includes a three-dimensional feature representation and a pose; determining multiple viewpoints relative to the geometric capsule; determining a first appearance representation of the geometric capsule for each of the multiple viewpoints based on the three-dimensional feature representation for the geometric capsule; determining a transform for each of the multiple viewpoints that moves each of the multiple viewpoints to a respective transformed viewpoint; determining second appearance representations that each correspond to one of the transformed viewpoints based on the three-dimensional feature representation for the geometric capsule; combining the second appearance representations to define an agreed appearance representation; and updating the three-dimensional feature representation for the geometric capsule based on the agreed appearance representation. ### Claim 2 (depends on 1) The method of claim 1, wherein defining the geometric capsule includes: receiving a group of elements that represent a three-dimensional visual entity, and determining an assignment of at least some elements from the group of elements to the geometric capsule. ### Claim 3 (depends on 2) The method of claim 2, wherein updating the three-dimensional feature representation for the geometric capsule based on the agreed upon appearance representation includes updating the assignment of at least some elements from the group of elements to the geometric capsule. ### Claim 4 (depends on 2) The method of claim 2, wherein the group of elements is a point cloud and each element from the group of elements is a point that is included in the point cloud. ### Claim 5 (depends on 2) The method of claim 2, wherein the group of elements is a group of lower-level geometric capsules. ### Claim 6 (depends on 1) The method of claim 1, wherein determining the transform for each of the multiple viewpoints is performed using a trained neural network. ### Claim 7 (depends on 6) The method of claim 6, wherein the trained neural network is configured to determine the transform for each of the multiple viewpoints such that the second appearance representations corresponding to the transformed viewpoints are constrained to match while the transformed viewpoints do not match. ### Claim 8 A non-transitory computer-readable storage device including program instructions executable by one or more processors that, when executed, cause the one or more processors to perform operations, the operations comprising: defining a geometric capsule, wherein the geometric capsule is an encoded form of three-dimensional data that is interpretable by a capsule neural network, and the geometric capsule includes a three-dimensional feature representation and a pose; determining multiple viewpoints relative to the geometric capsule; determining a first appearance representation of the geometric capsule for each of the multiple viewpoints based on the three-dimensional feature representation for the geometric capsule; determining a transform for each of the multiple viewpoints that moves each of the multiple viewpoints to a respective transformed viewpoint; determining second appearance representations that each correspond to one of the transformed viewpoints based on the three-dimensional feature representation for the geometric capsule; combining the second appearance representations to define an agreed appearance representation; and updating the three-dimensional feature representation for the geometric capsule based on the agreed appearance representation. ### Claim 9 (depends on 8) The non-transitory computer-readable storage device of claim 8, wherein defining the geometric capsule includes: receiving a group of elements that represent a three-dimensional visual entity, and determining an assignment of at least some elements from the group of elements to the geometric capsule. ### Claim 10 (depends on 9) The non-transitory computer-readable storage device of claim 9, wherein updating the three-dimensional feature representation for the geometric capsule based on the agreed upon appearance representation includes updating the assignment of at least some elements from the group of elements to the geometric capsule. ### Claim 11 (depends on 9) The non-transitory computer-readable storage device of claim 9, wherein the group of elements is a point cloud and each element from the group of elements is a point that is included in the point cloud. ### Claim 12 (depends on 9) The non-transitory computer-readable storage device of claim 9, wherein the group of elements is a group of lower-level geometric capsules. ### Claim 13 (depends on 8) The non-transitory computer-readable storage device of claim 8, wherein determining the transform for each of the multiple viewpoints is performed using a trained neural network. ### Claim 14 (depends on 13) The non-transitory computer-readable storage device of claim 13, wherein the trained neural network is configured to determine the transform for each of the multiple viewpoints such that the second appearance representations corresponding to the transformed viewpoints are constrained to match while the transformed viewpoints do not match. ### Claim 15 A system, comprising: a memory that includes program instructions; and a processor that is operable to execute the program instructions, wherein the program instructions, when executed by the processor, cause the processor to: define a geometric capsule, wherein the geometric capsule is an encoded form of three-dimensional data that is interpretable by a capsule neural network, and the geometric capsule includes a three-dimensional feature representation and a pose; determine multiple viewpoints relative to the geometric capsule; determine a first appearance representation of the geometric capsule for each of the multiple viewpoints based on the three-dimensional feature representation for the geometric capsule; determine a transform for each of the multiple viewpoints that moves each of the multiple viewpoints to a respective transformed viewpoint; determine second appearance representations that each correspond to one of the transformed viewpoints based on the three-dimensional feature representation for the geometric capsule; combine the second appearance representations to define an agreed appearance representation; and update the three-dimensional feature representation for the geometric capsule based on the agreed appearance representation. ### Claim 16 (depends on 15) The system of claim 15, wherein the program instructions to define the geometric capsule further cause the processor to: receive a group of elements that represent a three-dimensional visual entity, and determine an assignment of at least some elements from the group of elements to the geometric capsule. ### Claim 17 (depends on 16) The system of claim 16, wherein updating the three-dimensional feature representation for the geometric capsule based on the agreed upon appearance representation includes updating the assignment of at least some elements from the group of elements to the geometric capsule. ### Claim 18 (depends on 16) The system of claim 16, wherein the group of elements is a point cloud and each element from the group of elements is a point that is included in the point cloud. ### Claim 19 (depends on 16) The system of claim 16, wherein the group of elements is a group of lower-level geometric capsules. ### Claim 20 (depends on 15) The system of claim 15, wherein determining the transform for each of the multiple viewpoints is performed using a trained neural network. ### Claim 21 (depends on 20) The system of claim 20, wherein the trained neural network is configured to determine the transform for each of the multiple viewpoints such that the second appearance representations corresponding to the transformed viewpoints are constrained to match while the transformed viewpoints do not match. ### Claim 22 (depends on 1) The method of claim 1, wherein the geometric capsule represents a portion of a three-dimensional visual entity, the three-dimensional feature representation is encoded in the form of a pose-invariant feature vector, the pose describes a location and orientation of the portion of the three-dimensional visual entity, and determining the transform for each of the multiple viewpoints is based on the first appearance representations for each of the multiple viewpoints. ### Claim 23 (depends on 1) The method of claim 1, wherein: the first appearance representations are each described by a respective first Gaussian distribution, the second appearance representations are each described by a respective second Gaussian distribution, and combining the second appearance representations to define the agreed appearance representation includes setting the agreed appearance representation equal to a product of the second Gaussian distributions. ### Claim 24 (depends on 8) The non-transitory computer-readable storage device of claim 8, wherein the geometric capsule represents a portion of a three-dimensional visual entity, the three-dimensional feature representation is encoded in the form of a pose-invariant feature vector, the pose describes a location and orientation of the portion of the three-dimensional visual entity, and determining the transform for each of the multiple viewpoints is based on the first appearance representations for each of the multiple viewpoints. ### Claim 25 (depends on 8) The non-transitory computer-readable storage device of claim 8, wherein: the first appearance representations are each described by a respective first Gaussian distribution, the second appearance representations are each described by a respective second Gaussian distribution, and combining the second appearance representations to define the agreed appearance representation includes setting the agreed appearance representation equal to a product of the second Gaussian distributions. ### Claim 26 (depends on 15) The system of claim 15, wherein: the geometric capsule represents a portion of a three-dimensional visual entity, the three-dimensional feature representation is encoded in the form of a pose-invariant feature vector, the pose describes a location and orientation of the portion of the three-dimensional visual entity, and determining the transform for each of the multiple viewpoints is based on the first appearance representations for each of the multiple viewpoints. ### Claim 27 (depends on 15) The system of claim 15, wherein: the first appearance representations are each described by a respective first Gaussian distribution, the second appearance representations are each described by a respective second Gaussian distribution, and combining the second appearance representations to define the agreed appearance representation includes setting the agreed appearance representation equal to a product of the second Gaussian distributions. ### Claim 28 (depends on 1) The method of claim 1, wherein the three-dimensional feature representation is encoded in the form of a feature vector that describes a three-dimensional surface. ### Claim 29 (depends on 8) The non-transitory computer-readable storage device of claim 8, wherein the three-dimensional feature representation is encoded in the form of a feature vector that describes a three-dimensional surface. ### Claim 30 (depends on 15) The system of claim 15, wherein the three-dimensional feature representation is encoded in the form of a feature vector that describes a three-dimensional surface.