# US 11,113,845 B2 · Point cloud compression using non-cubic projections and masks > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,113,845 B2 - Document ID: US11113845B2 - Status: Granted - Kind: B2 - Application number: 16/133,677 - Filed: 2018-09-17 - Published: 2019-03-21 - Granted: 2021-09-07 - Earliest claimed: 2017-09-18 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Alexandros Tourapis; Khaled Mammou; Yeping Su; Fabrice A. Robinet; David W. Singer - Web page: https://patentgrep.com/patent/US11113845B2 ## CPC - G06T 9/00: Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00) - G06T 3/08: Projecting images onto non-planar surfaces, e.g. geodetic screens - G06T 9/001: {Model-based coding, e.g. wire frame} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A system comprises an encoder configured to compress attribute information and/or spatial for a point cloud and/or a decoder configured to decompress compressed attribute and/or spatial information for the point cloud. To compress the attribute and/or spatial information, the encoder is configured to convert a point cloud into an image based representation. Also, the decoder is configured to generate a decompressed point cloud based on an image based representation of a point cloud. ## Claims ### Claim 1 A system, comprising: one or more sensors configured to capture a plurality of points that make up a point cloud, wherein respective ones of the points comprise spatial information for the point and attribute information for the point; and an encoder configured to compress the point cloud, wherein to compress the point cloud, the encoder is configured to: generate, for the point cloud, one or more projected images; pack the generated one or more projected images into one or more image frames, wherein respective sets of pixels of the one or more image frames covered by respective ones of the projected images comprise pixel values that correspond to values for the respective ones of the projected images; pad pixels in the one or more image frames that are not covered by the one or more projected images with a pad; encode the one or more image frames; and encode information indicating pixels of the one or more image frames that correspond to active regions or non-active regions of the one or more image frames, wherein the pixels that are covered by the pad are indicated as non-active regions. ### Claim 2 (depends on 1) The system of claim 1, wherein to generate the one or more projected images, the encoder is configured to: project the point cloud onto a cylindrical surface or a spherical surface. ### Claim 3 (depends on 1) The system of claim 1, wherein to generate the one or more projected images, the encoder is configured to: project portions of the point cloud onto three or more parallel planes. ### Claim 4 (depends on 1) The system of claim 1, wherein the encoder is further configured to: encode, for points of the point cloud that are not visible from a particular point-of-view, information indicating the points are non-active points in the particular point-of-view. ### Claim 5 (depends on 4) The system of claim 4, wherein the encoder is further configured to: budget more bits to encoding active regions of the one or more image frames than are budgeted to encoding non-active regions or to encoding non-active points. ### Claim 6 (depends on 4) The system of claim 4, wherein the encoder is further configured to: encode the one or more image frames in accordance with a high efficiency video coding (HEVC) standard. ### Claim 7 (depends on 6) The system of claim 6, wherein the encoder is configured to encode the one or more image frames as a plurality of blocks, wherein the encoder is configured to, for each block of an image frame: determine whether the block includes only active regions, includes only non-active regions, or includes a mix of active and non-active regions; and for respective blocks including a mix of active and non-active regions: divide the block into sub-blocks; and for sub-blocks comprising only non-active regions exclude the sub-block from one or more distortion calculations performed as part of motion estimation or mode decision. ### Claim 8 (depends on 7) The system of claim 7, wherein: a block or sub-block adjacent to a given block or sub-block comprising only one or more non-active regions is assigned a distortion weighting factor different from a distortion weighting factor assigned to a block or sub-block comprising one or more active regions that are not adjacent to one or more non-active regions; or a block or sub-block comprising projected images for two or more different projections is assigned a distortion weighting factor different from a given block or sub-block comprising less than two different projected images; and wherein the encoder is configured to assign different distortion tolerances to the block or sub sub-block based, at least in part, the respective distortion weighting factor assigned to the block or sub-block. ### Claim 9 (depends on 6) The system of claim 6, wherein the encoder is configured to encode the one or more image frames as a plurality of blocks, wherein the encoder is configured to, for each block of an image frame: determine whether the block includes only active regions, includes only non-active regions, or includes a mix of active and non-active regions; and for respective blocks including a mix of active and non-active regions: assign a higher quality parameter for encoding the respective blocks than is assigned for other ones of the blocks that do not include a mix of active and non-active regions. ### Claim 10 (depends on 9) The system of claim 9, wherein the encoder is configured to: adjust a quantization factor or a lagrangian multiplier used to encode a block or sub-block, based, at least in part, on a quality parameter assigned to the block or sub-block. ### Claim 11 (depends on 6) The system of claim 6, wherein the encoder is configured to encode the one or more image frames as a plurality of blocks, wherein the encoder is configured to, for each block of an image frame: determine a depth of the point cloud in the one or more projected images; and assign a higher quality parameter for encoding blocks that correspond to deeper projected images, or portions thereof, than is assigned to blocks that correspond to less deep projected images, or portions thereof. ### Claim 12 (depends on 11) The system of claim 11, wherein the encoder is configured to: adjust a quantization factor or a lagrangian multiplier used to encode a block or sub-block, based, at least in part, on a quality parameter assigned to the block or sub-block. ### Claim 13 (depends on 6) The system of claim 6, wherein the encoder is configured to: adjust a quantized coefficient used in trellis quantization to encode the one or more image frames based, at least in part, on whether respective blocks of the one or more image frames comprise active or non-active regions or active or non-active points. ### Claim 14 (depends on 6) The system of claim 6, wherein the encoder is configured to: modify a sample adaptive offset decision used to encode the one or more image frames based, at least in part, on whether respective blocks of the one or more image frames comprise active or non-active regions or active or non-active points. ### Claim 15 A device comprising a computer system storing program instructions that, when executed, implement a decoder configured to: receive one or more encoded image frames comprising projected images for a compressed point cloud and padding in pixels of the or more images that are not occupied by the projected images, wherein respective sets of pixels of the one or more encoded image frames comprise pixel values that correspond to values for the respective ones of the projected images; decode the one or more encoded image frames, wherein less decoding resources are allocated to decoding the padded pixels of the one or more images than are allocated to decoding the projected image pixels of the one or more image frames; and generate a decompressed version of the compressed point cloud based, at least in part, on the decoded projected images. ### Claim 16 (depends on 15) The decoder of claim 15, wherein the decoder is further configured to: receive information indicating points of the point cloud that are not visible from a particular point-of-view, wherein the padded portions of the one or more images or the non-visible points are processed by the decoder as non-active regions of the one or more images. ### Claim 17 (depends on 16) The decoder of claim 16, wherein when performing de-blocking, sample adaptive offset processing, adaptive loop filtering, or de-ringing, the decoder is configured to exclude non-active regions. ### Claim 18 (depends on 16) The decoder of claim 16, wherein when performing post-filtering, the decoder is configured to exclude non-active regions. ### Claim 19 (depends on 16) The decoder of claim 16, wherein when performing overlapped block motion compensation, the decoder is configured to exclude non-active regions. ### Claim 20 (depends on 16) The decoder of claim 16, wherein the decoder is configured to adjust a degree to which a final value determined for a pixel of the one or more images is clipped based, at least in part, on whether the final value correspond to an active or non-active region of the one or more images. ### Claim 21 A method, comprising: receiving one or more encoded image frames comprising projected images for a compressed point cloud and padding in pixels of the or more image frames that are not occupied by the projected images, wherein respective sets of pixels of the one or more encoded image frames comprise pixel values that correspond to values for the respective ones of the projected images; decoding the one or more encoded image frames, wherein less decoding resources are allocated to decoding the padded pixels of the one or more images than are allocated to decoding the projected image pixels of the one or more image frames; and generating a decompressed version of the compressed point cloud based, at least in part, on the decoded projected images.