# US 2021/0243012 A1 · Multi-domain convolutional neural network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0243012 A1
- Document ID: US20210243012A1
- Status: Published application
- Kind: A1
- Application number: 16/948,304
- Filed: 2020-09-11
- Published: 2021-08-05
- Earliest claimed: 2017-12-28
- Assignees: Intel Corporation
- Applicants: Intel Corporation
- Inventors: Yen-Kuang Chen; Shao-Wen Yang; Ibrahima J. Ndiour; Yiting Liao; Vallabhajosyula S. Somayazulu; Omesh Tickoo; Srenivas Varadarajan
- Web page: https://patentgrep.com/patent/US20210243012A1

## CPC

- G06F 21/44: Program or device authentication
- G06F 9/4881: {Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues}
- G06F 9/5044: {considering hardware capabilities}
- G06F 9/5066: {Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs (mappping at compile time, see G06F8/451)}
- G06F 9/5072: {Grid computing}
- G06F 16/535: Filtering based on additional data, e.g. user or group profiles
- G06F 16/538: Presentation of query results
- G06F 16/54: Browsing; Visualisation therefor
- G06F 16/951: Indexing; Web crawling techniques
- G06F 18/21: Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F 18/211: Selection of the most significant subset of features
- G06F 18/213: Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
- G06F 18/2163: {Partitioning the feature space}
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 18/24: Classification techniques
- G06F 18/24143: {Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]}
- G06F 21/45: Structures or tools for the administration of authentication
- G06F 21/53: by executing in a restricted environment, e.g. sandbox or secure virtual machine
- G06F 21/6254: {by anonymising data, e.g. decorrelating personal data from the owner's identification}
- G06F 21/64: Protecting data integrity, e.g. using checksums, certificates or signatures
- G06K 15/1886: {Storage of the print image data or of parts thereof}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/063: using electronic means
- G06N 3/08: Learning methods
- G06N 5/022: Knowledge engineering; Knowledge acquisition
- G06T 7/11: Region-based segmentation
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 10/20: Image preprocessing
- G06V 10/40: Extraction of image or video features
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/75: Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
- G06V 10/771: Feature selection, e.g. selecting representative features from a multi-dimensional feature space
- G06V 10/82: using neural networks
- G06V 10/95: {structured as a network, e.g. client-server architectures}
- G06V 10/96: Management of image or video recognition tasks
- G06V 20/00: Scenes; Scene-specific elements (control of digital cameras H04N23/60)
- G06V 30/19173: {Classification techniques}
- G06V 30/274: {Syntactic or semantic context, e.g. balancing}
- G06V 40/161: {Detection; Localisation; Normalisation}
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- H04L 9/0643: {Hash functions, e.g. MD5, SHA, HMAC or f9 MAC}
- H04L 9/3239: {involving non-keyed hash functions, e.g. modification detection codes [MDCs], MD5, SHA or RIPEMD}
- H04L 67/12: specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
- H04L 67/51: Discovery or management thereof, e.g. service location protocol [SLP] or web services
- H04N 19/46: Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence)
- H04N 19/80: Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation (H04N19/635, H04N19/86 take precedence)
- H04W 4/70: Services for machine-to-machine communication [M2M] or machine type communication [MTC]
- G06F 18/24323: {Tree-organised classifiers}
- G06F 2209/503: Resource availability
- G06F 2209/506: Constraint
- G06F 2221/2117: User registration
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 7/223: using block-matching
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G06T 2207/20024: Filtering details
- G06T 2207/20052: Discrete cosine transform [DCT]
- G06T 2207/20056: Discrete and fast Fourier transform, [DFT, FFT]
- G06T 2207/20064: Wavelet transform [DWT]
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20221: Image fusion; Image merging
- G06T 2207/30242: Counting objects in image
- G06V 30/194: References adjustable by an adaptive method, e.g. learning
- G06V 2201/10: Recognition assisted with metadata
- H04L 9/50: {using hash chains, e.g. blockchains or hash trees}
- H04L 67/10: in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46)
- H04N 19/117: Filters, e.g. for pre-processing or post-processing (sub-band filter banks H04N19/635)
- H04N 19/12: Selection from among a plurality of transforms or standards, e.g. selection between discrete cosine transform [DCT] and sub-band transform or selection between H.263 and H.264
- H04N 19/124: Quantisation
- H04N 19/167: Position within a video image, e.g. region of interest [ROI]
- H04N 19/172: the region being a picture, frame or field
- H04N 19/176: the region being a block, e.g. a macroblock
- H04N 19/42: characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation (H04N19/635 takes precedence)
- H04N 19/44: Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
- H04N 19/48: using compressed domain processing techniques other than decoding, e.g. modification of transform coefficients, variable length coding [VLC] data or run-length data (motion estimation in a transform domain H04N19/547; processing of decoded motion vectors H04N19/513)
- H04N 19/513: Processing of motion vectors
- H04N 19/625: using discrete cosine transform [DCT]
- H04N 19/63: using sub-band based transform, e.g. wavelets
- H04W 12/02: Protecting privacy or anonymity, e.g. protecting personally identifiable information [PII]
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06K: GRAPHICAL DATA READING (image or video recognition or understanding G06V); PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)

## Abstract

In one embodiment, an apparatus comprises a memory and a processor. The memory is to store visual data associated with a visual representation captured by one or more sensors. The processor is to: obtain the visual data associated with the visual representation captured by the one or more sensors, wherein the visual data comprises uncompressed visual data or compressed visual data; process the visual data using a convolutional neural network (CNN), wherein the CNN comprises a plurality of layers, wherein the plurality of layers comprises a plurality of filters, and wherein the plurality of filters comprises one or more pixel-domain filters to perform processing associated with uncompressed data and one or more compressed-domain filters to perform processing associated with compressed data; and classify the visual data based on an output of the CNN.

## Claims

### Claim at position 1

1-25. (canceled)

### Claim 26

A computing device to perform content detection on compressed images, comprising:
interface circuitry; and
processing circuitry to:
receive, via the interface circuitry, a compressed image, wherein the compressed image represents an image in a compressed format;
extract a set of transform coefficients from the compressed image, wherein the set of transform coefficients represents a transform performed on a plurality of blocks of the image, wherein the set of transform coefficients comprises a corresponding subset of transform coefficients for each block of the plurality of blocks;
generate a three-dimensional (3D) data structure to represent the set of transform coefficients, wherein a first dimension and a second dimension of the 3D data structure represent the plurality of blocks, and wherein a third dimension of the 3D data structure represents the corresponding subset of transform coefficients for each block; and
detect image content captured in the compressed image, wherein the image content is detected based on analyzing the 3D data structure using a convolutional neural network (CNN).

### Claim 27 (depends on 26)

The computing device of claim 26, wherein the first dimension and the second dimension of the 3D data structure further represent spatial indices of the plurality of blocks within the image.

### Claim 28 (depends on 26)

The computing device of claim 26, wherein the third dimension of the 3D data structure further represents a plurality of channels, wherein each channel represents a corresponding group of correlated transform coefficients among the plurality of blocks, wherein the corresponding group of correlated transform coefficients is from the set of transform coefficients.

### Claim 29 (depends on 28)

The computing device of claim 28, wherein:
the transform is a discrete cosine transform (DCT);
the set of transform coefficients is a set of DCT coefficients; and
the plurality of channels comprises:
a first channel representing a group of DC coefficients among the plurality of blocks, wherein the group of DC coefficients is from the set of DCT coefficients; and
a second channel representing a group of AC coefficients among the plurality of blocks, wherein the group of AC coefficients is from the set of DCT coefficients.

### Claim 30 (depends on 26)

The computing device of claim 26, wherein:
the image is captured by a camera; and
the processing circuitry to receive, via the interface circuitry, the compressed image is further to receive the compressed image over a network from a device associated with the camera.

### Claim 31 (depends on 26)

The computing device of claim 26:
further comprising:
a camera to capture the image; and
image processing circuitry to compress the image into the compressed image; and
wherein the processing circuitry to receive, via the interface circuitry, the compressed image is further to receive the compressed image from the image processing circuitry.

### Claim 32 (depends on 26)

The computing device of claim 26, wherein:
the computing device is an edge processing device for performing visual analytics; or
the computing device is a smart camera.

### Claim 33 (depends on 26)

The computing device of claim 26, wherein the CNN is to detect the image content based on analyzing the 3D data structure using one or more 3D convolution filters, wherein the one or more 3D convolution filters are trained to perform content detection based on transform coefficients from compressed images.

### Claim 34 (depends on 26)

The computing device of claim 26, wherein:
the transform is a discrete cosine transform (DCT); and
the set of transform coefficients is a set of DCT coefficients.

### Claim 35 (depends on 26)

The computing device of claim 26, wherein:
the transform is a wavelet transform; and
the set of transform coefficients is a set of wavelet transform coefficients.

### Claim 36 (depends on 26)

The computing device of claim 26, wherein:
the transform is an integer transform; and
the set of transform coefficients is a set of integer transform coefficients.

### Claim 37

At least one non-transitory computer-readable storage medium having instructions stored thereon, wherein the instructions, when executed on processing circuitry, cause the processing circuitry to:
receive, via interface circuitry, a compressed image, wherein the compressed image represents an image in a compressed format;
extract a set of transform coefficients from the compressed image, wherein the set of transform coefficients represents a transform performed on a plurality of blocks of the image, wherein the set of transform coefficients comprises a corresponding subset of transform coefficients for each block of the plurality of blocks;
generate a three-dimensional (3D) data structure to represent the set of transform coefficients, wherein a first dimension and a second dimension of the 3D data structure represent the plurality of blocks, and wherein a third dimension of the 3D data structure represents the corresponding subset of transform coefficients for each block; and
detect image content captured in the compressed image, wherein the image content is detected based on analyzing the 3D data structure using a convolutional neural network (CNN).

### Claim 38 (depends on 37)

The storage medium of claim 37, wherein:
the first dimension and the second dimension of the 3D data structure further represent spatial indices of the plurality of blocks within the image; and
the third dimension of the 3D data structure further represents a plurality of channels, wherein each channel represents a corresponding group of correlated transform coefficients among the plurality of blocks, wherein the corresponding group of correlated transform coefficients is from the set of transform coefficients.

### Claim 39 (depends on 38)

The storage medium of claim 38, wherein:
the transform is a discrete cosine transform (DCT);
the set of transform coefficients is a set of DCT coefficients; and
the plurality of channels comprises:
a first channel representing a group of DC coefficients among the plurality of blocks, wherein the group of DC coefficients is from the set of DCT coefficients; and
a second channel representing a group of AC coefficients among the plurality of blocks, wherein the group of AC coefficients is from the set of DCT coefficients.

### Claim 40 (depends on 37)

The storage medium of claim 37, wherein:
the image is captured by a camera; and
the instructions that cause the processing circuitry to receive, via the interface circuitry, the compressed image further cause the processing circuitry to receive the compressed image over a network from a device associated with the camera.

### Claim 41 (depends on 37)

The storage medium of claim 37, wherein the CNN is to detect the image content based on analyzing the 3D data structure using one or more 3D convolution filters, wherein the one or more 3D convolution filters are trained to perform content detection based on transform coefficients from compressed images.

### Claim 42 (depends on 37)

The storage medium of claim 37, wherein:
the transform is a discrete cosine transform (DCT); and
the set of transform coefficients is a set of DCT coefficients.

### Claim 43 (depends on 37)

The storage medium of claim 37, wherein:
the transform is a wavelet transform; and
the set of transform coefficients is a set of wavelet transform coefficients.

### Claim 44 (depends on 37)

The storage medium of claim 37, wherein:
the transform is an integer transform; and
the set of transform coefficients is a set of integer transform coefficients.

### Claim 45

A method of performing content detection on compressed images, comprising:
receiving, via interface circuitry, a compressed image, wherein the compressed image represents an image in a compressed format;
extracting a set of transform coefficients from the compressed image, wherein the set of transform coefficients represents a transform performed on a plurality of blocks of the image, wherein the set of transform coefficients comprises a corresponding subset of transform coefficients for each block of the plurality of blocks;
generating a three-dimensional (3D) data structure to represent the set of transform coefficients, wherein a first dimension and a second dimension of the 3D data structure represent the plurality of blocks, and wherein a third dimension of the 3D data structure represents the corresponding subset of transform coefficients for each block; and
detecting image content captured in the compressed image, wherein the image content is detected based on analyzing the 3D data structure using a convolutional neural network (CNN).

### Claim 46 (depends on 45)

The method of claim 45, wherein:
the first dimension and the second dimension of the 3D data structure further represent spatial indices of the plurality of blocks within the image; and
the third dimension of the 3D data structure further represents a plurality of channels, wherein each channel represents a corresponding group of correlated transform coefficients among the plurality of blocks, wherein the corresponding group of correlated transform coefficients is from the set of transform coefficients.

### Claim 47 (depends on 46)

The method of claim 46, wherein:
the transform is a discrete cosine transform (DCT);
the set of transform coefficients is a set of DCT coefficients; and
the plurality of channels comprises:
a first channel representing a group of DC coefficients among the plurality of blocks, wherein the group of DC coefficients is from the set of DCT coefficients; and
a second channel representing a group of AC coefficients among the plurality of blocks, wherein the group of AC coefficients is from the set of DCT coefficients.

### Claim 48 (depends on 45)

The method of claim 45, wherein the CNN is to detect the image content based on analyzing the 3D data structure using one or more 3D convolution filters, wherein the one or more 3D convolution filters are trained to perform content detection based on transform coefficients from compressed images.

### Claim 49

A system for performing content detection on compressed images, comprising:
a camera to capture an image;
image processing circuitry to compress the image into a compressed image; and
processing circuitry to:
receive the compressed image from the image processing circuitry;
extract a set of transform coefficients from the compressed image, wherein the set of transform coefficients represents a transform performed on a plurality of blocks of the image, wherein the set of transform coefficients comprises a corresponding subset of transform coefficients for each block of the plurality of blocks;
generate a three-dimensional (3D) data structure to represent the set of transform coefficients, wherein a first dimension and a second dimension of the 3D data structure represent the plurality of blocks, and wherein a third dimension of the 3D data structure represents the corresponding subset of transform coefficients for each block; and
detect image content captured in the compressed image, wherein the image content is detected based on analyzing the 3D data structure using a convolutional neural network (CNN).

### Claim 50 (depends on 49)

The system of claim 49, wherein the processing circuitry to detect the image content captured in the compressed image comprises:
artificial intelligence (AI) accelerator circuitry, wherein the AI accelerator circuitry is to detect the image content captured in the compressed image based on analyzing the 3D data structure using the CNN.