# US 2021/0342634 A1 · Precomputed similarity index of files in data protection systems with neural network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342634 A1
- Document ID: US20210342634A1
- Status: Published application
- Kind: A1
- Application number: 15/929,437
- Filed: 2020-05-01
- Published: 2021-11-04
- Earliest claimed: 2020-05-01
- Assignees: EMC IP Holding Company LLC
- Applicants: EMC IP Holding Company LLC
- Inventors: Qiang Chen; Jing Yu; Pengfei Wu; Naveen Rastogi
- Web page: https://patentgrep.com/patent/US20210342634A1

## CPC

- G06F 16/137: {Hash-based (content-based indexing of textual data G06F16/31)}
- G06F 16/3347: {using vector based model}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 5/022: Knowledge engineering; Knowledge acquisition
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06V 10/763: {Non-hierarchical techniques, e.g. based on statistics of modelling distributions}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/82: using neural networks
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Described is a system and method that provides a data protection risk assessment for the overall functioning of a backup and recovery system. Accordingly, the system may provide a single overall risk assessment score that provide an operator with an “at-a-glance” overview of the entire system. Moreover, the system may account for changes that occur over time based on leveraging statistical methods to automatically generate assessment scores for various components (e.g. application, server, network, load, etc.). In order to determine a risk assessment score, the system may utilize a predictive model based on historical data. Accordingly, residual values for newly observed data may be determined using the predictive model and the system may identify potentially anomalous or high risk indicators.

## Claims

### Claim 1

A computer-implemented method, comprising:
for each file from a plurality of files:
chunking the file into a plurality of chunks;
loading the plurality of chunks into a neural network and operating the neural network to generate a plurality of feature vectors, one feature vector for each of the plurality of chunks, wherein the neural network is pre-trained to generate similar feature vector for similar chunks and dissimilar vector for dissimilar chunks;
assemble the plurality of feature vectors into a single vector;
loading the single vector into the neural network to generate a file feature vector, wherein the neural network is pre-trained to generate similar file feature vector for similar assembled feature vectors and dissimilar file feature vector for dissimilar assembled feature vectors; and
storing the file feature vectors of the plurality of files into a file feature vector database.

### Claim 2 (depends on 1)

The method of claim 1, further comprising converting each of the single vectors into a fixed-length single vector prior to loading the fixed-length single vector into the neural network.

### Claim 3 (depends on 2)

The method of claim 2, wherein the converting is performed by defining a plurality of bins, pooling the single vector into the plurality of bins, and concatenating the plurality of bins to generate the fixed-length single vector.

### Claim 4 (depends on 1)

The method of claim 1, further comprising, upon receiving a query file performing the operations:
chunking the query file into a plurality of chunks;
loading the plurality of chunks into the neural network and operating the neural network to generate a plurality of feature vectors, one feature vector for each of the plurality of chunks;
concatenating the plurality of feature vectors into a query vector;
loading the query vector into the neural network and operating the neural network to search the file feature vector database for all file feature vectors within a specified distance from the query vector.

### Claim 5 (depends on 4)

The method of claim 4, wherein a distance between the query vector and a file feature vector equals a distance between the query file and a file corresponding to the file feature vector to within a predefined error.

### Claim 6 (depends on 1)

The method of claim 1, further comprising pre-training the neural network by the process comprising:
obtaining a plurality of training files;
for each training file from a plurality of training files performing:
chunking the training file into a plurality of chunks;
loading the plurality of chunks into a neural network and operating the neural network to generate a plurality of feature vectors, one feature vector for each of the plurality of chunks, wherein the neural network is pre-trained to generate similar feature vector for similar chunks and dissimilar vector for dissimilar chunks;
assemble the plurality of feature vectors into a training vector;
generating a modified vector by modifying the training vector;
selecting a negative vector determined to be dissimilar to the training vector;
loading the training vector, the modified vector and the negative vector into the neural network to generate a file feature vector for each of the training vector, the modified vector and the negative vector; and,
adapting internal weights of the neural network to minimize a distance for the file feature vectors for the training vector and the modified vector, and maximize the distance for the file feature vectors for the training vector and the negative vector.

### Claim 7

A non-transitory computer-readable medium storing instructions which, when executed by one or more processors of a computing device, cause the computing device to perform operations comprising:
for each file from a plurality of files:
chunking the file into a plurality of chunks;
loading the plurality of chunks into a neural network and operating the neural network to generate a plurality of feature vectors, one feature vector for each of the plurality of chunks, wherein the neural network is pre-trained to generate similar feature vector for similar chunks and dissimilar vector for dissimilar chunks;
assemble the plurality of feature vectors into a single vector;
loading the single vector into the neural network to generate a file feature vector, wherein the neural network is pre-trained to generate similar file feature vector for similar assembled feature vectors and dissimilar file feature vector for dissimilar assembled feature vectors; and
assembling the file feature vectors of the plurality of files into a file feature vector database.

### Claim 8 (depends on 7)

The medium of claim 7, the operations further comprising: converting each of the single vectors into a fixed-length single vector prior to loading the fixed-length single vector into the neural network.

### Claim 9 (depends on 8)

The medium of claim 8, wherein the converting is performed by defining a plurality of bins, pooling the single vector into the plurality of bins, and concatenating the plurality of bins to generate the fixed-length single vector.

### Claim 10 (depends on 7)

The medium of claim 7, further comprising, upon receiving a query file performing the operations:
chunking the query file into a plurality of chunks;
loading the plurality of chunks into the neural network and operating the neural network to generate a plurality of feature vectors, one feature vector for each of the plurality of chunks;
concatenating the plurality of feature vectors into a query vector;
loading the query vector into the neural network and operating the neural network to search the file feature vector database for all file feature vectors within a specified distance from the query vector.

### Claim 11 (depends on 10)

The medium of claim 10, wherein a distance between the query vector and a file feature vector equals a distance between the query file and a file corresponding to the file feature vector to within a predefined error.

### Claim 12 (depends on 7)

The medium of claim 7, further comprising pre-training the neural network by the process comprising:
obtaining a plurality of training files;
for each training file from a plurality of training files performing:
chunking the training file into a plurality of chunks;
loading the plurality of chunks into a neural network and operating the neural network to generate a plurality of feature vectors, one feature vector for each of the plurality of chunks, wherein the neural network is pre-trained to generate similar feature vector for similar chunks and dissimilar vector for dissimilar chunks;
assemble the plurality of feature vectors into a training vector;
generating a modified vector by modifying the training vector;
selecting a negative vector determined to be dissimilar to the training vector;
loading the training vector, the modified vector and the negative vector into the neural network to generate a file feature vector for each of the training vector, the modified vector and the negative vector; and,
adapting internal weights of the neural network to minimize a distance for the file feature vectors for the training vector and the modified vector, and maximize the distance for the file feature vectors for the training vector and the negative vector.

### Claim 13

A data processing system, comprising:
a memory storing instructions; and
a processor coupled to the memory to execute the instructions from the memory, the processor being configured to perform operations, the operations comprising:
for each file from a plurality of files:
chunking the file into a plurality of chunks;
loading the plurality of chunks into a neural network and operating the neural network to generate a plurality of feature vectors, one feature vector for each of the plurality of chunks, wherein the neural network is pre-trained to generate similar feature vector for similar chunks and dissimilar vector for dissimilar chunks;
assemble the plurality of feature vectors into a single vector;
loading the single vector into the neural network to generate a file feature vector, wherein the neural network is pre-trained to generate similar file feature vector for similar assembled feature vectors and dissimilar file feature vector for dissimilar assembled feature vectors; and
assembling the file feature vectors of the plurality of files into a file feature vector database.

### Claim 14 (depends on 13)

The system of claim 13, the operations further comprising converting each of the single vectors into a fixed-length single vector prior to loading the fixed-length single vector into the neural network.

### Claim 15 (depends on 14)

The system of claim 14, wherein the converting is performed by defining a plurality of bins, pooling the single vector into the plurality of bins, and concatenating the plurality of bins to generate the fixed-length single vector.

### Claim 16 (depends on 13)

The system of claim 13, further comprising, upon receiving a query file performing the operations:
chunking the query file into a plurality of chunks;
loading the plurality of chunks into the neural network and operating the neural network to generate a plurality of feature vectors, one feature vector for each of the plurality of chunks;
concatenating the plurality of feature vectors into a query vector;
loading the query vector into the neural network and operating the neural network to search the file feature vector database for all file feature vectors within a specified distance from the query vector.

### Claim 17 (depends on 16)

The system of claim 16, wherein a distance between the query vector and a file feature vector equals a distance between the query file and a file corresponding to the file feature vector to within a predefined error.

### Claim 18 (depends on 13)

The system of claim 13, further comprising pre-training the neural network by the process comprising:
obtaining a plurality of training files;
for each training file from a plurality of training files performing:
chunking the training file into a plurality of chunks;
loading the plurality of chunks into a neural network and operating the neural network to generate a plurality of feature vectors, one feature vector for each of the plurality of chunks, wherein the neural network is pre-trained to generate similar feature vector for similar chunks and dissimilar vector for dissimilar chunks;
assemble the plurality of feature vectors into a training vector;
generating a modified vector by modifying the training vector;
selecting a negative vector determined to be dissimilar to the training vector;
loading the training vector, the modified vector and the negative vector into the neural network to generate a file feature vector for each of the training vector, the modified vector and the negative vector; and,
adapting internal weights of the neural network to minimize a distance for the file feature vectors for the training vector and the modified vector, and maximize the distance for the file feature vectors for the training vector and the negative vector.

### Claim 19

A data processing system, comprising:
a backup database comprising a plurality of files;
a similarity index database comprising a plurality of similarity vectors, one similarity vector representing each of the files of the plurality of files, wherein each of the similarity vectors is generated from a concatenation of a plurality of chunk feature vectors, each of the chunk feature vectors representing a chunk of a file from the plurality of files;
a neural network pre-trained to receive a query file, generate a query vector, and provide an indication of all similarity vectors from the plurality of similarity vectors that are within a specified distance from the query vector.

### Claim 20 (depends on 19)

The data processing system of claim 19, further comprising a chunk feature database storing the plurality of chunk feature vectors, and wherein the neural network is further pre-trained to receive a query chunk, generate a chunk query vector, and provide an indication of all similarity chunk vectors from the plurality of chunk feature vectors that are within a specified distance from the chunk query vector.
