Back to results
US 12,755,515 B2Granted
Method of obtaining an attention matrix for use in a transformer-based model, non-transitory computer readable storage medium and apparatus
AssigneeMilestone Systems A/S
InventorMarc Oliu Simón
Life of this patent
Estimating the term…
Abstract
A method of obtaining an attention matrix for use in a transformer-based model, a non-transitory computer readable storage medium, and an apparatus are provided. The method includes applying an attentional mechanism over a quadtree representation of an input image. The method may be used for image classification, for example in the field of video surveillance.
Claims18
A method of obtaining an attention matrix for use in a transformer-based machine learning model, the method comprising:
- inputting a d-dimensional piece of data, where d represents the number of dimensions of the data, Di represents the length of the i-th dimension for i∈[1, . . . , d] and D1× . . . ×Dd corresponds to the size of the input data,
- factorizing each of the lengths of the input into the form
- D
- i
- =
- ∏
- j
- =
- 1
- k
-
- p
- j
- n
- j
- ,
- wherein pj is the j-th prime factor and nj is the number of repetitions of said factor,
- representing the input X of size
- (
- ∏
- i
- =
- 1
- k
-
- p
- j
- )
- ×
- …
- ×
- (
- ∏
- i
- =
- 1
- k
-
- p
- j
- )
- as:
- X
- =
- ℝ
- (
- (
- D
- 1
- )
- 1
- ×
- …
- ×
- (
- D
- 1
- )
- m
- )
- ×
- …
- ×
- (
- (
- D
- d
- )
- 1
- ×
- …
- ×
- (
- D
- d
- )
- m
- )
- ×
- F
- >
- where |(Di)j|=pk for j∈[1, . . . , m] and |F| corresponds to the feature length of the feature vector of each spatial location,
- reshuffling the representation of the input to represent the input as:
- X
- =
- ℝ
- (
- (
- D
- 1
- )
- 1
- ×
- …
- ×
- (
- D
- d
- )
- 1
- )
- ×
- …
- ×
- (
- (
- D
- 1
- )
- m
- ×
- …
- ×
- (
- D
- d
- )
- m
- )
- ×
- F
- >
- reshaping the representation of the input to form a first generation quadtree representation of the input as:
- X
- =
- ℝ
- a
- 1
- ×
- …
- ×
- a
- m
- ×
- F
- >
- wherein |ai|=|(D1)i|× . . . ×|(Dd)i| for i∈[1, . . . , m], wherein the spatial axes are the axes of the quadtree representation excluding the feature vector F,
- wherein the m spatial axes are ordered by granularity such that the rightmost axis represents the finest granularity level of the quadtree representation of the input,
- selecting a first set of k consecutive spatial axes of the quadtree representation to form a first attention window, wherein 1≤k≤m,
- computing an attention matrix over the first attention window, with the remaining spatial axes of the quadtree representation being a batch size, the attention matrix being applied to an image without requiring the image to be sub-sampled.
137 more elements
A non-transitory computer readable storage medium storing a program for causing a computer to execute a method of obtaining an attention matrix for use in a transformer-based machine learning model, the method comprising:
- inputting a d-dimensional piece of data, where d represents the number of dimensions of the data, Di represents the length of the i-th dimension for i∈[1, . . . , d] and D1× . . . ×Dd corresponds to the size of the input data,
- factorizing each of the lengths of the input into the form
- D
- i
- =
- ∏
- j
- =
- 1
- k
-
- p
- j
- n
- j
- ,
- wherein pj is the j-th prime factor and nj is the number of repetitions of said factor,
- representing the input X of size
- (
- ∏
- i
- =
- 1
- k
-
- p
- j
- )
- ×
- …
- ×
- (
- ∏
- i
- =
- 1
- k
-
- p
- j
- )
- as:
- X
- =
- ℝ
- (
- (
- D
- 1
- )
- 1
- ×
- …
- ×
- (
- D
- 1
- )
- m
- )
- ×
- …
- ×
- (
- (
- D
- d
- )
- 1
- ×
- …
- ×
- (
- D
- d
- )
- m
- )
- ×
- F
- >
- where |(Di)j|=pk for j∈[1, . . . , m] and |F| corresponds to the feature length of the feature vector of each spatial location,
- reshuffling the representation of the input to represent the input as:
- X
- =
- ℝ
- (
- (
- D
- 1
- )
- 1
- ×
- …
- ×
- (
- D
- d
- )
- 1
- )
- ×
- …
- ×
- (
- (
- D
- 1
- )
- m
- ×
- …
- ×
- (
- D
- d
- )
- m
- )
- ×
- F
- >
- reshaping the representation of the input to form a first generation quadtree representation of the input as:
- X
- =
- ℝ
- a
- 1
- ×
- …
- ×
- a
- m
- ×
- F
- >
- wherein |ai|=|(D1)i|× . . . ×|(Dd)i| for i∈[1, . . . , m], wherein the spatial axes are the axes of the quadtree representation excluding the feature vector F,
- wherein the m spatial axes are ordered by granularity such that the rightmost axis represents the finest granularity level of the quadtree representation of the input,
- selecting a first set of k consecutive spatial axes of the quadtree representation to form a first attention window, wherein 1≤k≤m,
- computing an attention matrix over the first attention window, with the remaining spatial axes of the quadtree representation being a batch size, the attention matrix being applied to an image without requiring the image to be sub-sampled.
137 more elements
An apparatus for obtaining an attention matrix for use in a transformer-based machine learning model, the apparatus having at least one processor configured to:
- input a d-dimensional piece of data, where d represents the number of dimensions of the data, Di represents the length of the i-th dimension for i∈[1, . . . , d] and D1× . . . ×Dd corresponds to the size of the input data,
- factorize each of the lengths of the input into the form
- D
- i
- =
- ∏
- j
- =
- 1
- k
-
- p
- j
- n
- j
- ,
- wherein pj is the j-th prime factor and nj is the number of repetitions of said factor,
- represent the input X of size
- (
- ∏
- i
- =
- 1
- k
-
- p
- j
- )
- ×
- …
- ×
- (
- ∏
- i
- =
- 1
- k
-
- p
- j
- )
- as:
- X
- =
- ℝ
- (
- (
- D
- 1
- )
- 1
- ×
- …
- ×
- (
- D
- 1
- )
- m
- )
- ×
- …
- ×
- (
- (
- D
- d
- )
- 1
- ×
- …
- ×
- (
- D
- d
- )
- m
- )
- ×
- F
- >
- where |(Di)j|=pk for j∈[1, . . . , m] and |F| corresponds to the feature length of the feature vector of each spatial location,
- reshuffle the representation of the input to represent the input as:
- X
- =
- ℝ
- (
- (
- D
- 1
- )
- 1
- ×
- …
- ×
- (
- D
- d
- )
- 1
- )
- ×
- …
- ×
- (
- (
- D
- 1
- )
- m
- ×
- …
- ×
- (
- D
- d
- )
- m
- )
- ×
- F
- >
- reshape the representation of the input to form a first generation quadtree representation of the input as:
- X
- =
- ℝ
- a
- 1
- ×
- …
- ×
- a
- m
- ×
- F
- >
- wherein |ai|=|(D1)i|× . . . ×|(Dd)i| for i∈[1, . . . , m], wherein the spatial axes are the axes of the quadtree representation excluding the feature vector F,
- wherein the m spatial axes are ordered by granularity such that the rightmost axis represents the finest granularity level of the quadtree representation of the input,
- select a first set of k consecutive spatial axes of the quadtree representation to form a first attention window, wherein 1≤k≤m,
- compute an attention matrix over the first attention window, with the remaining spatial axes of the quadtree representation being a batch size, the attention matrix being applied to an image without requiring the image to be sub-sampled.
137 more elements
Dependencies are inferred from the claim text.