# US 2023/0053618 A1 · Recurrent unit for generating or processing a sequence of images > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2023/0053618 A1 - Document ID: US20230053618A1 - Status: Published application - Kind: A1 - Application number: 17/797,198 - Filed: 2021-02-08 - Published: 2023-02-23 - Earliest claimed: 2020-02-07 - Assignees: DeepMind Technologies Limited - Applicants: DeepMind Technologies Limited - Inventors: Pauline Luc; Aidan Clark; Sander Etienne Lea Dieleman; Karen Simonyan - Web page: https://patentgrep.com/patent/US20230053618A1 ## CPC - G06N 3/045: Combinations of networks - G06T 11/00: Two-dimensional [2D] image generation - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0475: Generative networks - G06N 3/09: Supervised learning - G06N 3/092: Reinforcement learning - G06N 3/094: Adversarial learning - G06T 3/18: Image warping, e.g. rearranging pixels individually - G06T 3/4046: using neural networks - G06T 3/4092: Image resolution transcoding, e.g. by using client-server architectures - G06T 7/10: Segmentation; Edge detection (motion-based segmentation G06T7/215) - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - G06T 2207/10016: Video; Image sequence - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/20221: Image fusion; Image merging - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A recurrent unit is proposed which, at each of a series of time steps receives a corresponding input vector and generates an output at the time step having at least one component for each of a two-dimensional array of pixels. The recurrent unit is configured, at each of the series of time steps except the first, to receive the output of the recurrent unit at the preceding time step, and to apply to the output of the recurrent unit at the preceding time step at least one convolution which depends on the input vector at the time step. The convolution further depends upon the output of the recurrent unit at the preceding time step. This convolution generates a warped dataset which has at least one component for each pixel of the array. The output of the recurrent unit at each time step is based on the warped dataset and the input vector. ## Claims ### Claim 1 A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to implement: a recurrent unit arranged, at each of a series of time steps, to receive a corresponding input vector and to generate an output having at least one respective value for each of a two-dimensional array of pixels, the recurrent unit being configured at each of the series of time steps except the first: to receive the output of the recurrent unit at the preceding time step, to apply to the output of the recurrent unit at the preceding time step at least one convolution dependent on the input vector at the time step and on the output of the recurrent unit at the preceding time step, to generate a warped dataset which has at least one component for each pixel of the array, and to generate the output at the time step based on the warped dataset and the input vector. ### Claim 2 (depends on 1) A system according to claim 1 in which the recurrent unit is operative to generate the at least one component of the warped dataset for each pixel of the array by convolving the output of the recurrent unit at the preceding time step in a respective manner for each pixel. ### Claim 3 (depends on 2) A system according to claim 2 wherein the recurrent unit is configured to generate a respective kernel for each pixel of the array using the input vector, and to generate the at least one component of the warped dataset for each pixel based on the corresponding kernel. ### Claim 4 (depends on 2) A system according to claim 2 wherein the recurrent unit is configured to: generate the at least one component of the warped dataset as a weighted sum of convolutions of the corresponding component of the output of the recurrent unit at the preceding time step with a respective plurality of kernels which are each dependent on the input vector and the output of the recurrent unit at the preceding time step, the weights of the weighted sum being different for different said pixels of the array. ### Claim 5 (depends on 1) A system according to claim 1 wherein the recurrent unit is configured to generate the output at each time step as a sum of (i) a component-wise product of the warped dataset with a fusion vector, and (ii) a component-wise product of a vector varying inversely with the fusion vector and a refined vector generated by a rectified linear unit of the recurrent unit. ### Claim 6 (depends on 5) A system according to claim 5 wherein the recurrent unit is configured to generate each element of the fusion vector by applying a function to: a respective component of a component-wise product of a first weight vector with a concatenation of the output of the network at the preceding time step and the input vector plus a respective first offset value. ### Claim 7 (depends on 5) A system according to claim 5 wherein the recurrent unit is configured to generate each element of the fusion vector by applying a function to: a respective component of a component-wise product of a first weight vector with a concatenation of the warped dataset and the input vector plus a respective first offset value. ### Claim 8 (depends on 5) A system according to claim 5 in which the rectified linear unit is configured to generate each element of the refined vector by applying a rectified linear function to: a respective component of a component-wise product of a second weight vector with a concatenation of the output of the network at the preceding time step and the input vector plus a respective second offset value. ### Claim 9 (depends on 5) A system according to claim 5 in which the rectified linear unit is configured to generate each element of the refined vector by applying a rectified linear function to: respective components of a component-wise product of a second weight vector with a concatenation of the output of the warped dataset and the input vector plus a respective second offset value. ### Claim 10 (depends on 1) A system according to claim 1, wherein the instructions further cause the one or more computers to implement a generator network to generate a sequence of images representing a temporal progression and composed of values for each of a two-dimensional array of pixels, the generator network comprising a recurrent unit according to any preceding claim, the generator network being configured to generate each of the sequence of images based on the respective output of the recurrent unit in a respective one of the time steps. ### Claim 11 (depends on 1) A system according to claim 1, wherein the instructions further cause the one or more computers to implement a segmentation network to identify within a sequence of images a portion of each image having one or more characteristics, the segmentation network comprising the recurrent unit, the recurrent unit being arranged at each of series of time steps to receive an input vector comprising a corresponding one of the sequence of images, the segmentation network being configured to generate in each time step data from the output of the recurrent unit in the corresponding time step and which identifies a portion of the corresponding image. ### Claim 12 (depends on 1) A system according to claim 1, wherein the instructions further cause the one or more computers to implement a classification network to generate data which classifies a sequence of images as being in one or more of a set of classes, the classification network comprising the recurrent unit, the recurrent unit being arranged at each of series of time steps to receive an input vector comprising a corresponding one of the sequence of images, the classification network being configured to generate from the outputs of the recurrent unit at each of the respective series of time steps data identifying one or more of the classes. ### Claim 13 (depends on 1) A system according to claim 1, wherein the instructions further cause the one or more computers to implement an adaptive system for increasing the spatial and/or temporal resolution of a sequence of images, the adaptive system comprising the recurrent unit, the recurrent unit being arranged at each of a series of time steps to receive an input vector comprising a one of a first sequence of images, the adaptive system being configured to generate a sequence of images having higher spatial and/or temporal resolution than images of the first sequence of images. ### Claim 14 A computer-implemented method of jointly training a generator network and a discriminator network, the discriminator network being for distinguishing between sequences of images generated by the generator network and sequences of images which are not generated by the generator network, the method comprising: receiving one or more first sequences of images representing a temporal progression; and repeatedly performing the iteration steps of: generating, by the generator network, one or more second sequences of images; generating, by the discriminator network, at least one discriminator score for one or more of the first sequences of images and for each of the second sequence of images; and varying weights of at least one of the discriminator network and the generator network based on the at least one discriminator score. ### Claim 15 (depends on 14) A computer-implemented method according to claim 14, in which the discriminator network comprises a spatio-temporal discriminator network for discriminating based on temporal features and a spatial discriminator network for discriminating based on spatial features, the spatio-temporal discriminator network and the spatial discriminator network each comprising a multi-layer network of neurons in which each layer performs a function defined by corresponding weights; said generation of the at least one discriminator score comprising: (i) forming, from the input sequence, a first set of one or more images having a lower temporal resolution than the input sequence, and inputting the first set into the spatial discriminator network to determine, based on the spatial features of each image in the first set, a first discriminator score representing a probability that the input sequence has been generated by the generator network; and (ii) forming, from the input sequence, a second set of images having a lower spatial resolution than the input sequence, and inputting the second set into the spatio-temporal discriminator network to determine, based on the temporal features of the images in the second set, a second discriminator score representing a probability that the input sequence has been generated by the generator network; and said varying the weights of at least one of the discriminator network and the generator network comprising updating the weights based on the first discriminator score and the second discriminator score. ### Claim at position 16 16-23. (canceled) ### Claim 24 One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to implement: a recurrent unit arranged, at each of a series of time steps, to receive a corresponding input vector and to generate an output having at least one respective value for each of a two-dimensional array of pixels, the recurrent unit being configured at each of the series of time steps except the first: to receive the output of the recurrent unit at the preceding time step, to apply to the output of the recurrent unit at the preceding time step at least one convolution dependent on the input vector at the time step and on the output of the recurrent unit at the preceding time step, to generate a warped dataset which has at least one component for each pixel of the array, and to generate the output at the time step based on the warped dataset and the input vector. ### Claim 25 (depends on 24) One or more non-transitory computer-readable storage media according to claim 24 in which the recurrent unit is operative to generate the at least one component of the warped dataset for each pixel of the array by convolving the output of the recurrent unit at the preceding time step in a respective manner for each pixel. ### Claim 26 (depends on 25) One or more non-transitory computer-readable storage media according to claim 25 wherein the recurrent unit is configured to generate a respective kernel for each pixel of the array using the input vector, and to generate the at least one component of the warped dataset for each pixel based on the corresponding kernel. ### Claim 27 (depends on 25) One or more non-transitory computer-readable storage media according to claim 25 wherein the recurrent unit is configured to: generate the at least one component of the warped dataset as a weighted sum of convolutions of the corresponding component of the output of the recurrent unit at the preceding time step with a respective plurality of kernels which are each dependent on the input vector and the output of the recurrent unit at the preceding time step, the weights of the weighted sum being different for different said pixels of the array. ### Claim 28 (depends on 24) One or more non-transitory computer-readable storage media according to claim 24 wherein the recurrent unit is configured to generate the output at each time step as a sum of (i) a component-wise product of the warped dataset with a fusion vector, and (ii) a component-wise product of a vector varying inversely with the fusion vector and a refined vector generated by a rectified linear unit of the recurrent unit.