# US 12,217,169 B2 · Local neural implicit functions with modulated periodic activations

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,217,169 B2
- Document ID: US12217169B2
- Status: Granted
- Kind: B2
- Application number: 17/198,670
- Filed: 2021-03-11
- Published: 2022-09-15
- Granted: 2025-02-04
- Earliest claimed: 2021-03-11
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Ishit bhadresh Mehta; Michaël Gharbi; Connelly Barnes; Elya Shechtman
- Web page: https://patentgrep.com/patent/US12217169B2

## CPC

- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/048: Activation functions
- G06N 3/0499: Feedforward networks
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06T 3/4007: based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403)
- G06T 5/77: Retouching; Inpainting; Scratch removal
- G06T 2207/10016: Video; Image sequence
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Systems and methods for signal processing are described. Embodiments receive a digital signal comprising original signal values corresponding to a discrete set of original sample locations, generate modulation parameters based on the digital signal using a modulator network, wherein each of a plurality of modulator layers of the modulator network outputs a set of the modulation parameters, and generate a predicted signal value of the digital signal at an additional location using a synthesizer network, wherein each of a plurality of synthesizer layers of the synthesizer network operates based on the set of the modulation parameters from a corresponding modulator layer of the modulator network.

## Claims

### Claim 1

A method for signal processing, comprising:
receiving a digital signal comprising signal values corresponding to discrete sample locations;
generating modulation parameters based on the digital signal using a modulator network, wherein each of a plurality of modulator layers of the modulator network outputs a set of the modulation parameters; and
generating a predicted signal value of the digital signal at an additional location using a synthesizer network, computing, at a first synthesizer layer, a product of a set of modulation parameters output by a modulator layer and a continuous function of a first set of features to produce a second set of features, wherein the second set of features are input to a second synthesizer layer, wherein each of a plurality of synthesizer layers of the synthesizer network operates based on the set of the modulation parameters from a corresponding modulator layer of the modulator network.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
encoding at least a portion of a digital image to produce a latent vector, wherein the modulation parameters are generated based on the latent vector.

### Claim 3 (depends on 2)

The method of claim 2, wherein:
the discrete set original sample locations correspond to pixel location within the digital image, and the predicted signal value comprises color values at the additional location.

### Claim 4 (depends on 2)

The method of claim 2, further comprising:
inpainting at least one pixel of the digital image based on the predicted signal value, wherein the additional location comprises a location of the inpainted at least one pixel.

### Claim 5 (depends on 2)

The method of claim 2, further comprising:
generating a refined digital image based on the predicted signal value, wherein the refined digital image comprises a higher resolution than the digital image.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
interpolating an intermediate frame of a digital video based on the predicted signal value to produce an updated digital video with a higher frame rate than the digital video, wherein the digital signal comprises the digital video.

### Claim 7 (depends on 1)

The method of claim 1, wherein:
the digital signal comprises a digital audio signal.

### Claim 8 (depends on 1)

The method of claim 1, wherein:
the digital signal comprises a three dimensional (3D) image, and the original signal values represent distances from a surface of the 3D image.

### Claim 9 (depends on 1)

The method of claim 1, wherein:
the continuous function comprises a sine function, and the product comprises a Hadamard product.

### Claim 10 (depends on 1)

The method of claim 1, wherein:
the modulator network and the synthesizer network are trained based on training signals other than the digital signal.

### Claim 11 (depends on 1)

The method of claim 1, wherein:
the discrete set of original sample locations does not include the additional location.

### Claim 12

An apparatus for signal processing, comprising:
at least one processor;
at least one memory;
a modulator network comprising a plurality of modulator layers, wherein each of the plurality of modulator layers of the modulator network is configured to output a different set of modulation parameters based on a same digital signal;
an encoder configured to produce a latent vector representing the digital signal, wherein the modulator network takes the latent vector as input; and
a synthesizer network comprising a plurality of synthesizer layers, wherein the synthesizer network represents a continuous function of a signal parameter of the digital signal, and wherein each of the synthesizer layers is configured to receive the set of modulation parameters from a corresponding modulator layer of the modulator network.

### Claim 13 (depends on 12)

The apparatus of claim 12, wherein:
the modulator layers comprise multi-layer perceptron (MLP) layers with rectified linear unit (ReLU) activation.

### Claim 14 (depends on 12)

The method of claim 12, wherein:
the synthesizer layers comprise MLP layers with sine function activation.

### Claim 15

A method for training a neural network, comprising:
receiving a digital signal comprising signal values corresponding to discrete sample locations;
generating modulation parameters based on the digital signal using a modulator network, wherein each of a plurality of modulator layers of the modulator network outputs a set of the modulation parameters, encoding the digital signal to produce a latent vector, wherein the modulator network takes the latent vector as input to produce the modulation parameters;
generating a predicted signal value of the digital signal for at least one of the original sample locations using a synthesizer network, wherein each of a plurality of synthesizer layers of the synthesizer network receives the set of the modulation parameters from a corresponding modulator layer of the modulator network;
computing a loss function based on the predicted signal value and a value of the original signal values corresponding to the at least one of the original sample locations; and
training the modulator network and the synthesizer network based on the loss function.

### Claim 16 (depends on 15)

The method of claim 15, wherein:
the predicted signal values correspond to predicted color values of a training image at pixel locations corresponding to the original sample locations.

### Claim 17 (depends on 15)

The method of claim 15, wherein:
the training is based on an auto-encoder training process.