# US 2026/0310953 A1 · Optical sensing with nonlinear optical neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0310953 A1
- Document ID: US20260310953A1
- Status: Published application
- Kind: A1
- Application number: 18/998,430
- Filed: 2023-07-25
- Published: 2026-10-08
- Earliest claimed: 2022-07-25
- Assignees: Center for Technology Licensing Cornell University
- Applicants: NTT Research, Inc.; Center for Technology Licensing, Cornell University
- Inventors: Tianyu Wang; Mandar Sohoni; Logan Wright; Peter McMahon
- Web page: https://patentgrep.com/patent/US20260310953A1

## CPC

- G06N 3/067: using optical means
- G06V 10/766: using regression, e.g. by projecting features on hyperplanes
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/776: Validation; Performance evaluation
- G06V 10/82: using neural networks
- G06V 10/88: Image or video recognition using optical means, e.g. reference filters, holographic masks, frequency domain filters or spatial domain filters
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Methods, devices, and systems for optical sensing with nonlinear optical neural network (ONNs) are provided. In one aspect, a method includes: receiving light from a visual scene by a first optical linear layer in a nonlinear ONN apparatus, linearly transforming the light into first optical outputs by the first optical linear layer trained to perform a first optical linear operation, non-linearly generating second optical outputs based on the first optical outputs by an optical nonlinear layer in the ONN apparatus, and linearly transforming the second optical outputs into third optical outputs by a second optical linear layer in the ONN apparatus trained to perform a second optical linear operation. The first optical linear layer, the optical nonlinear layer, and the second optical linear layer are sequentially arranged in series in the ONN apparatus.

## Claims

### Claim 1

(canceled)

### Claim at position 2

-43. (canceled)

### Claim 44

A method, comprising:
receiving light from a visual scene by a first optical linear layer in a nonlinear optical neural network (ONN) apparatus;
linearly transforming the light into first optical outputs by the first optical linear layer that is trained to perform a first optical linear operation;
nonlinearly generating second optical outputs based on the first optical outputs by an optical nonlinear layer in the ONN apparatus; and
linearly transforming the second optical outputs into third optical outputs by a second optical linear layer in the ONN apparatus that is trained to perform a second optical linear operation,
wherein the first optical linear layer, the optical nonlinear layer, and the second optical linear layer are sequentially arranged in series in the ONN apparatus.

### Claim 45 (depends on 44)

The method of claim 44, further comprising:
performing, by an optoelectronic device coupled to the ONN apparatus, an optical-to-electrical (OE) conversion of an optical output signal of the ONN apparatus into an electrical data signal.

### Claim 46 (depends on 45)

The method of claim 45, further comprising:
processing, by a computing device coupled to the optoelectronic device, digital data associated with the electrical data signal to generate a digital output corresponding to the visual scene,
wherein the digital output identifies one or more objects in the visual scene, and
wherein an identification accuracy of the digital output is higher than an identification accuracy of a digital output using a second apparatus that includes an optical linear layer followed by a digital linear layer without nonlinearity.

### Claim 47 (depends on 44)

The method of claim 44, wherein an information percentage of an object at interest in an optical output signal from the nonlinear ONN apparatus is at least one order of magnitude higher than an information percentage of the object at interest in the light from the visual scene received by the nonlinear ONN apparatus.

### Claim 48 (depends on 44)

The method of claim 44, wherein the light comprises incoherent light or coherent light.

### Claim 49 (depends on 44)

The method of claim 44, wherein the light comprises different colors of light corresponding to a plurality of wavelengths, and
wherein at least one of the first optical linear layer or the second optical linear layer is configured to perform a respective optical linear operation for each of the plurality of wavelengths.

### Claim 50 (depends on 44)

The method of claim 44, wherein the light from the visual scene comprises at least one of reflected light, scattered light, transmitted light, diffracted light, or excited light.

### Claim 51 (depends on 44)

The method of claim 44, wherein the visual scene comprises at least one of: an image, a video, a real-life scene, a virtual scene, a two-dimensional (2D) object, a three-dimensional (3D) object, an image of a plurality of drawings corresponding to one or more objects, one or more fluorescence images of biological objects, or one or more real-scene objects.

### Claim 52 (depends on 44)

The method of claim 44, wherein at least one of the first optical linear operation or the second optical linear operation comprises at least one of multiplication, convolution, or multiplexing.

### Claim 53 (depends on 44)

The method of claim 44, wherein at least one of the first optical linear layer and the second optical linear layer comprises:
an optical fan-out device configured to multiplex one optical input into multiple optical inputs;
an optical modulator arranged downstream of the optical fan-out device and configured to modulate corresponding weights to the multiple optical inputs to generate multiple weighted optical inputs; and
an optical fan-in device arranged downstream of the optical modulator and configured to sum up the multiple weighted optical inputs into one or more optical outputs, wherein each of the one or more optical outputs corresponds to two or more respective weighted optical inputs.

### Claim 54 (depends on 53)

The method of claim 53, wherein the optical fan-out device comprises a microlens array, wherein the optical modulator comprises a spatial light modulator (SLM), and wherein the optical fan-in device comprises an optical lens, a microlens array, or a waveguide.

### Claim 55 (depends on 44)

The method of claim 44, wherein the first optical linear layer and the second optical linear layer are trained together in a digital neural network model corresponding to the ONN apparatus.

### Claim 56 (depends on 44)

The method of claim 44, wherein the optical nonlinear layer comprises an optical nonlinear device configured to perform an optical nonlinear operation, and
wherein the optical nonlinear operation comprises at least one of optical amplifying, saturating, or rectifying and attenuating.

### Claim 57 (depends on 56)

The method of claim 56, wherein the optical nonlinear operation corresponds to a ReLU function, a sigmoid function, or an unconventional nonlinear function that is different from a nonlinear function used in digital computing,
wherein the optical nonlinear device comprises an optical intensifier, a photoconductor configured to response nonlinearly to light intensity, a light source configured to response nonlinearly to a driving current, or a nonlinear optical medium, and
wherein the optical nonlinear operation is unrelated to the first optical linear operation and the second optical linear operation.

### Claim 58

The method of claim wherein the optical nonlinear layer comprises an optical nonlinear device,
wherein the method further comprises: calibrating the optical nonlinear device to determine at least one property of the optical nonlinear device, and
wherein calibrating the optical nonlinear device comprises:
measuring an input intensity into the optical nonlinear device;
measuring an output intensity from the optical nonlinear device; and
determining a mathematical function or model between the input intensity and the output intensity.

### Claim 59 (depends on 58)

The method of claim 58, further comprising:
building a digital neural network model corresponding to the ONN apparatus based on the at least one property of the optical nonlinear device;
training the digital neural network model to determine one or more parameters for the first optical linear layer and the second optical linear layer;
controlling the first optical linear layer and the second optical linear layer in the ONN apparatus with the determined parameters;
measuring the first optical outputs for the first optical linear layer;
measuring the third optical outputs for the second optical linear layer,
first comparing the measured first optical outputs to first ground truth information for the first optical linear layer;
second comparing the measured third optical outputs to second ground truth information for the second optical linear layer; and
calibrating the ONN apparatus or re-training the digital neural network model based on a first result of the first comparing and a second result of the second comparing,
wherein training the digital neural network model by adjusting the one or more parameters for the first optical linear layer and the second optical linear layer to achieve a performance result substantially close to a performance result obtained by using a ground truth neural network, and
wherein the method further comprises:
performing data augmentation on training data of the digital neural network model with random image misalignments and convolutions; and
performing layer-by-layer fine tuning of the digital neural network model with experimentally collected data.

### Claim 60 (depends on 44)

The method of claim 44, further comprising:
using feature vectors produced by the ONN apparatus as input to a digital backend for a further operation that comprises at least one of neural-network decoding or image reconstruction, unsupervised learning, or nonlinear regression.

### Claim 61 (depends on 44)

The method of claim 44, wherein the ONN apparatus comprises a plurality of optical nonlinear layers, each of the plurality of optical nonlinear layers being coupled between adjacent optical linear layers, and
wherein the ONN apparatus comprises multiple pairs of optical nonlinear layer and optical linear layer that are sequentially arranged downstream the first optical linear layer.

### Claim 62

An optical neural network (ONN) apparatus, comprising:
a first optical linear layer configured to receive light from a visual scene and linearly transform the light into first optical outputs according to a first optical linear operation;
an optical nonlinear layer coupled to the first optical linear layer and configured to nonlinearly generate second optical outputs based on the first optical outputs; and
a second optical linear layer coupled to the optical nonlinear layer and configured to linearly transform the second optical outputs into third optical outputs according to a second optical linear operation,
wherein the first optical linear layer, the optical nonlinear layer, and the second optical linear layer are sequentially coupled in series in the ONN apparatus.

### Claim 63

A system, comprising:
an optical neural network (ONN) apparatus comprising:
a first optical linear layer configured to receive light from a visual scene and linearly transform the light into first optical outputs according to a first optical linear operation;
an optical nonlinear layer coupled to the first optical linear layer and configured to nonlinearly generate second optical outputs based on the first optical outputs; and
a second optical linear layer coupled to the optical nonlinear layer and configured to linearly transform the second optical outputs into third optical outputs according to a second optical linear operation,
wherein the first optical linear layer, the optical nonlinear layer, and the second optical linear layer are sequentially coupled in series in the ONN apparatus;
an optoelectronic device coupled to the ONN apparatus and configured to perform an optical-to-electric (OE) conversion of an optical signal output of the ONN apparatus into an electric data signal, the optical signal output of the ONN apparatus being associated with the third optical outputs; and
a computing device coupled to the optoelectronic device and configured to process digital data associated with the electric data signal to generate a digital output corresponding to the visual scene.
