# US 12,008,804 B2 · Transferring data between different frequency bands

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,804 B2
- Document ID: US12008804B2
- Status: Granted
- Kind: B2
- Application number: 17/510,305
- Filed: 2021-10-25
- Published: 2022-02-10
- Granted: 2024-06-11
- Earliest claimed: 2020-10-26
- Applicants: Mohammad Hadi Kefayati; Vahid Pourahmadi
- Inventors: Mohammad Hadi Kefayati; Vahid Pourahmadi
- Web page: https://patentgrep.com/patent/US12008804B2

## CPC

- G06V 10/95: {structured as a network, e.g. client-server architectures}
- H04B 7/0413: MIMO systems
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06T 3/4046: using neural networks
- G06T 7/194: involving foreground-background segmentation
- G06T 7/55: from multiple images
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- H01Q 21/0025: {Modular arrays}
- H04B 7/0469: {taking special antenna structures, e.g. cross polarized antennas into account}
- H04B 7/0626: {Channel coefficients, e.g. channel state information [CSI]}
- H04L 25/0204: {of multiple channels}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20224: Image subtraction
- H04N 7/18: Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04B: TRANSMISSION
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H01Q: ANTENNAS, i.e. RADIO AERIALS (radiators or antennas for microwave heating H05B6/72)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method for generating a data sample in a first frequency band from measurements in a second frequency band. The method includes obtaining a first plurality of samples, obtaining a second plurality of samples, obtaining a mapping model based on the first plurality of samples and the second plurality of samples, obtaining a third plurality of samples, and obtaining the data sample based on the mapping model and the third plurality of samples. Obtaining the first plurality of samples includes measuring a first frequency response of an environment in the first frequency band. Obtaining the second plurality of samples includes measuring a second frequency response of the environment in the second frequency band. Obtaining the third plurality of samples includes measuring a third frequency response of the environment in the second frequency band. Obtaining the data sample includes applying the mapping model on the third plurality of samples.

## Claims

### Claim 1

A method for transferring data between different frequency bands by generating an image from channel state information (CSI) samples in a communication system, the method comprising:
capturing, utilizing one or more processors, video data associated with a plurality of video frames by capturing video data associated with an ith video frame of the plurality of video frames where i≥1;
obtaining, utilizing the communication system, a first plurality of CSI samples by obtaining an ith subset of the first plurality of CSI samples simultaneously with capturing video data associated with the ith video frame;
obtaining, utilizing the one or more processors, a mapping model by training a neural network, comprising:
initializing the neural network with a plurality of initial weights; and
repeating an iterative process until a termination condition is satisfied, the iterative process comprising:
extracting a training image from an output of the neural network by applying the neural network on the ith subset of the first plurality of CSI samples, comprising:
obtaining a first plurality of feature maps by applying a first plurality of convolutional layers of the neural network on the ith subset of the first plurality of CSI samples;
obtaining a second plurality of feature maps by feeding the first plurality of feature maps to an input of a fully connected layer of the neural network;
obtaining a third plurality of feature maps by applying a residual neural network (ResNet) of the neural network on the second plurality of feature maps; and
upsampling the third plurality of feature maps;
generating a plurality of updated weights by minimizing a loss function of the training image and the ith video frame; and
replacing the plurality of updated weights with the plurality of initial weights;
obtaining, utilizing the communication system, a second plurality of CSI samples; and
obtaining, utilizing the one or more processors, the image by applying the mapping model on the second plurality of CSI samples.

### Claim 2

A method for transferring data between different frequency bands by generating a data sample in a first frequency band from measurements in a second frequency band, the method comprising:
obtaining, utilizing a first measuring system, a first plurality of samples by measuring a first frequency response of an environment in the first frequency band;
obtaining, utilizing a second measuring system, a second plurality of samples by measuring a second frequency response of the environment in the second frequency band;
obtaining, utilizing one or more processors, a mapping model based on the first plurality of samples and the second plurality of samples;
obtaining, utilizing the second measuring system, a third plurality of samples by measuring a third frequency response of the environment in the second frequency band; and
obtaining, utilizing the one or more processors, the data sample by applying the mapping model on the third plurality of samples.

### Claim 3 (depends on 2)

The method of claim 2, wherein:
measuring the first frequency response comprises capturing, utilizing a video device, a plurality of video frames by capturing video data associated with an ith video frame of the plurality of video frames in an ith time interval where i≥1;
measuring the second frequency response comprises obtaining, utilizing a communication system, a first plurality of channel state information (CSI) samples by obtaining an ith subset of the first plurality of CSI samples in the ith time interval;
measuring the third frequency response comprises obtaining a second plurality of CSI samples; and
obtaining the data sample comprises obtaining an image.

### Claim 4 (depends on 3)

The method of claim 3, wherein obtaining the ith subset of the first plurality of CSI samples comprises:
transmitting a plurality of transmit signals by a transmitter of the communication system;
receiving a plurality of receive signals associated with the plurality of transmit signals by a receiver of the communication system; and
generating the ith subset of the first plurality of CSI samples by:
obtaining a plurality of raw CSI samples based on the plurality of receive signals; and
extracting the ith subset of the first plurality of CSI samples from the plurality of raw CSI samples by compensating a phase offset of each of the plurality of raw CSI samples.

### Claim 5 (depends on 4)

The method of claim 4, wherein:
transmitting the plurality of transmit signals comprises transmitting each of the plurality of transmit signals in a respective sub-carrier of a plurality of sub-carriers and at a respective transmit moment of a plurality of transmit moments in the ith time interval by a respective transmit antenna of a plurality of transmit antennas associated with the transmitter;
receiving the plurality of receive signals comprises receiving each of the plurality of receive signals in a respective sub-carrier of the plurality of sub-carriers and at a respective receive moment of a plurality of receive moments in the ith time interval by a respective receive antenna of a plurality of receive antennas associated with the receiver; and
obtaining the plurality of raw CSI samples comprises obtaining a CSI array of size N1×N2×N3 by:
generating each of a plurality of CSI vectors of size
N
1
2
×
1
by estimating a respective multiple-input multiple output (MIMO) channel of a plurality of MIMO channels by processing a respective subset of the plurality of receive signals, each of the plurality of MIMO channels comprising a wireless channel between the transmitter and the receiver in a respective sub-carrier of the plurality of sub-carriers and at a respective receive moment of the plurality of receive moments; and
generating the CSI array by setting each element in the CSI array to one of a real part or an imaginary part of a respective element in a respective CSI vector of the plurality of CSI vectors,
wherein:
N1=2MtMr,
Mt is a number of the plurality of transmit antennas,
Mr is a number of the plurality of receive antennas,
N2 is a number of the plurality of sub-carriers, and
N3 is a number of the plurality of receive moments.

### Claim 6 (depends on 5)

The method of claim 5, wherein obtaining the plurality of raw CSI samples further comprises extracting a CSI sub-array of size N1×N2×N4 from the CSI array where 1≤N4<N3 by:
randomly selecting N4 sub-arrays of size N1×N2 out of N3 sub-arrays of size N1×N2 in the CSI array; and
generating the CSI sub-array by stacking N4 sub-arrays.

### Claim 7 (depends on 3)

The method of claim 3, wherein obtaining the mapping model comprises training a neural network by:
initializing the neural network with a plurality of initial weights; and
repeating an iterative process until a termination condition is satisfied, the iterative process comprising:
extracting a training image from an output of the neural network by applying the neural network on the ith subset of the first plurality of CSI samples;
generating a plurality of updated weights by minimizing a loss function of the training image and the ith video frame; and
replacing the plurality of updated weights with the plurality of initial weights.

### Claim 8 (depends on 7)

The method of claim 7, wherein applying the neural network on the ith subset of the first plurality of CSI samples comprises:
obtaining a first plurality of feature maps by applying a first plurality of convolutional layers of the neural network on the ith subset of the first plurality of CSI samples;
obtaining a second plurality of feature maps by feeding the first plurality of feature maps to an input of a fully connected layer of the neural network;
obtaining a third plurality of feature maps by applying a residual neural network (ResNet) of the neural network on the second plurality of feature maps; and
upsampling the third plurality of feature maps.

### Claim 9 (depends on 8)

The method of claim 8, wherein applying the first plurality of convolutional layers comprises extracting the first plurality of feature maps from an output of a (1, L1)th convolutional layer of the first plurality of convolutional layers where L1 is a number of the first plurality of convolutional layers, extracting the first plurality of feature maps comprising obtaining a (1, l1+1)th plurality of feature maps where 1≤l1≤L1 by:
generating a (1, L1)th plurality of filtered feature maps by applying a (1, l1)th plurality of filters on a (1, l1)th plurality of feature maps, a (1, 1)st plurality of feature maps comprising the ith subset of the first plurality of CSI samples;
generating a (1, l1)th plurality of normalized feature maps by applying an instance normalization process on the (1, l1)th plurality of filtered feature maps; and
generating the (1, l1+1)th plurality of feature maps by implementing a (1, l1)th non-linear activation function on each of the (1, l1)th plurality of normalized feature maps.

### Claim 10 (depends on 8)

The method of claim 8, wherein obtaining the second plurality of feature maps further comprises:
extracting an output vector from an output of the fully connected layer;
generating a first latent feature map by generating a matrix from a plurality of elements in the output vector;
obtaining a second latent feature map by applying a padding process on the first latent feature map;
obtaining a fourth plurality of feature maps by applying a second plurality of convolutional layers of the neural network on the second latent feature map by extracting the fourth plurality of feature maps from an output of a (2, L2)th convolutional layer of the second plurality of convolutional layers where L2 is a number of the second plurality of convolutional layers, extracting the fourth plurality of feature maps comprising obtaining a (2, l2+1)th plurality of feature maps where 1≤l2≤L2 by:
generating a (2, l2)th plurality of filtered feature maps by applying a (2, l2+1)th plurality of filters on a (2, l2)th plurality of feature maps, a (2, 1)st plurality of feature maps comprising the second latent feature map;
generating a (2, l2)th plurality of normalized feature maps by applying the instance normalization process on the (2, l2)th plurality of filtered feature maps; and
generating the (2, l2+1)th plurality of feature maps by implementing a (2, l2)th non-linear activation function on each of the (2, l2)th plurality of normalized feature maps; and
generating the second plurality of feature maps by applying the padding process on each of the fourth plurality of feature maps.

### Claim 11 (depends on 8)

The method of claim 8, wherein applying the ResNet comprises applying a plurality of ResNet blocks by extracting the third plurality of feature maps from an output of an Lrth ResNet block of the plurality of ResNet blocks where Lr is a number of the plurality of ResNet blocks, extracting the third plurality of feature maps comprising obtaining an (lr+1)th plurality of residual feature maps by applying an lrth ResNet block of the plurality of ResNet blocks on an lrth plurality of residual feature maps where 1≤lr≤Lr, a first plurality of residual feature maps comprising the second plurality of feature maps and the lrth ResNet block comprising two cascaded convolutional layers and a residual connection.

### Claim 12 (depends on 8)

The method of claim 8, wherein upsampling the third plurality of feature maps comprises extracting the training image from an output of a (3, L3)th convolutional layer of a third plurality of convolutional layers of the neural network where L3 is a number of the third plurality of convolutional layers, extracting the training image comprising obtaining a (3, l3+1)th plurality of feature maps where 1≤3≤L3 by:
generating an l3th plurality of upsampled feature maps by implementing an upsampling process on a (3, l3)th plurality of feature maps, a (3, 1)st plurality of feature maps comprising the third plurality of feature maps;
generating a (3, l3)th plurality of filtered feature maps by applying a (3, l3)th plurality of filters on the l3th plurality of upsampled feature maps;
generating a (3, l3)th plurality of normalized feature maps by applying the instance normalization process on the (3, l3)th plurality of filtered feature maps; and
generating the (3, l3+1)th plurality of feature maps by implementing a (3, l3)th non-linear activation function on each of the (3, l3)th plurality of normalized feature maps.

### Claim 13 (depends on 7)

The method of claim 7, wherein minimizing the loss function of the training image and the ith video frame comprises:
obtaining a mask image by extracting a plurality of masks from the ith video frame, each of the plurality of masks associated with a respective dynamic object of a plurality of dynamic objects in the ith video frame;
obtaining a background-subtracted image by multiplying the mask image by the ith video frame; and
minimizing the loss function of the training image and the background-subtracted image.

### Claim 14

A system for transferring data between different frequency bands by generating an image from channel state information (CSI) samples in a communication system, the system comprising:
a memory having processor-readable instructions stored therein; and
one or more processors configured to access the memory and execute the processor-readable instructions, which, when executed by the one or more processors configures the one or more processors to perform a method, the method comprising:
capturing video data associated with a plurality of video frames by capturing an ith video frame of the plurality of video frames in an ith time interval where i≥1;
obtaining, utilizing the communication system, a first plurality of CSI samples by obtaining an ith subset of the first plurality of CSI samples in the ith time interval;
obtaining a mapping model based on the plurality of video frames and the first plurality of CSI samples;
obtaining, utilizing the communication system, a second plurality of CSI samples; and
obtaining the image by applying the mapping model on the second plurality of CSI samples.

### Claim 15 (depends on 14)

The system of claim 14, wherein obtaining the ith subset of the first plurality of CSI samples comprises:
transmitting a plurality of transmit signals by a transmitter of the communication system, comprising transmitting each of the plurality of transmit signals in a respective sub-carrier of a plurality of sub-carriers and at a respective transmit moment of a plurality of transmit moments in the ith time interval by a respective transmit antenna of a plurality of transmit antennas associated with the transmitter;
receiving a plurality of receive signals associated with the plurality of transmit signals by a receiver of the communication system, comprising receiving each of the plurality of receive signals in a respective sub-carrier of the plurality of sub-carriers and at a respective receive moment of a plurality of receive moments in the ith time interval by a respective receive antenna of a plurality of receive antennas associated with the receiver; and
generating the ith subset of the first plurality of CSI samples by:
obtaining a plurality of raw CSI samples based on the plurality of receive signals; and
extracting the ith subset of the first plurality of CSI samples from the plurality of raw CSI samples by compensating a phase offset of each of the plurality of raw CSI samples.

### Claim 16 (depends on 15)

The system of claim 15, wherein obtaining the plurality of raw CSI samples comprises:
obtaining a CSI array of size N1×N2×N3 by:
generating each of a plurality of CSI vectors of size
N
1
2
×
1
by estimating a respective multiple-input multiple output (MIMO) channel of a plurality of MIMO channels by processing a respective subset of the plurality of receive signals, each of the plurality of MIMO channels comprising a wireless channel between the transmitter and the receiver in a respective sub-carrier of the plurality of sub-carriers and at a respective receive moment of the plurality of receive moments; and
generating the CSI array by setting each element in the CSI array to one of a real part or an imaginary part of a respective element in a respective CSI vector of the plurality of CSI vectors,
wherein:
N1=2MtMr,
Mt is a number of the plurality of transmit antennas,
Mr is a number of the plurality of receive antennas,
N2 is a number of the plurality of sub-carriers, and
N3 is a number of the plurality of receive moments; and
extracting a CSI sub-array of size N1×N2×N4 from the CSI array where 1≤N4<N3 by:
randomly selecting N4 sub-arrays of size N1×N2 out of N3 sub-arrays of size N1×N2 in the CSI array; and
generating the CSI sub-array by stacking N4 sub-arrays.

### Claim 17 (depends on 14)

The system of claim 14, wherein obtaining the mapping model comprises training a neural network by:
initializing the neural network with a plurality of initial weights; and
repeating an iterative process until a termination condition is satisfied, the iterative process comprising:
extracting a training image from an output of the neural network by applying the neural network on the ith subset of the first plurality of CSI samples, comprising:
obtaining a first plurality of feature maps by applying a first plurality of convolutional layers of the neural network on the ith subset of the first plurality of CSI samples;
obtaining a second plurality of feature maps by feeding the first plurality of feature maps to an input of a fully connected layer of the neural network;
obtaining a third plurality of feature maps by applying a residual neural network (ResNet) of the neural network on the second plurality of feature maps; and
upsampling the third plurality of feature maps;
generating a plurality of updated weights by minimizing a loss function of the training image and the ith video frame; and
replacing the plurality of updated weights with the plurality of initial weights.

### Claim 18 (depends on 17)

The system of claim 17, wherein:
applying the first plurality of convolutional layers comprises extracting the first plurality of feature maps from an output of a (1, L1)th convolutional layer of the first plurality of convolutional layers where L1 is a number of the first plurality of convolutional layers, extracting the first plurality of feature maps comprising obtaining a (1, l1+1)th plurality of feature maps where 1≤l1≤L1 by:
generating a (1, l1)th plurality of filtered feature maps by applying a (1, l1)th plurality of filters on a (1, l1)th plurality of feature maps, a (1, 1)st plurality of feature maps comprising the ith subset of the first plurality of CSI samples;
generating a (1, l1)th plurality of normalized feature maps by applying an instance normalization process on the (1, l1)th plurality of filtered feature maps; and
generating the (1, l1+1)th plurality of feature maps by implementing a (1, l1)th non-linear activation function on each of the (1, l1)th plurality of normalized feature maps; and
obtaining the second plurality of feature maps further comprises:
extracting an output vector from an output of the fully connected layer;
generating a first latent feature map by generating a matrix from a plurality of elements in the output vector;
obtaining a second latent feature map by applying a padding process on the first latent feature map;
obtaining a fourth plurality of feature maps by applying a second plurality of convolutional layers of the neural network on the second latent feature map by extracting the fourth plurality of feature maps from an output of a (2, L2)th convolutional layer of the second plurality of convolutional layers where L2 is a number of the second plurality of convolutional layers, extracting the fourth plurality of feature maps comprising obtaining a (2, l2+1)th plurality of feature maps where 1≤l2≤L2 by:
generating a (2, l2)th plurality of filtered feature maps by applying a (2, l2)th plurality of filters on a (2, l2)th plurality of feature maps, a (2, 1)st plurality of feature maps comprising the second latent feature map;
generating a (2, l2)th plurality of normalized feature maps by applying the instance normalization process on the (2, l2)th plurality of filtered feature maps; and
generating the (2, l2+1)th plurality of feature maps by implementing a (2, l2)th non-linear activation function on each of the (2, l2)th plurality of normalized feature maps; and
generating the second plurality of feature maps by applying the padding process on each of the fourth plurality of feature maps.

### Claim 19 (depends on 17)

The system of claim 17, wherein:
applying the ResNet comprises applying a plurality of ResNet blocks by extracting the third plurality of feature maps from an output of an Lrth ResNet block of the plurality of ResNet blocks where Lr is a number of the plurality of ResNet blocks, extracting the third plurality of feature maps comprising obtaining an (lr+1)th plurality of residual feature maps by applying an lthr ResNet block of the plurality of ResNet blocks on an lrth plurality of residual feature maps where 1≤lr≤Lr, a first plurality of residual feature maps comprising the second plurality of feature maps and the lrth ResNet block comprising two cascaded convolutional layers and a residual connection; and
upsampling the third plurality of feature maps comprises extracting the training image from an output of a (3, L3)th convolutional layer of a third plurality of convolutional layers of the neural network where L3 is a number of the third plurality of convolutional layers, extracting the training image comprising obtaining a (3, l3+1)th plurality of feature maps where 1≤l3≤L3 by:
generating an l3th plurality of upsampled feature maps by implementing an upsampling process on a (3, l3)th plurality of feature maps, a (3, 1)st plurality of feature maps comprising the third plurality of feature maps;
generating a (3, l3)th plurality of filtered feature maps by applying a (3, l3)th plurality of filters on the l3th plurality of upsampled feature maps;
generating a (3, l3)th plurality of normalized feature maps by applying the instance normalization process on the (3, l3)th plurality of filtered feature maps; and
generating the (3, l3+1)th plurality of feature maps by implementing a (3, l3)th non-linear activation function on each of the (3, l3)th plurality of normalized feature maps.

### Claim 20 (depends on 17)

The system of claim 17, wherein minimizing the loss function of the training image and the ith video frame comprises:
obtaining a mask image by extracting a plurality of masks from the ith video frame, each of the plurality of masks associated with a respective dynamic object of a plurality of dynamic objects in the ith video frame;
obtaining a background-subtracted image by multiplying the mask image by the ith video frame; and
minimizing the loss function of the training image and the background-subtracted image.