Apparatus and method for canceling interference in relay station in a communication system

An apparatus for canceling interference between a transmission signal and a received signal in a full-duplex relay station (RS) in a communication system, comprising:

• a receiver for receiving a radio frequency/intermediate frequency (RF/IF) signal from a first node through an antenna;
• a first converter for converting the received RF/IF signal to a baseband received signal;
• a reception module for demodulating and decoding the baseband received signal;
• a transmission module for generating a baseband transmission signal by encoding and modulating the demodulated and decoded data;
• a baseband interference controller for calculating a baseband interference estimate using the baseband received signal and the baseband transmission signal;
• a second converter for eliminating a baseband interference from the baseband received signal according to the baseband interference estimate and converting the baseband transmission signal to an RF/IF transmission signal to be transmitted to a second node;
• and
• an RF/IF interference controller for calculating an RF/IF interference estimate by converting amplitude of the RF/IF transmission signal and delaying the RF/IF transmission signal with the converted amplitude by a predetermined time period,
• wherein the receiver eliminates an RF/IF interference from the RF/IF received signal according to the RF/IF interference estimate.

A method for canceling interference between a transmission signal and a received signal in a full-duplex relay station (RS) in a communication system, comprising:

• converting a radio frequency/intermediate frequency (RF/IF) received signal to a baseband received signal;
• demodulating and decoding the baseband received signal;
• generating a baseband transmission signal by encoding and modulating the demodulated and decoded baseband received signal;
• calculating a baseband interference estimate;
• subtracting the baseband interference estimate from the baseband received signal;
• converting the baseband transmission signal to an RF/IF transmission signal;
• calculating an RF/IF interference estimate;
• and
• subtracting the RF/IF interference estimate from the RF/IF received signal.