The Art of Recording the Right Amount of Data

Handle Large Data Files

Sample and Data Rates

The Nyquist theorem states that a signal must be acquired at a rate of at least twice the bandwidth, but it is desired to have a higher sample rate due to anti-alias filter roll-off and other parameters. In signal processing and RF it is common to work with IQ-data (In-phase and Quadrature data). The explanation of IQ-data can be found in both literature and online. Each sample in time is a pair of I and Q. A common oversampling factor is IQ rate=1.25*bandwidth which gives enough room for the antialiasing filter. A modern vector signal analyzer can have bandwidths of 800 MHz and beyond and with the oversampling just mentioned, the IQ-rate is 1 GHz. With 16-bit I and 16-bit Q the data rate becomes 4GByte/s. In RF monitoring applications the systems often need to monitor the spectrum for long periods of time and 24/7 operation is not uncommon. A data rate of 4 GB/s produces 345.6 TB of data per day which requires a lot of storage space and the storage space can also be quite expensive since it must be able to have high write speeds. One way around this is to only store the last hours of data and continuously delete the oldest recorded data. Storage devices do however have a lifespan and a measure of that is TBW, Terabytes Written. Consumer grade SSDs have a TBW up to 600 TBW for a 1 TB model. With the data rate mentioned above, 600 TBW would give less than 42 hours of recording time on a single device, which is not much for 24/7 operation. By increasing the number of drives or buying industry grade disk units with a TBW up to 2000 TBW for a 1 TB model, the time to failure improves but also the cost. Thus, recording all data is not always an option. Something else to consider is how to process large amount of data to be able to extract the needed information and where to do it. When the data has been stored and needs to be processed other bottlenecks become apparent. It is often a little bit faster to read data from the storage media than writing to it, but we are always limited by the slowest link in the system. Even if you have a RAID array with multiple NVMe drives that theoretically would deliver 72 GB/s just based on the number of drives and read/write speed, the data backplane, e.g., a PCIe bus, will limit the transfer rate. This bottleneck will be there no matter if the data is to be sent to the CPU for processing or sent to an external device like a back-end server. It does not make that much of a difference if you have the largest computational cluster in the world if you cannot send data to it fast enough. Because of this, it is important to find ways to only record the data that is needed. By doing that we will be able to reduce storage space, transfer time as well as processing time. The results are shorter time from recording to result and reduced cost.

Automatic Signal Detection

The above mentioned technologies can all be used manually by letting the operator configure the bandwidth and then start and stop the recording. A better way is to have the recording system automatically detect and record the signals of interest.

To determine when and for how long duration the signal is active or how much bandwidth it occupies, different kinds of signal detection algorithms can be used. Most general methods rely on detecting if there is an increase in energy. Two common methods are a frequency mask trigger and an analog edge trigger. The analog edge detection measures the signal level or power of the entire band and triggers when it goes above a threshold. Even if it is called analog edge, it is often done digitally. With this method it is possible to detect very short signals and get a very accurate trigger time for when the signal started. One downside is that all frequencies in the band will be triggered on, so if there are multiple signals in the band it may trigger on the wrong signal. The frequency mask trigger is built on Fast Fourier Transforms, FFT. The FFT divides the spectrum into many frequency bins and the signal level in each bin can be compared to a threshold. An FFT is calculated on a certain number of samples, e.g., 1024. A longer FFT gives better frequency resolution but requires more samples and thus has worse time resolution. For triggered applications, even if not all data is saved, all data must be processed in real-time to be able to detect if a trigger condition has occurred. As mentioned before, when the bandwidths become wider the amount of data increases which puts more strain on the triggering task. Processing 4GB/s in a deterministic way on a CPU is challenging which is why the task of triggering is typically handed off to hardware like an FPGA which continuously will be able to process all data being sent to it. A common measure of FFT-based systems is Probability of Intercept, POI. 100% Probability Of Intercept is a measure for how long time a signal must be active so that the system is guaranteed to detect it at full amplitude with 100% certainty. For this to happen, the signal must be present during at least one entire FFT and by changing the parameters sample rate, FFT-length and FFT overlap the POI number can be tailored to the application.