# Detecting signals using statistics

I have sampled a section of spectrum every few Hz. I have mean, standard deviation, and range data for every frequency sampled (and I can gather more statistics if necessary). I've sampled for several seconds before storing the spectrum statistics.

I would like to be able to guess where man-made signals (i.e. channels) exist in this spectrum (as opposed to noise floor). This is proving to be a difficult task, especially since the noise-floor power sometimes changes (different antenna inputs for different bands, for example.)

I doubt anyone has a ready solution for this, but what are some starting points, or tips, to help me begin to identify channels?

In the below image 'normal' is really autocorrelation from Pandas

You might try taking a image capture of the spectrum waterfall for some duration, and feeding that image to a machine learning inference engine, perhaps a DNN.

The inference engine could be trained on a large image database with lots of waterfalls of lots of known or suspected signal types, similar to these signal ID databases:

• An answer that addresses what I asked, and is creative and potentially effective. Thank you – HH- Apologize to Carole Baskin Sep 22 '17 at 17:39

There is a range of possibilities, depending on what kind of signal you want to find. I'll start from easy and move up to hard. I'm assuming that you are using an FFT to get your spectra.

1. RFI. An earlier poster referenced some papers on finding RFI. I don't know precisely how that is defined, but lets assume that it is unintentional RF from things like switching power supplies (line spectra) and poorly wired auto engines (impulsive noise). Impulsive noise is narrow in time and wide in frequency, which suggests that you simply have a look at your receiver output. There is probably some statistical test you could apply (say, the null hypothesis is white Gaussian noise). For line spectra, do the same in the frequency domain. Perhaps those who know more about RFI than I do would say otherwise. I would suggest, however, that rather than process your signal as you're doing, that you use the initial processing that I discuss next.
2. Conventional analog signals. By this I mean the AM, FM, SSB, etc. transmissions that you run across all the time. These live in an intermediate region between impulses and line spectra, which makes things a bit harder. I would suggest that you compute a power spectrum, and from that, the autocorrelation. This is easy if you are already using an FFT: compute the FFT, then multiply each element by its complex conjugate. Now you have the power spectrum. Next do the inverse FFT to get the autocorrelation. If there's a signal there, you'll see a smeared out peak near $\tau=0$, where $\tau$ is the delay. If it's relatively wideband, the peak will be narrower; if it's narrowband (compared to your FFT) the peak will be wider. Look for peaks in both the power spectrum and the autocorrelation. This works better than statistics-gathering the way you describe because it takes advantage of the coherence of the signal. If you use a large FFT, you'll be averaging over a lot of signal, which will suppress the noise and better show you what's there. If it's just noise, you'll see a very narrow peak right at $\tau=0$ and a flat power spectrum.
3. Conventional digital signals. I mean conventional from a ham radio perspective; modes like PSK31, etc. An autocorrelation/power spectrum approach can work well here, too. The same physical principles apply: noise averages incoherently, while signal averages coherently. There is a caveat, however. As signals get more bandwidth-efficient, they start to look more like white noise. An AFSK signal like those often used in packet radio, for example, is terribly (forgive me, packet radio enthusiasts) inefficient. It takes a lot of bandwidth to get a few bits through. PSK31 is much better. Some of the more modern modes, such as WSPR and relatives, are really good. You'll see a flat spectrum across the entire signal bandwidth. They're not trying to be sneaky. That's just the physics of the situation.
4. Wideband digital signals. This is a broad category, but includes spread-spectrum signals as well as very highly optimized signals like the ones your cell phone uses. There has been a remarkable amount of R&D put into these signals to make them efficient, and because they have to push a lot of bits across, they are wideband, and so the signal-to-noise ratio in any given frequency bin a few Hz across is going to be poor. They're just plain hard to see, and there are a surprising number of them out there. This is where cyclostationary processing, a form of higher-order statistics, comes into its own. Cyclostationary processing is used to extract the bit rate from an unknown signal. Even if you can't see the signal through second-order statistics (like the autocorrelation) the bit rate of the signal will often pop right out of a cyclostationary approach. The math can be intimidating, but if you're really interested, stick with it and you'll find that it's really not that hard.

Here are a few examples to illustrate the autocorrelation/power spectrum approach. In the images, I plot the signal first, then the power spectrum, then the autocorrelation. These are simulated signals with sampling interval $\Delta t = 1$.

### White Gaussian noise

Flat power spectrum, autocorrelation has a peak at zero lag, flat everywhere else.

### Barker-coded pulse

Radars use these to reduce their peak power requirements while maintaining range resolution. $\sin^2 \omega/\omega^2$ structure due to the pulse shape. Autocorrelation shows a narrow peak at zero lag. In the wild, these pulses are much shorter than I show here, leading to flatter, wider spectra. They can be difficult to distinguish from noise unless you have some independent way to calculate the SNR in the channel (which is difficult if you don't actually know that there is a signal in the channel). Best to just look directly in the time domain.

### FM with sinusoidal modulation

Easy to see in both the power spectrum and autocorrelation.

### Sum of all three

Here I've set the noise $\sigma = 1/4$, the peak amplitude of the pulse to $2$, and the peak amplitude of the FM signal to $1/4$. The fourth plot is a close-up of the autocorrelation near zero lag.

• About 2 - I think need two FFT snapshots of the spectrum to do this, not arrays over time, like I have now? So for each frequency, I would take a sample at time t, wait a bit, then take a second sample at t + n, and use the two samples, and repeat for each freq? – HH- Apologize to Carole Baskin Jul 20 '17 at 14:26
• No, you will lose coherence that way. Do what I said above. Take one FFT snapshot, multiply each sample by its complex conjugate, then take the inverse FFT to get the autocorrelation. – Rodney Price Jul 20 '17 at 14:35
• One sample is enough to perform autocorrelation? That seems counterintuitive to me (but obviously I'm not very familiar with this) – HH- Apologize to Carole Baskin Jul 20 '17 at 15:09
• By "sample," most people would think you are referring to one number representing one moment in time. If that's what you mean, please go read the FFT article I linked to above. If not, presumably you mean one series of samples, forming a time series, which you then process using the FFT. That's what I mean, and yes, that's where you start when you want to compute an autocorrelation. – Rodney Price Jul 20 '17 at 15:25
• @MikeWaters, I'm glad to have him! horsehair, if you're using Python, Numpy has an FFT routine. I don't know a lot about Pandas, but if it uses numpy arrays you're probably all right. As to the autocorrelation, it will show many signals just as well as one. The quantity you are probably interested in is the ratio of the value at $\tau=0$ (and very close vicinity) to the max value everywhere else. You could make that more precise with the right kind of statistical test, but seat-of-the-pants will probably work just fine for your purposes. – Rodney Price Jul 21 '17 at 0:22

Well, this is a well-studied field (Radio Frequency Interference detection and mitigation). There are tons of literature about it.

The noise you receive is theoretically Additive White Gaussian Noise (AWGN). That means that its Probability Density Function (PDF) is Gaussian, and the pdf of its samples' power is exponential. By setting a false alarm probability according to the power of the signal, you can easily detect samples that potentially are not noise. Check the point 2.2 of this paper, for example. This threshold method is applied in the frequency domain in Frequency Blanking and Spectrogram Blanking techniques.

Normality Test are used as well to evaluate if a signal is AWGN. I have read a lot of mentions to Anderson-Darling test, but I never worked with it.

You can use Kurtosis to see if a signal is Gaussian or not (but be careful because Kurtosis is weak against certain sinusoidal signals). If it is not Gaussian, it is likely to be man-made (however, some frequency Jammers could emit Gaussian signals). Kurtosis has been also used to detect RFI in the frequency domain, not only in the time domain (look for spectral kurtosis).

Here you have an other interesting (and open!) paper about RFI detection.

I am not sure if there are AWGN-like signals made by humans different that noise for jamming. The spectrum of signals encrypted with codes (such as GPS signals, which use Pseudo-Random-Noise codes) are sincs, and they are only flat in a small fraction of its bandwidth. Even if you are sampling this small fraction, I am not sure if the signal in time will seem Gaussian.

• Here you have an other interesting (and open!) paper about RFI detection: mdpi.com/2072-4292/2/1/191 – Raul O. Jul 19 '17 at 15:05
• I have a big spectrum with many signals in it. It sounds like what you propose is for a give swath of bandwidth. I could roll over the spectrum checking each section (of some width), but the signals are of various widths as well; some are of width '1' (the sampling size), some are of hundreds or thousands of Hz... – HH- Apologize to Carole Baskin Jul 19 '17 at 15:14
• Welcome to Amateur Radio Stack Exchange and thanks for your informed answer! Just one thing — if you think of more to say, or if someone asks for clarification, please make sure to edit your answer to include the additional material. Comments should be considered ephemeral — to be deleted after they're handled. – Kevin Reid AG6YO Jul 19 '17 at 17:12
• @horsehair: when you just gather incoherent statistics, as you are doing when you take mean and standard deviation in each frequency bin, you destroy the coherent information that you need in order to determine whether signals exist. In that case, the methods given in the papers above aren't going to work. In particular, wavelet thresholding and multiresolution Fourier methods referred to in the papers above can't work. One thing you could try: check if the signal is white; that is, whether the mean is zero and the standard deviation is constant scross all your frequency bins. -Rod KD0FFJ – Rodney Price Jul 20 '17 at 0:27
• @horsehair: the trouble with the method I outlined above is that many wideband signals (meaning much wider than your frequency bins) vary smoothly over their bandwidth. Across any small number of bins, it will look white. That's why coherent methods are needed. I can make suggestions if you are interested. -Rod KD0FFJ – Rodney Price Jul 20 '17 at 0:33