I'm not sure whether it's more appropriate to ask this question here, or in the dsp.stackexchange. But I'll try here first.
There seems to be a lot of information available on the filtering and demodulation of SSB signals in the analog or time-domain. But my iOS RTL-SDR radio app is doing all of it's fine tuning and filtering in the frequency domain (via overlap-add FFT fast convolution). So I'm wondering what would be the cleanest method of filtering in the frequency domain, without just emulating the chain of processes needed in the time domain or with an analog SSB demodulator. For instance, a Hilbert transformer isn't needed, since the input to the complex FFT filter already has the imaginary component of the complex or IQ signal provided by an RTL-SDR (or other similar SDR front-end). And asymmetric filtering can be accomplished in the complex IQ frequency domain without shifting the signal’s spectrum.
Here's one possible idea: Shift my FFT lowpass filter from being magnitude symmetric around DC to either only the positive or negative frequency side of DC (equivalent to using a complex FIR filter impulse response of the same length or order). Mirror/flip the retained side (positive or negative frequencies for USB or LSB) to the other side of DC and complex conjugate it, resulting in a strictly real time-domain signal after the fast-convolution IFFT. Feed that result to a simple AM demodulator. Would that work?
Is there another canonical and clean SSB demodulation method?