Yes, that's mathematically and technologically possible. Because Mike already said the latter, let me elaborate on the math side:
Remember that SSB is really just AM, with one half of the transmit spectrum suppressed.
Now, do the following thought experiment:
You want to transmit a single RF tone at frequency 100.5 kHz. All you have is a SSB transmitter that can tune in 1 kHz steps.
What you can do is simply tune to 100 kHz, use the upper side band, and send a 500 Hz tone. Simply because AM does nothing but mixing up your baseband signal by a carrier frequency.
If you think about that, you could also PSK-modulate your 500 Hz tone. You will have a PSK transmission at 100.5 kHz.
Now, regarding what things to worry about:
- filtering: the bandwidth of the sideband you're using must fit the bandwidth of PSK you're transmitting.
- That bandwidth in turn is defined by your PSK modulator's pulse shape. A common pulse shaping filter is the root raised cosine, but that's just a popular choice that allows for easy matched filtering in the receiver. If you want to use a digital mode, you'd want to stick with the pulse shape they define¹.
- AGC: TX Side: won't do anything, since PSK is inherently constant-envelope, if you look at it from the perspective of the control loop in charge of keeping voice amplitude about constant: the symbols themselves have constant power, and the transitions between them happen much faster than a control loop that mustn't destroy voice would be allowed to correct; in control theory speech: the bandwidth of amplitude variations of any reasonable PSK transmission is much higher than the AGC loop filter's bandwidth.
- AGC: RX Side: adjusting the gain of an amplifier does have an effect on phase, which is ummm not optimal if your information is in that phase. So, in a proper PSK receiver system, you'd start with high gain, and wait to see the start of a transmission by its preamble. You'd use that preamble, which usually is long and robust enough, to "train" the AGC and then "lock" it to the resulting gain setting. Nice thing about PSK is that it's relatively robust against driving receiver amplifiers slightly nonlinearly, so in doubt, the algorithm that does the "what's the proper AGC setting" is allowed to risk slight overdrive in case the transmission strength increases during one packet. Now, you'd typically design your packets to have a length that is no longer than what you'd assume your channel to "hold" a gain value, but that's system design, and might be leading too far.
- Now, you mentioned that the SSB input might very well be fitted for voice comms, and that's a real problem: No way on earth that thing is going to have a flat passband – it might emphasize voice frequency components that are expecially relevant for understandability, and cut off very early. Also, might intentionally not be a linear system. Your only way around that is actually compensating that on TX side. If you want more bandwidth than your voice input is reasonably flat about, you'll need to pre-equalize your signal so that it comes out of that voice filter "flat". Finding the equalizer that does that is usually a non-trivial task. It'd be enough to find the perfect equalizer once, and use it to counter the emphasis of the voice filter, but from a practical point of view: Have you considered sniffing your own signal with e.g. an RTL dongle and using your known own "perfect" transmit signal to find an equalizer that reverts the filter effect? Do that in a loop, and iteratively convolve the equalizer coefficients that get automatically estimated with your momentary TX equalizer taps to pre-equalize your transmissions. If my thinking is right, your estimator will in the end converge to a "need no more equalization" state, which means that your pre-equalizer does what it should.
- Since equalizers are hard, start with low baud rates = low bandwidth first. Maybe a 100 Hz tone with 50 bd – that'd be trivial to come up with: GRC File - open in GNU Radio companion