Assuming I wanted to reverse engineer 13FSK, what steps would I take to begin reverse engineering it? I have seen tools such as inspectrum and baudline that can help me view the raw signals, but I am unsure as to how to reverse engineer the 13FSK and then build a way to receive and transmit data using 13FSK.
If you mean you are looking at something like Pactor 3 or 4 ... then good luck.
I did have a look at this a few years ago after buying a very expensive (but functional) Pactor 3 Modem....
I apologize that this is not a more constructive comment. However such Digital transmissions are usually compressed (possibly using commercial algorithms) and may additionally use proprietary codecs.