Assuming I wanted to reverse engineer 13FSK, what steps would I take to begin reverse engineering it? I have seen tools such as inspectrum and baudline that can help me view the raw signals, but I am unsure as to how to reverse engineer the 13FSK and then build a way to receive and transmit data using 13FSK.
To reverse engineer anything, you need a sample to look at. Can you generate valid 13FSK signals? If so, then you need to capture it.
Also, you'll need manipulate the transmitted payload so you can compare "before and after" to see what changed.
If you mean you are looking at something like Pactor 3 or 4 ... then good luck.
I did have a look at this a few years ago after buying a very expensive (but functional) Pactor 3 Modem....
I apologize that this is not a more constructive comment. However such Digital transmissions are usually compressed (possibly using commercial algorithms) and may additionally use proprietary codecs.