The display is simple:
- Pass the incoming signal through two band-pass filters, one at the mark frequency and one at the space frequency.
- Use the two filters' output signals as the X and Y coordinates of the plot.
- (Since you are in the SDR domain:) Do as much as you want to make a nice-looking digital oscilloscope display of that signal. The simplest accurate way to do it is to keep a buffer of the last however-many filtered samples and draw them as dots (not lines).
If you had perfect signals and perfect filters, the display would not display ellipses but a straight line on one axis or the other. Since instead filters have roll-off, the signal is merely highly attenuated in the other axis, and an elliptical trace (for an input signal at a single frequency) results. The sharper your filters, the narrower the ellipses will be. The exact shape of the ellipse depends on the phase response of the filters (for example, an ellipse that is perfectly axis-aligned implies a 90° phase shift).
The "crossed" ellipses result from the modulated signal alternating between the mark and space frequencies and thus between the two possible displayed ellipses. A signal between the two frequencies will appear as a severely off-axis ellipse or a circle (because both filters are passing the signal), and one outside them will appear as an attenuated ellipse. Thus, you can tune a RTTY receiver by changing the receive frequency until the distinct horizontal and vertical ellipses appear.
(Disclaimer: I have neither used or implemented one. All of the details in this answer are based on reasoning from signal processing theory.)