For the purpose of this answer, I'll lump interesting unknown signals and RFI together and call them "mysterious signals", since the techniques for finding where they come from are often the same. Some of the techniques listed below may not work for all bands and power levels.
Often it would help to know how local the mysterious signal is. If you can ask another operator (or an automated receiver connected to the internet) to listen to the signal, then that might help; if the signal can be heard some distance away, then its origin may not be local. Of course line-of-sight and other range issues make this technique more practical at some bands and power levels than others.
Some mysterious signals could be spurious, generated by your own station, or even inside your receiver. Receivers, computers, switching power supplies, and other station equipment are full of oscillators, and sometimes these oscillators are heard in a receiver, in which case they are called "birdies" or "spurs". Birdies are often unmodulated sine waves, but if the source of the birdie is a digital signal, then it may not be just a pure sine wave. If the receiver is self-contained (not connected to a computer), is being powered by a battery, and is connected to a dummy load, then the signal you're hearing is probably a birdie. (A receiver connected to a dummy load can sometimes pick up strong external signals, because coaxial cable shielding isn't perfect.)
Another source of spurious signals is intermodulation, often called "intermod" or intermodulation distortion (IMD), which is caused by the unintentional mixing of two or more signals and/or their harmonics. Intermod is sometimes generated inside a receiver, for instance when a strong signal inside the receiver's passpand gets sent to a mixer in a superheterodyne receiver. Any nonlinear device can act as a mixer, whether it's in a radio or not, and sometimes the nonlinear device is two different metals touching that are exposed to RF, such as an aluminum gutter touching a steel pipe. The RF signals generally must be strong to generate noticeable intermod from just two pieces of metal touching. If you suspect intermod from inside your receiver, check to see if there are very strong signals inside the passband, and then see if the mysterious signal is still there when the strong signals go away.
Then of course there is plain old distortion, caused by too much gain in an amplifier stage of the receiver. Distortion is at least easy to troubleshoot: if reducing the gain or adding attenuation makes the mysterious signal go away, then it's probably distortion.
A good way to check if a mysterious signal is spurious and generated inside the receiver is to try a different receiver, preferably of a different type.
In the modern era when we are surrounded by more and more electronic equipment, often mysterious signals are generated from other equipment in the same house or building. Troubleshooting these can be simple: switch off circuit breakers, or unplug equipment from mains power, and if the mysterious signal suddenly disappears, then it probably was generated by a piece of equipment that had its power interrupted.
If you suspect that the mysterious signal comes from a neighboring property, and if you are on very good terms with your neighbor, perhaps he or she will let you switch things off to see if the mysterious signal stops.
If you believe that the mysterious signal is real, and not coming from equipment you can disconnect from power, then it's time for other troubleshooting techniques.
If you have a portable receiver with a signal strength indicator (sometimes just "full of static" vs "clear" is good enough) and a portable antenna, which doesn't necessarily need to be directional, then you can go looking for nearby sources of a mysterious signal. In the simplest case, if the signal strength goes up when you are moving around, then you are getting closer to the transmitter or antenna. But be aware that radio signals can reflect off objects (mountains, hills, buildings, vehicles, trees, etc.) or refract around them. It's common for a radio signal to not be coming from the direction of the transmitter or antenna. But if you can rove around the entire area where the mysterious radio signal might be coming from, then the signal is probably strongest near the transmitter or antenna.
You might experience the problem that near the transmitter the receiver's signal strength indicator reads maximum whichever way you go, which prevents you from finding the exact location of the transmitter. In that case the solution is an attenuator. Sometimes traditional attenuators don't have enough attenuation, because leakage around the attenuator is still strong enough to cause a maximum signal strength reading in the receiver. In that case an offset attenuator can sometimes help. An offset attenuator is a simple circuit, often home-brewed, that mixes the signal with a local oscillator to produce a mixed signal near the original signal, but not so close as to be in the receiver's passband with the original signal. The receiver is then tuned to the mixed signal rather than the original signal. The strength of the mixed signal can be easily reduced by reducing the strength of the local oscillator. In that way an offset attenuator can inexpensively permit more attenuation than a traditional attenuator.
If the portable station has a directional antenna, then that often allows a portable station to find the transmitter or antenna more quickly. Often an antenna doesn't need to be very directional to be helpful. Sometimes just using one's own body as an attenuator between the transmitter and the receiving antenna can give some idea which way the transmitter is.
If the mysterious signal's transmitter is too far away to rove around, or if the receiver or its antenna isn't portable, then all may not be lost. Directional antennas can give an idea of the direction of the transmitter. Ideally, several stations working together can triangulate the approximate location of the transmitter. Unfortunately this technique often doesn't work as well as it does in movies, because many directional antennas have very broad lobes, and because the radio signal may have been reflected or refracted. Or the signal may be arriving from several directions simultaneously, which are also interfering constructively or destructively, a situation known as multipath interference, which can make determining the direction of the transmitter difficult. So sometimes triangulation doesn't work very well.
