What restrictions exist for non-HAMs in the 5.8 GHz band in the USA?
You should be covered by FCC Part 15 unlicensed use as long as
For the band 5.725-5.85 GHz, the maximum conducted output power over the frequency band of operation shall not exceed 1 W. In addition, the maximum power spectral density shall not exceed 30 dBm in any 500-kHz band.
FCC 15.407 section covers in more detail. For example you must reduce power when using a directional transmitting antenna..
I believe that in order to comply with licence-free use, all transmitting equipment must be type-approved.
One option available for home-built equipment would be to work within the regulations of the amateur radio licence.
By this, I mean that if you were going to build some equipment yourself and experiment with it, even with a view to producing it commercially later, then in order to test that it works correctly you would need SOME sort of licence. The amateur licence would serve well for this purpose (self-training, etc.), although it does not of course permit commercial use of the equipment. It would also require that the person/people doing the work would hold the right kind of licence themselves.
Once developed to the point where you have something that works, you could apply for type approval, but my understanding is that this is a lengthy and costly procedure.