Your line of sight to the ship transmitting the AIS signal is one of the more limiting factors. Usable AIS VHF signals will travel just slightly further than line of sight conditions. This is known as the radio horizon. I have plotted the height in meters you will require in order to obtain a desired receive distance of AIS signals:
This chart assumes the transmitting station on the ship or boat has an antenna height of 2.5 meters ASL (Above Sea Level). If you do not obtain sufficient height of your receive antenna for the distance you desire, no other remedy will reliably increase your range. If you have trees, buildings, poles, etc. between your receive antenna and the transmitting antenna, the range will be less than this chart predicts.
Assuming your receiver is optimized, and you have met the height requirement, there are a other few things to think about.
Your coax cable introduces losses into your AIS signal which effectively weakens the signal. Keeping the coax as short as possible and using higher grade coax can mitigate the effects. Hams would typically use LMR-400 or equivalent for a VHF application such as this but you may also have success with a high grade 75 ohm cable TV coax.
The location of the antenna will determine its susceptibility to man made interference from a variety of household and industrial electronics. Keeping the antenna as far away from these as possible will minimize the interference.
The antenna design can have a profound effect on received signal strength. Your current antenna is marginal in this respect. A collinear style antenna would offer you much more gain while retaining your omnidirectional coverage. The antenna will be taller than the one you are presently using. It is generally enclosed in a thin fiberglass radome to protect the antenna itself.
Another antenna option is a small, vertical yagi antenna. A three or four element yagi will dramatically improve your received signal strength. It is a directional antenna but when pointed toward the water, its beamwidth should offer quite good coverage for AIS signals. But it is directional so it will favor some areas over others. The yagi antenna will not be any taller than your current antenna and it resembles a typical TV antenna to the untrained eye.