Back rail will always beat mast top for one simple reason: Cable losses. A typical antenna coax will lose 80% of it's signal strength per 100' run. Going off the back rail is at most a 20' run, whereas going up the mast is at least 40'. The 20' run loses half the power as the 40' run. Cable length is a much bigger factor than antenna height off the deck with VHF in the 155MHz range of the marine channels. VHF curves with the ionosphere and thermal ducts (marine layer) that are common on the ocean surface. That's why they chose that band for marine comms.
Rule of thumb: Every 3db gained or lost is twice or half the power.
Height of an antenna doesn't begin to dominate until you're well into the UHF range above 600MHz and it's not critical until you're well into microwave ranges above 1.2GHz.
The other major factor with external antennas especially with handhelds is the length of the antenna itself. A 2 meter dipole antenna is theoretically perfect for omnidirectional VHF at 155MHz, and a one meter whip is the next best. The small handhelds have to use impedance matching in their smaller antennas that cost power which results in much lower gain and thus lower range.
I'm not about to put a monster 2m antenna on the back of my boat even though it's gain is double that of a 3', but a 3' whip is a good choice for a Mac IMHO. Go for thicker coax, either RG-213 or RG-8. RG-58 loses considerably more signal. For reasons Jim points out above, try to buy the cable to-length and pre-terminated. Crimping Coax sucks and you'll probably never use the crimper again.
Keep in mind though: Your floating handheld isn't going to float if it's tethered to your external antenna
