Re: stereo cuts off at half volume or more?
Is the stereo itself powering down? Or are your amps?
It sounds more like your amps are going into safe mode from power starvation.
How are they ran? 3 subs and 8 speakers on only two amps?

Are they 2 ch. or 4ch.? While were at it, what wattage is the stereo head?
The power supply to the amps should have no less than 10g. wire feeding the amps.Did you install it yourself, previous owner, or shop?
To correctly install such a large system, you should be feeding a large gold plated fuse/splitter block with 8ga. and then run a 10g. to the amps. The ga. of speaker wire you use is also very important. As well as the stereo itself. I do not use any of the stock wiring in a boat for a new stereo installation.
Your best bet is to check the entire system and probably start over. I think you have some serious installation issues. There are a lot more factors here than just what size wire should have been used. How are all of the speakers hooked up, where are they hooked, ground to ground, etc. and so forth. The wire is the simplest of the process.