Reviewing this whole matter, I'm going to give you an opinion. My opinion is that your 10-1/2 x12 is ok for your boat. Full throttle operating range for your motor is 4000-5000 RPM. So you are within the ballpark, albeit at the lower end of the ballpark. You also say you are not interested so much about top speed and find that it is almost too fast now, and you don't even normally run it at wide open throttle. With all that in mind why increase revs and fuel consumption? Overstressing the motor is not an issue if you aren't even asking it for maximum effort.
Just my opinion. Others may disagree.