"The boat had a 15.5 17p prop on it when I bought it and it was even worse, probably topping out at 3800RPM"
gb... that engine, "if healthy" should be turning a prop at 4400-4800 rpms in it's "sweet spot". what's that? that's the recommended WOT power band with the correct prop for most Mercruiser engines recommended rpm range is 4400-4800. The size/weight/hull configuration/motor power are all different, the right prop for a 3.0 engine may not be the same as a 4.3 or a 5.0.. etc..,, ie- the correct prop for your boat, wont necessarrily be correct for a different boat of different configuration.. you need to determine if your engine is healthy,, ie- get a compression check.. if that's normal, what your reporting is odd. a lower pitch prop- ie- 17 should be turning faster than a 23. ie- your rpms should go up,,about 200 per inch of change "down in pitch".. you should not be getting the same results.. or worse.. I'd get your engine checked first..