Even with a boat that large, the 5.7 should be spinning a lot faster with a 17 pitch prop. I would start verifying your RPM readings with a shop tach. The dash tachs are notoriously inaccurate. Also, obtain the drive ratio.
After you have verified the reading, as previously mentioned, lighten up the load as much as possible.
Have the boat weighed as well. Dropping the trailer on a truck scale and also weighing the empty trailer to find your weight is the easiest method. You want to determine if the boat is water logged. If the weight is off, start exploring for water in the transom, stringers, and foam. You do have a boat that is beyond its designed lifespan at 30+ years old, from a period where Bayliner was not known for its quality builds. This boat has also had some work done to it as it is rare to see an OMC drive mated to a Merc engine.
Do all of these items before investing in props.
I have a 2460 hardtop bayliner trophy 1988 with a omc cobra drive and mercruiser alpha one 5.7 engine that gets me 35mph @ 3800 rpm WOT with a 14.5 X 17 4 blade prop. do I need to go down to a 15p and a 15" prop to get around 4200+ rpm?