...Though I don't know if the Vengeance is a good hole shot prop.Keep in mind an I/O is tail heavy and you added a little more with the 5.7.
I have a couple of Vengeance props and although not as good as a 5 blade (for obvious reasons) is still quite impressive. Even cruising at around 20knots, if I open the throttle hard, I get a kick in the backside...
The weight of a 5.0 and a 5.7 are EXACTLY the same... So he has not added any weight whatsoever.
It seems I recall that someone said Vented props aren't that effective with a I/O.
If that's the case, they must be nothing short of heart-stopping on an outboard. On my 4.3LX a vented 21" Laser II was accelerating far quicker than I even expected, or believe my boat could ever do...
I don't think there is a rev limiter your motor was simply at its design limit.
Need to know what ignition system he has to know if there's a rev limit. One thing he does need to consider, and this may be contibuting to his performance problem (and I've only just realised it :facepalm

is he changed from a 5.0 engine block to a 5.7 engine block. Now, the rest of the gear is bolt-off, bolt-on, but the ignition module controls timing advance, and there is quite a difference in the curves between those 2 engines....
You will need to increase pitch to get your rpm down. Increasing pitch usually slows hole shot.
The gurus will be along to suggest some ideas I'm thinking a 24" pitch if your tach was accurate and you weren't at wot yet.
He also needs to get a 'shop-tach' in there. Dash tachos are notoriously inaccurate....
Do vented props work without through prop exhaust? My exhaust ports out well above the prop.
Last drive without thru-prop exhaust was a VERY old Volvo or Chrysler. What drive do you have? (knowing it's likely a Mercruiser since you're running a Vengeance prop... ??????)
And there in lies another issue... Props like Vengeance, Laser, High-5, Mirage, etc, etc are designed to have exhaust gasses running through them. It's part of the 'tuning' of the prop. No exhaust gases means a massive vacuum is created behind the prop, seriously limiting performance.... This may well be the reason your acceleration is so off.... It would be like trying to drag a small parachute behind you.
If your exhaust ports are 'well above the prop' because of thru-hull.... Ditch them, and go back to thru-prop. The performance benefits of thru-hull on a 5.7 engine can't be measured, it's that little.... And you're acceleration will be considerably improved.
Chris.......