Ok, plenty to unpack here. I've skim read most replies, and I think you're missing a few bits of information....
I am using this boat offshore in SOCal to occasionally go to the Channel islands, yes I know it's a bowrider but it works great. I recently rebuilt the Alpha 1 gen 2 drive myself successfully.
Excellent, you used all the right shimming and rolling torque tools I assume.
So the propeller on the boat is a 14.5x18 aluminum, not the original size which broke, one that I bought awhile back. I am not too happy with it especially in semi choppy waters.
Need more information on this prop. 3 blade, 4 blade? Manufacturer?
The gas mileage sucks but these are gas guzzlers compared to let's say my friend's Trophy II 115 outboard that can do 247 miles per tank.
You're engine should be making around 260hp. More than double that outboard. I'd be shocked if you
didn't use more fuel. Also, he's getting 247 miles (statue miles or nautical miles?) but you didn't say how big the tank was (or the boat). If it's 250 litres, that's lousy, if it's 50 litres, that's brilliant. It's like my wife's car only gets 850km on a tank, but my car gets 1200km. Which one uses more fuel do you think? It looks like my wife's, but her car has a 70L tank, and mine has a 145L... Looks a bit different now, yes?
So I will never be able to reach such a performance since I have a 5.7l sterndrive.
Rubbish! Don't measure your performance by how many miles you can go 'on a tank'. To do it properly (and compare apples with apples) you need to be working in litres/nautical mile, and account for different boat sizes! As a comparison, at cruising I get around 1.3litres/NM. Given your boat is very similar to mine, with a slightly larger engine, I would expect you to be in the 1.5 to 1.7L/NM range.
Carb is rebuilt to specs and adjusted properly. Going to the Channel Islands from Oxnard is roughly a 12 miles trip with another 6 there and coming back I spend roughly 14 gallons of gas.
US gallons or imperial gallons (not everybody lives in usa). I'll assume US gallons (which are only 3.78L to the 'gallon'.) 53 litres for 16 NM (I have assumed you're using statute miles as you are using imperial for everything else

). That's roughly 3.3L/NM, and that's not good. Something's up.
And in semi rough water with winds of over 12 knots even more. You guys probably know about it.
Haha, I'm on the lower west coast of Australia. We call wind under 15 knots a 'breeze'. Our 'afternoon seabreeze' is almost always over 20 knots, and I've been caught out with it belting in at over 30 knots (this is what happens when you're underwater for an hour

).
My question is can I get a different propeller that can get the boat on plane below 20mph that it does right now.
Yes. Have you tried a stainless steel 4 blade?
At WOT I get 4100rpm and 40 miles per hour speed.
Ok, we have a problem. That WOT engine speed is WAY too low! You need to be getting it up around 4500rpm as a minimum. And that low a WOT RPM will also be contributing to a few driveability issues you may not even notice you have. Boat's sluggish and unresponsive to throttle. Struggles up swells then races down the other side.
30 rpm is 30 miles an hour. This motor specs say 4200-4600 RPM. I don't care about speed since yesterday was the first time I could take it and actually reach 40mph the ocean was the calmest I have experienced in a long time. It's August after all.
So what size propeller and pitch and why? Thanks in advance.
So, I have run the current figures through my prop calculator, and it all looks ok for what you have (with respect to the prop)...
But I think you have other issues. A 260hp engine should be pushing that boat at WAY quicker than 35 knots! I have a similar boat with 40hp less and I'm pulling up at the 40 knot mark (with a 3 blade prop. 38 with a 4 blade). I'd expect you to be up around 42 knots with that engine. I suggest you do some more engine checks and make sure you're getting all the horses it should be making. I know speed isn't what you're looking for, but a low top speed indicates a engine performance problem somewhere, and that is likely the reason for the unexpected high fuel consumption.
Start with a compression test.
I'd also like to know exactly which '5.7L' engine you have.
Please post engine serial number, or at the very least, engine model and year.
Chris..........