Re: what is the safe size
You can do it safely if you plan well. Check the weather forcast, bring good lifepreservers, be sure your bilge pump is working, and be sure that someone knows where you are going and when you should be back. Its called a "float plan". If you have cell phone service in that area, call them when you leave, and again when you arrive on the island, again when you leave the island and arrive on the mainland.
As long as someone knows what you are doing and when you should be back, then even if you get lost or break down, someone will call the coast guard to go save you. Make sure they have the coast guard number.
In nice weather, you should be able to scoot over to the island in less than half an hour at a steady 22mph. Sounds like a nice trip.