Re: how young is too young?
the question has two unrelated but equally important components: the safety of the child and the happiness in the boat.
Safety: A swim sweater with an inner tube is not enough and in fact both illegal (not in compliance) and dangerous. You nee a properly fitted USCG approved PFD.
The child has to be secure, but not tied in, so that if there is a sudden stop or roll he doesn't fly out of the boat (eg your wife's arms) or slam into something in the boat. a backwards=facing car seat works (don't strap him in).
If he's mobile, he has to be confined to an area in the boat where he can't go over the side. Bass boats by design will be difficult.
Sun protection is a safety issue
Happiness: If the mother is nervous, the child will be cranky. If the child is cranky, the mother will be agitated. Some babies will sleep through it all; some will have fits.
You need to stay within a few minutes of the house (or car if you are trailering), abort the trip at the first sign of trouble, and if yo are trailering, have two cars so your wife can go straight home while you deal with the boat.
Of course that's a generalization; some mothers, especially those who are truly boaters, will deal with it fine. it sounds like yours is not.
Frankly, I think taking an infant out accomplishes little and is more likely to end up a negative experience than positive. But I also believe in getting children to be boaters early, giving them responsibilities and teaching skills at age appropriate levels, so they can operate a motor boat by 12, solo by 14 and trailer by 16. Teaching them to sail at a young age is a great way to do this.