They certainly not my Favorite Item either, but many times I have removed them on Carbs that the Owner insisted were spotless, and found a pile of Gum/Varnish/Dirt hiding behind them with no way to get out.
On the Carbs with fixed Idle Jets, Settings like the Float become Critical, as well as proper Gasket Fitment, as you mentioned above. Wear on the Throttle Shafts can become an issue over time, as that can allow excessive Air to enter, which is undesirable as the Throttle Plates are Fully Closed at Idle, actually quite higher than Idle Speed, and an Lean Condition can occur.