The pinion gear is in constant mesh with both forward and reverse gears. Thus: Both gears are turning in opposite directions any time the engine is turning. The dog clutch is splined to the prop shaft and turns it. If it is slid forward to mesh with the dogs on the forward gear the prop spins in the forward direction. If it is slid backwards to mesh with the dogs on the reverse gear the prop spins backward.
The dogs on the gears and the dogs on the dog clutch are finished vertical to the face of the gears and clutch. Usually the dogs are tapered on the top surface to ease engagement. Once engaged, it takes very little force from the shift linkage to hold the dogs in place. As they wear, the forward edges of the dogs become rounded and thus more pressure is necessary to keep them engaged. When they wear enough, pressure from the engine turning the prop will cause them to pop out then re-engage with a bang. As wear progresses this will happen at a progressively lower speed.
Old OMC engines did have two spring loaded balls passing through the prop shaft. These balls engaged detents on either side of the dog clutch to help prevent it from slipping out of engagement.
