Some interesting notions on the source of the yawing effect, but have to say that anyone who has raced sailboats intuitively knows the real cause--vortex shedding.
Basically, if you're flying a spinnaker dead downwind, it's acting like a giant parachute, with wind spilling over both edges. A little vortex is created at the edge, which quickly grows and eventually breaks off and spreads to the center of the spinnaker backside. A vortex breaking off the starboard edge will spin to the center rear and the friction against the back side of the spinnaker will pull the spinnaker to starboard. But then after a few seconds the vortex will break away, to be replaced by a similar vortex, this time from the port edge, which then pulls the spinnaker to port, and so on. This back and forth rolling effect can become so strong that it makes the boat heel strongly back and forth as it continues downwind.
A ski boat has a very blunt edge separation to water flow on port and starboard stern sides at low speeds, and similar vortices are formed in the water. One vortex current will shove the stern in one direction, followed by another vortex pushing in the opposite direction. And the boat wanders like a drunken sailor.
If you try, you can make minor steering changes in anticipation of the effect and minimize the yaw.
Good YouTube videos showing Vortex Shedding.