Somewhat guessing but it must be a dirty contact somewhere. Key being on is giving it more power and going down requires less amps than going up.
My motor the relay steals power from the hot side of the starter solenoid. Look at and clean all contacts including ground.
Weird one.