there may be a small plate with a fin on it below your cavitation plate. Try rotating that plate opposite the direction your motor turns, a little at a time until the motor tracks straight.
My old 25 Merc had one. It's actually the zinc anode, and the fin is the first part to wear off with age. Your fin may be gone, or just not pointed in the right direction.
I had the same problem with mine. I let go of the tiller once at full speed just to see what happened. Let's just say I didn't do it again.
The steering tension is controlled by what Merc calls a "co-pilot". It's the bolt at the top of the stem (or steering tube or whatever). I always had to adjust mine depending on what I was doing. Tight enough for hi speed was too tight for trolling etc etc.