Hi Rat,
If it is shimmed incorrectly then removing some of the shims will fix it but if it is already in the swivel pre loads then removing shims will only stiffen the steering by binding between the end of the pin and pulling on the bearing with the slop still remaining between the sides of the pin and the bush. If you remove your steering arms and the swivel is relatively free then possibly removing shims will do

Check the pin when you take it out for concentricity and feel the bush inside, you will feel if there is wear inside it. If the movement is in the bottom then shimming will fix it,
otherwise it's pull the shaft, drop the bottome pin, remove back seal take swivel off, bash out raiko bush with long bolt/punch, press new one in, assemble bottom bearing and shim for loading, grease back seal, drive shat etc. 1 hour job with the right sized hammers
Mike