When people swap tire sizes on a 4wd they really can mess things up!
In your case they have put on too small of front tires and to large of rear tires, throws the gear ratio all out of whack!
Your front should be 8.3-16 and yes there is a RC (rolling circumference) difference between that size and the 8-16.
Your Rear tires should be 13.6-24.
Yes you probably can get away with the front tires being off the little they are but the rears are pushing the front around.
Now as far as why it won't shift into 4wd, that could be a direct result of binding or worse yet damage to the drive from being the wrong ratio.
Jack up the front axle to get just one tire off the ground, while someone turns the tire back and forth work the 4wd lever to see if it will engage the drive, caution do not run it in 4wd till you fix the tire miss match!