Why is it so stressful asking for a reference.

I just got an offer from another company offering 40% more but wanted some references before continuing and offering a contract. Now I've had to go to my manager and ask them to be my reference. God why is this so stressful.