For me, without a shadow of a doubt, it is some of my colleagues, the drama, the bitching, the bullying, the sheer nastiness of the gossipping.
I clock in, do my job and clock out.
I am a fairly easy going, easy to talk to kind of chap, but I would take a pay cut to work alone.