Far too often people ask who they know/have worked for/with, for a reference and take it at face value that the referee will give them a good or positive reference, when called by a prospective employer. Don't be so sure. Sometimes a referee could give you a bad reference, without even realizing it.
For example, they might say something like, “I know X because at work we used to chat when it wasn't busy” which will be followed by “how often was it not busy” with the answer “oh, every other day, 1-2 hours, here and there.” Basically, your referee just outed you that you have the tendency to talk too much on company time, as opposed to working. It's stupid and petty, but these are the things some HR may look at very closely.
Of course, then there's the referees who will straight up give you a bad reference on purpose, despite telling you otherwise. I doubt the prospective employer will tell you about the bad referee (they have no reason/obligation to tell you) and it's not like the referee will be honest with you. So, be careful out there.