If it's a big company with a dedicated HR department, you'll PROBABLY be OK, since they won't know you very well personally and will just basically be following a script if some prospective employer calls them.
On the other hand, if it's a small office and the person who will be recommending/verifying is a petty, vindictive, grudge-holding douche nozzle, get out of there as quickly as you can. Hopefully, quick enough that you won't have to put them on your resume.
You don't want to work for some place that will bad mouth you to prospective employers when you go out job hunting again. Some people love nothing more than to ruin your job prospects at every opportunity.
Remember, you're only dating your job. You're not married to it.