I refused and was asked to leave. When I informed my boss that he wasn't entitled to my phone, he said he was.
I'm in the UK so if I'm wrong please feel free to correct me. I'll find a job quick enough as there's plenty of work here but it's annoying that people like my manager can do what they want..